Win a copy of Transfer Learning for Natural Language Processing (MEAP) this week in the Artificial Intelligence and Machine Learning forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Tim Cooke
  • Paul Clapham
  • Devaka Cooray
  • Bear Bibeault
Sheriffs:
  • Junilu Lacar
  • Knute Snortum
  • Liutauras Vilda
Saloon Keepers:
  • Ron McLeod
  • Stephan van Hulst
  • Tim Moores
  • Tim Holloway
  • Piet Souris
Bartenders:
  • salvin francis
  • Carey Brown
  • Frits Walraven

Need Guidelines about the Application

 
Ranch Hand
Posts: 37
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Dear Friends;
I am confuse how to develop an application. Here i write some points which confuses me.

Actually the application is related to store the parameters of Gas turbine. The operator task is to just enter the data after one hour while The manager level wants to see that data upon choosing parameters which they want to see.

Now my thinking is that develop an application in JAVA (Because i just have some experience on java), Buy a domain and install MYSQL. Create tables and database and normalization done.
After that develops forms for operator from which he insert the data.

But the Data insert when Internet connection available. When we loss the internet connection then what can i do? I am confuse here then what strategy i adopt?  

The second one as the time passes the database becomes heavier i.e after 1 to 2 years. and data send to remote server and fetch from that takes time. what procedure i adopt?

In my mind there is two solution may be i am wrong and friends you will guide me better.
1. Save the data in excel sheet ot CSV file first and then send to database.
2. Yesterday i read about Mirroring of database (But i am unfamiliar with that) I do not know how to use it in Java?


Please advise me what is the best strategy for develop the application?

Your early reply will be appreciated.

Thanks      
 
Marshal
Posts: 25452
65
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Hamad Sultan wrote:But the Data insert when Internet connection available. When we loss the internet connection then what can i do? I am confuse here then what strategy i adopt?  

The second one as the time passes the database becomes heavier i.e after 1 to 2 years. and data send to remote server and fetch from that takes time. what procedure i adopt?



If you design a solution which requires an internet connection, then there isn't much that you can do via programming to deal with the situation where the connection is down. Now, if you have an internet connection then it WILL go down sometimes. For example I live in a fairly large city in a First World country and last month my doctor's office lost their connection. Or maybe their server went down. So they couldn't make appointments for a day. There's no fallback because that almost never happens so it's not worth setting up a system to deal with the situation where there's no connection.

But if you're in a place where internet failures are a regular feature of life, then yes, you might consider designing your system so that it doesn't use the internet connection that much. For example you could save your hourly input data in a local table, and have a process which can upload it to the internet when there's a connection. Or you could just run the whole system locally and not use the internet.

And when the database gets bigger and the system gets slower? Most of the time people fix that by getting more hardware, or faster hardware.
 
Hamad Sultan
Ranch Hand
Posts: 37
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello Paul
Thanks for reply. As you mention
" But if you're in a place where internet failures are a regular feature of life, then yes, you might consider designing your system so that it doesn't use the internet connection that much. For example you could save your hourly input data in a local table, and have a process which can upload it to the internet when there's a connection. Or you could just run the whole system locally and not use the internet."

Suppose i import and export the database  from localhost to on a domain server when connection available.
For example 100 records in an table stored on the domain server. after 2 days suppose 75 records now in a localhost database now i want to insert these 75 records to the domain server when internet connection available.

I never seen any solution to import or export selected records. MySQL just import and export whole database with tables.

Is there any option to insert the data from localhost to server domain. Please help me i need some detail explanation with example what strategy is best from insert the data from localhost to domain server.

Regards  
 
Saloon Keeper
Posts: 6384
158
Android Mac OS X Firefox Browser VI Editor Tomcat Server Safari
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
While there is a whole bunch of tools for dealing with problems like DB synchronization (ETL tools), in this case you would probably write the solution yourself, rather than have the DB deal with it. For example, each newly created local record could have a flag "not stored centrally yet". When the app is next used with an active connection, you'd upload just those records to the server. You might also want to check at that point if there are new records in the central DB, and download those to the local DB.
 
Saloon Keeper
Posts: 21975
150
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I do a lot of work with IoT devices these days and almost always when they have an Internet connection I design things to allow for both Internet and power failures.

The edge devices I use are generally based on either Raspberry Pi or Arduino-like technologies. The Pi is especially good for databases, since it's running a full-featured Linux (or other) OS, but an Arduino can post to a PC or server using any of a number of protocols, including ReST.

If you want a local PC and a cloud server to share identical copies of a database, you can do that. In fact, MySQL has several mirroring services that can deal with that, I believe. Even without vendor mirroring, however, you can make the syncing job easier by simply timestamping logged entries and only shipping ones whose time is newer than the last sync. No need to replicate the entire database.

Databases do not get radically slower as they get bigger. They're designed to be big. Some tuning may be required, especially in cases where you want especially good performance for common functions, but that's what Database Administrators are for.
 
Paul Clapham
Marshal
Posts: 25452
65
Eclipse IDE Firefox Browser MySQL Database
  • Likes 2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Hamad Sultan wrote:The second one as the time passes the database becomes heavier i.e after 1 to 2 years.



But let's not worry about the abstract problem. Let's do a back-of-the-envelope calculation instead, to see what you're dealing with.

You said that the operator enters data about a gas turbine once an hour. Now in a year there are, um, 24 x 365 hours, that's about 9000 hours. I'm assuming here that the turbine is running 24 hours a day or at least that's the plan. Maybe it will be offline sometimes but we're only after an approximation. So you are adding 9000 rows per year to your database. This is an extremely small database and even after running for a century it's still going to be a small database.

But maybe you have ten gas turbines at the plant? Then you're adding 90000 rows a year. Still very small.



* "Back-of-the-envelope calculation": This is an old engineering term for a rough estimate. Back in the old days you didn't have e-mail, you got paper letters sent to you in envelopes. The empty envelopes could be discarded but often sat around on your desk. If you wanted to do a small calculation, you didn't have any kind of mechanized calculation device except maybe a slide rule. So you would take your pencil and scribble down the calculation on the back of one of those empty envelopes.
 
Hamad Sultan
Ranch Hand
Posts: 37
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello Friends;

Thanks for help on each step. I browse the internet and search "How to transfer data from one database to another bu using JAVA"

I found some ideas at different forums that shows it is very easy task. Mostly mention the following statement

INSERT INTO database2.table2 SELECT * from database1.table1

But i do not understand how i execute it?

Now i done first step if internet connection not active then Connection Manager return me the connection of localhost and record save on it. If internet available then record save to domain server and Connection Manager return the Domain server Conn for sql. I also generate the ID as a extra column to search out which record are missing from domain server and present in the localhost DB.

Now i am thinking the easiest way or logic that  the records at localhost now should be transferred to domain server. I have an idea:

1. Fetch all the records from localhost into the JTable or array(what you are recommended) and also fetch records as well from domain server. Then Select those records which are missing and insert into from localhost to domain server. But it looks a complicated logic. am i right?

Please give me the best solution for it? and how the above mentioned query run which i read in the forum.

Regards

 
Sheriff
Posts: 7056
184
Eclipse IDE Postgres Database VI Editor Chrome Java Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I would try to do it all in SQL, but I don't know how in MySQL.  Maybe try googling mysql merge table from remote to local.
 
Tim Holloway
Saloon Keeper
Posts: 21975
150
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Knute Snortum wrote:I would try to do it all in SQL, but I don't know how in MySQL.  Maybe try googling mysql merge table from remote to local.



You can do a mysqdump to get SQL, scp or rsync (recommended) the dumpfile from remote to local, then use the mysql command-line utility to apply the dumped SQL.

But first, I would recommend checking out remote replication options. No need to set up fiddly transfer processes if you can tell mysql to do it all automatically.

Note that remote replication options may differ signicantly between Oracle MySQL and MariaDB.
 
Hamad Sultan
Ranch Hand
Posts: 37
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello Friends;

" Now i am thinking the easiest way or logic that  the records at localhost now should be transferred to domain server. I have an idea:

1. Fetch all the records from localhost into the JTable or array(what you are recommended) and also fetch records as well from domain server. Then Select those records which are missing and insert into from localhost to domain server. But it looks a complicated logic. am i right?

Please give me the best solution for it? and how the above mentioned query run which i read in the forum."

Still waiting for advise of my question?

Regards
 
Knute Snortum
Sheriff
Posts: 7056
184
Eclipse IDE Postgres Database VI Editor Chrome Java Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The only reason you would use a JTable is if you were going to create a GUI using Swing and select the records by hand.  JTable is a GUI element, not a data structure.

How do you determine which rows are missing in the local table?  Do the rows have IDs?  Are those IDs the same in the local table and the remote?  Does the local table use some other kind of key, like two fields in combination are unique?

Once you know the details of the task you are trying to accomplish, then you can talk about data structures.

 
Hamad Sultan
Ranch Hand
Posts: 37
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello Knute Snortum/ Friends;

Hope you will fine. Yes i am developing Database application with GUI by using JAVAx.Swing.

Yes i have a column ID with Unique and auto increment in both Local and domain database. When internet connection available then all the insert data put into domain other wise it save on Localhost.

Now Please give me the best solution how i manage the data in a efficient way? I want the things done from the front end means(From GUI )


Regards
 
Paul Clapham
Marshal
Posts: 25452
65
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
That really doesn't give us much information about what you need to know. All we know so far is that you have a JTable. And that you want to update a database.

We don't know what's in the JTable or how it relates to the database. We don't know how the data gets into the JTable either.

As for updating one database or the other database depending on whether there's an internet connection, I think that's a bad idea. First of all there is no method in the standard API which allows you to determine whether there is an internet connection or not. That's because there is no way to determine that easily. You might not be able to connect to the remote database; that might be because there's no internet connection, or it might be for some other reason. And anyway if you're updating a local database sometimes and a remote database at other times, you're going to end up with two inconsistent databases, which requires you to rely on database mirroring, or replication, to clean up the resulting mess. But the aim of mirroring is that both databases will be current, and that if access to one version fails then you can fall back to using the other version. This sort of setup suggests that you have two remote databases which are automatically synchronized by the mirroring software, and if one goes down then you can switch immediately to the other one. And it also suggests that you are only updating one of the two at any particular time. Applying updates to both databases could confuse the mirroring process.

If it were my system and it was to be installed in a place with an unreliable internet connection, I wouldn't put my database on the other end of that connection. I would just use a local database. It's true that a database in the cloud means that you don't have to manage it or back it up etc, that's all done by the hosting company. But when you decide to have a local database to mirror it, those advantages all disappear. In that case, why not just the simple local database?

But I would first need to know the risks involved in the internet connection. Does it go down every day for several hours? Or once every month for a few minutes? You can't really produce good solutions to a problem if you don't know whether it exists and how bad it is. It may be that this internet connection issue isn't something to be dealt with at the beginning of the design.
 
Hamad Sultan
Ranch Hand
Posts: 37
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello Paul;

Good day. Yes welcome to your advise. Actually why i want to use Local and cloud database. The reason is that;

1. Localhost database just save the data and generate monthly reports within an office. It seems good and i will do it .
2. Why i need cloud database ? Our Manger office sit another city and he wants to see some parameters on its desktop like what turbine is running and how much flow is pass from the selected Compressor Station.

I have 15 compressor stations and Each compressor station have separate  application and save the record at localhost individually.  

Now our manager wants to see the data which station he choose to see some parameters.


That is the story now what is your opinion how i mange it. Please forget JTable for this time.


Regards
 
Tim Holloway
Saloon Keeper
Posts: 21975
150
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I think that in that case, I'd dump the data to SQL, upload it to the cloud server, and then run the mysql command to import the SQL.
 
Hamad Sultan
Ranch Hand
Posts: 37
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello Tim Holloway;

How it is done from the front end means from the GUI? " I think that in that case, I'd dump the data to SQL, upload it to the cloud server, and then run the mysql command to import the SQL. "

Please give me an example.

Regards
 
Tim Holloway
Saloon Keeper
Posts: 21975
150
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The GUI stores data in the database. You could then schedule a service to run at regular intervals to do the upload. This could be a simple cron job (Unix/Linux) running a shell script or a Windows Scheduler event running a batch command file. The commands in the file would be a mysqldump of the records to that are be sent to a temporary file, followed by an upload to the cloud server using a service such as rsync or scp. Once uploaded, a script running on the cloud server could then do the MySQL load.

Don't expect us to actually develop all of that for you. We're in the helping business, not programmers-for-free. But this sort of thing is done all the time, so there are plenty of people who can do the work for you if you can't figure it out yourself.
 
Paul Clapham
Marshal
Posts: 25452
65
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Hamad Sultan wrote:1. Localhost database just save the data and generate monthly reports within an office. It seems good and i will do it .
2. Why i need cloud database ? Our Manger office sit another city and he wants to see some parameters on its desktop like what turbine is running and how much flow is pass from the selected Compressor Station.

I have 15 compressor stations and Each compressor station have separate  application and save the record at localhost individually.  

Now our manager wants to see the data which station he choose to see some parameters.



So each station has a local database containing only the data for that station, and there's a central database which contains the data for all of the stations? Then yeah, that's different than mirroring. So what Tim says is the general idea. You have to decide if you want to send only new records to the central database, or whether you want to send the whole local database every time. There are pros and cons to both approaches.
 
Hamad Sultan
Ranch Hand
Posts: 37
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Dear Paul/Friends;

Now i conclude that the only those parameters of turbine send to Domain database which is required to manager to evaluate it.

So guide me what is the best option between following:
1. first save the data to the excel sheet and when internet connection available send it to domain database after successfully the records on the excel sheet will be deleted and save new records ?
2. Second one is save to localhost and then send to domain and delete that records after insertion.

What you guys suggest?

Regards
 
Paul Clapham
Marshal
Posts: 25452
65
Eclipse IDE Firefox Browser MySQL Database
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Hamad Sultan wrote:So guide me what is the best option between following:
1. first save the data to the excel sheet and when internet connection available send it to domain database after successfully the records on the excel sheet will be deleted and save new records ?
2. Second one is save to localhost and then send to domain and delete that records after insertion.



Excel???

I would only use Excel if the system was going to send a spreadsheet to the manager for them to examine. Otherwise, using Excel as a place to store data is just an awful thing to do. A horrible programming task. I have written code to do that but I would only do it if the spreadsheet was coming from a person and its data had to be put into the system, or if data had to be extracted from the system and sent to a person in a human-readable form.

To store data temporarily which needs to be inserted into another database, store it in a database.
 
Every plan is a little cooler if you have a blimp. And a tiny ad.
Two software engineers solve most of the world's problems in one K&R sized book
https://coderanch.com/wiki/718759/books/Building-World-Backyard-Paul-Wheaton
    Bookmark Topic Watch Topic
  • New Topic