I have a problem with memory usage on my application. A short introduction to my app: The application collects data from a number of MySQL servers (we call these dbs external) and puts the data into one single database (MySQL, we call this db local). The collection should be done periodically. These two classes are of intrest: - Fetcher. Upon creation of this class it sets up a connection to one of the external servers. This connection will be open until the application terminates. Fetcher has a method, fetch(), which executes a query and gets a ResultSet from the database.
- FetcherThread extends Thread. Dont think this will need any explanation.
Since Fetcher.fetch only is accessed by one thread (there is one FetcherThread and one Fetcher for each external database) there shouldnt be any special real time requirements here. Everything concerning the local database can be ignored at this point, it has nothing to do with my problem (same shit happens when qh.updateAlarms() is commented).
So to my problem. The code above eats memory like crazy (for one extrenal database, i.e. one Fetcher and one FetcherThread, 1mb each 10 minutes), even if I just execute the query and then ignore the ResultSet. The result returned from the query are at max 10-15 rows. I've tried setting everything to null in the end of Fetcher.fetch(), calling alarms.removeAllElements(), replacing Vector with LinkedList and explicitly call System.gc() but with no success.
I did some profiling (with TPTP), and it seems like the GC misses to collect about 50% of the vectors elemets (instances of Alarm). The Alarm class simply is a container, and has no references to other objects. But as I mentioned above, even with this part removed it leaks memory.
btw. the constant SMTDataCollector.ALARM_FETCH_DELAY = 3000, i.e. the thread will sleep for 3 seconds.
Any thoughts about this are very appriciated.
Joined: Aug 31, 2006
System.gc() just suggests the garbage collector run. It may or may not run. Also when it runs it doesn't have to collect all eligible objects.
At first glance I would say you aren't calling close() on your Statement or ResultSet so these will not be closed until the garbage collector collects them.
I'd call close() or your statement before exiting fetch().
Joined: Jan 15, 2006
Besides the good comments made about closing stuff I have a few other questions: 1. Where does con come from? 2. Do all dbs have the same structure and that is why you can just pass different connections and collect data, I assume? 3. Where is sequenceNbr defined? I know you said only one thread per fetcher but putting some printouts or logs would be interesting to see. 4. If gc is not collecting 50% of Vector elements it maybe doing so for optimization lets say it was keeping a pool and reusing values, are this strings in binary??, as you do not seem to be reassigning individual elements somewhere check your qh.updateAlarms() code. 5. If nothing works start removing/rewriting code line by line, you are probably doing that , and isolate the problem inside or out of a particular block. 6. Since you have multithreading make sure your assumptions on what is happening are actually correct.