aspose file tools*
The moose likes Performance and the fly likes A[nother] question about tracking down memory leaks Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Performance
Bookmark "A[nother] question about tracking down memory leaks" Watch "A[nother] question about tracking down memory leaks" New topic
Author

A[nother] question about tracking down memory leaks

Stuart Rogers
Ranch Hand

Joined: Oct 02, 2008
Posts: 139
I've crafted a program that reads a tab-delimited text file a-line-at-a-time and for each line attempts to insert/update some rows in a few MySql tables.
Here's the corresponding method - it works great until dying at the 2050-ish record mark with a

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

message.

I can watch the memory usage growing using the value of highwater printed out every 100 records.

As of now the program runs locally on a server. In the future I'll be transmorgifying it into a servlet.
The default -Xmx setting on the server is 64M

I cannot know in advance how much memory I'll need (ie how big users' files will be) so cannot rely on just upping the -Xmx value.
I'd much rather fix the leak. Any ideas?





Thanks in advance,

Still-learning Stuart
Siva Masilamani
Ranch Hand

Joined: Sep 19, 2008
Posts: 385
Make sure to close the database connection.I don't see one.


SCJP 6,SCWCD 5,SCBCD 5

Failure is not an option.
Stuart Rogers
Ranch Hand

Joined: Oct 02, 2008
Posts: 139
Yes, of course, I close the connection at the end of the main() method. The main() method contains the declaration for the connection and the call to the process_student_file() method.

Should I 'recycling' all my sql-related declarations (sqlcmd, rs, stmt, etc) by instantiating them inside the while-loop like so



I think declaring them once, outside the while-loop, would be more efficient, but I could very well be wrong.

Still-learning Stuart
Siva Masilamani
Ranch Hand

Joined: Sep 19, 2008
Posts: 385
I don't think so to put all the declaration inside a while loop.

But it would be highly advisable to close the resultset and statement after their use.

So try closing all the resultset,statements after their use.

That may be helpful because

The JavaDoc for the Connection.close() method starts with "Releases this Connection object's database and JDBC resources immediately instead of waiting for them to be automatically released."

Therefore, if you close() the connection, all the achquired objects, such as Statements and ResultSets will be closed.

However, if you use connection pool, the close() method returns the connection to the pool and doesn't actually close the connection. In this case dependant objects may be left open
Stuart Rogers
Ranch Hand

Joined: Oct 02, 2008
Posts: 139
Your suggestion helps by a factor of four - that is to say, closing the various PreparedStatements and ResultSets causes memory to be consumed 1/4 as fast as before.
So that plugs most of the leaks but not all - any other suggestions?

Thanks so far,

Still-learning Stuart
Tim Holloway
Saloon Keeper

Joined: Jun 25, 2001
Posts: 16136
    
  21

Stuart Rogers wrote:Your suggestion helps by a factor of four - that is to say, closing the various PreparedStatements and ResultSets causes memory to be consumed 1/4 as fast as before.
So that plugs most of the leaks but not all - any other suggestions?

Thanks so far,

Still-learning Stuart


That's one of the best reasons for using the Spring Framework. It makes it easy to create code that automatically releases connections even when you have database exceptions, By wrapping the database functions, it keeps you from having to code all the tedious, repetitive, and error-prone error handling stuff.
Siva Masilamani
Ranch Hand

Joined: Sep 19, 2008
Posts: 385
You could define statements like this outside while loop



Inside the loop just set its parameter and execute it.

Also insted of creating multiple statements and resultsets just create one of each and reuse the same for the other queries.

But make sure to close them before every reuse.

E.g

stmt = this.conn.prepareStatement( sqlcmd);
after executing this statement and got the deired reult from resultset,close them
stmt.close()
rs.close()

Then for the query2
stmt = this.conn.prepareStatement( other query);
after executing this statement and got the deired reult from resultset,close them
stmt.close()
rs.close() and so on.

I am not sure how much this would help you.

This is just my idea.Try it

Stuart Rogers
Ranch Hand

Joined: Oct 02, 2008
Posts: 139
Thanks for the tips. Since each separate sql cmd never varies I moved them out of the loop entirely and moved the
statement-compiling to just inside the outermost try{ block . I also close all the statements at the end and the two Result Sets variables
at the end of their try{ blocks . Like this:



Using the heap measurements I can see where my total memory usage now never goes above the initial alloted amount of 20647936 bytes. So the problem
sure seems to be solved.

My thanks to everyone who chimed in!! I am always amazed at the collective wisdom here at the ranch.

Thanks again,

CASE CLOSED

Still-learning Stuart

 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: A[nother] question about tracking down memory leaks