File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes JDBC and the fly likes Writing object to the Database using Externalizable Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Databases » JDBC
Bookmark "Writing object to the Database using Externalizable" Watch "Writing object to the Database using Externalizable" New topic
Author

Writing object to the Database using Externalizable

Joel Sander
Ranch Hand

Joined: Jan 09, 2009
Posts: 38
For performance reasons, I just swapped from writing objects to my JDBC using Serializable to using Externalizable. However, I found that as I write objects to the database, the database grows in size (both in memory and on the hard drive). If I close my connection to the database, the memory is cleared and the hard drive space used shrinks from hundreds of MBs to a few MBs:

java -jar "%DB_HOME%\lib\derbyrun.jar" server shutdown


If I instead continue to add objects, I eventually get an OutOfMemoryError:


c:\Program Files (x86)\Apache Software Foundation\Tomcat 5.5\webapps\Bball\WEB-I
NF\classes>java -jar "c:\Program Files (x86)\Sun\JavaDB\lib\derbyrun.jar" server start
Security manager installed using the Basic server security policy.
Apache Derby Network Server - 10.4.2.0 - (689064) started and ready to accept connections on port 1527 at 2009-07-12 03:26:03.708 GMT
Exception in thread "DRDAConnThread_4" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
at java.lang.StringBuffer.append(StringBuffer.java:224)
at java.io.StringWriter.write(StringWriter.java:95)
at java.io.PrintWriter.write(PrintWriter.java:412)
at java.io.PrintWriter.write(PrintWriter.java:429)
at java.io.PrintWriter.print(PrintWriter.java:559)
at java.io.PrintWriter.println(PrintWriter.java:695)
at org.apache.derby.iapi.error.ErrorStringBuilder.appendln(Unknown Source)
at org.apache.derby.iapi.services.context.ContextManager.cleanupOnError UnknownSource)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown ource)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)


Do I need to do something like:



inside the writeExternal() method? Or is there something else, that needs to happen to prevent this exception?
Jason Cohen
Smart Bear Support
Greenhorn

Joined: Jul 08, 2009
Posts: 6
Note that the error is happening as it is trying to format an error message.

That is, the real error is being obscured. It's trying to form the error and running out of memory, probably because it's doing something like dumping a query that itself is massive.

The short-term fix is to increase the amount of memory allowed to Java. You're probably at the 64MB default client limit!

Then you can read the error message, then you can figure out the real problem.

Or it might just be that you need 100MB of RAM instead of 64MB, so just increasing memory is itself enough.


Jason Cohen
Founder, Smart Bear Software
http://blog.asmartbear.com
Joel Sander
Ranch Hand

Joined: Jan 09, 2009
Posts: 38
Thanks for the reply.

Using the task manger I found that the JDBC was using about 1GB of memory which doesn't seem right since when I closed it, the hard drive space it used dropped to a few tens of MBs..

I've since found that using out.flush() at the end of each class's writeExternal method has decreased the memory usage to about 500MBs...But that seems very high for a database holding about 20MBs of data...If I shut down the database are then reopen it and start my processing updating it & writing to it, it's memory usage pretty quickly grows back to 500MBs.

A major portion of the database access is updating BLOBs (that combined take about 3-4MBs of disk space). I've noticed that as the updating process runs, it gets faster. Could it be that the database is buffering all the structures that are written to it? If so, is it reasonable for the structures to be this much larger? And if my database grows, won't the buffering cause more out of memory exceptions?
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Writing object to the Database using Externalizable