• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Help!! - java.lang.OutOfMemoryError

 
Ranch Hand
Posts: 42
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I am trying to generate an XML string from a table with 13160 rows in the result set.
I checked the archives and found this helpful discussion: http://www.javaranch.com/ubb/Forum3/HTML/000465.html
So I tried to implement the same solution, but it's not working.
The code itself is somewhat long. I've posted it at:
http://y42.briefcase.yahoo.com/bc/iamsanjayonly/vwp2?.tok=bc J5XF7AtegBHF8g&.dir=/Code&.dnm=AllDataXML.java&.src=bc
Here are some relevant snippets:
 
Ranch Hand
Posts: 625
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Which jdk are you using? I used to run out of memory when I was playing with graphics- copying all the pixels into a multi-dimensional array and then manipulating the picture. I found if the picture had too many pixels, then at times I'd run out of memory. I haven't looked at the code, as I haven't the time, but you might want to go through it and clean things up and when you're done using resources let the garbage collector know that you don't need it anymore by assigning the reference to null. This should help.
 
Sanjay Anand
Ranch Hand
Posts: 42
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks for your reply, Sean.
I have tried to clean up the code as best I can. I was hoping for some more specific suggestions. I have, for instance, done the resetting to null.
Anyway, I will keep working on it.
Sanjay
 
Wanderer
Posts: 18671
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Ye gods! What do you need all those spaces for in front of every line? Four spaces per level is plenty - eight just forces your code over off the right side of the screen. And why isn't it indented consistently? This is a pet peeve for me - making your code easily readable for others is key to getting people to help. Of course this is much better than some I've seen but still...
> I am trying to generate an XML string from a table with
> 13160 rows in the result set.
Well, there's your problem right there.
OK, the simplest possible fix - consult the documentation for the JVM you're using to see if you can set the max allocated memory to a higher value. For example, for JDK 1.3 on Windows the documentation is here. Use java -Xmx 128m to set max memory to 128 MB - the default is 64 MB if you don't set it otherwise.
While you're at it, set the -verbose:gc option so you can get a printout of the JVM's attempts to reclaim memory. This is very useful in figuring out where & when memory is being consumed by a program.
At a guess though, I strongly suspect that the best place for you to reduce the memory usage of your program is the StringBuffer. Sure, it's better than using Strings, but your XML string is going to be pretty big overall. Is it necessary to store the whole thing in memory? What are you going to do with it afterwards - write it to a file? If so, consider replacing the StringBuffer with a FileWriter wrapped in a BufferedWriter. That way each time you read a new row from the ResultSet, you can basically dump the information straight to a file rather than taking more and more memory in a StringBuffer.
Even if you don't need to write the XML to a file in the long run, and you really do need a full-size String eventually to apss to some other method, consider writing the XML to a file as an intermediate step, and then read the file later to create the String from that. The problem is that the ResultSet is already going to take up a large amount of memory - if you build the StringBuffer at the same time, then that will be two large amoounts of memory being allocated at the same time. If you delay creation of the String until after you've discarded the ResultSet, you can substantially decrease the total memory needed.
The other thing to look at is the ResultSet. Is it necessary to read all 13160 rows at once? That probably means your ResultSet will have to be big enough to store all the data at once. (Although it's possible some drivers find ways around this; I dunno.) Is there a way of breaking the ResultSet into smaller groups? For example, if you the data represent employee records which you're accessing by last name, you could first query for all names beginning with "A" and process those results, then "B", etc. It really depends how your data are organized, but there may well be a way to do something like this.
Good luck...
 
Sanjay Anand
Ranch Hand
Posts: 42
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks, Jim, for the slew of helpful suggestions.
I wanted to do the writefile solution but my manager thought that file reading and writing would create its own performance problems. I don't agree but who can argue with managers
I'll see if setting the max memory allocation does the trick (the platform is Solaris, by the way). And, if not, I'll just write to file as I go.
Sorry about the spacing of the code. I just find it easier to hit the tab and pasting it on the ubb creates its own problems. But I'll edit next time before I post.
Thanks again,
Sanjay
 
Jim Yingst
Wanderer
Posts: 18671
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Well, if you have enough memory, creating the StringBuffer directly in memory will be quicker than writing to a file and then reading it later. But I'm betting that the main speed limitation will be the database access, and the time spent reading and writing files won't really be noticeable compared to the time spent accessing the DB. Furthermore, if you don't have enough memory, then it doesn't matter if one solution is faster if it ends in an OutOfMemoryError. Also, if your JVM memory usage is large enough, it may not all fit into RAM anyway, and the system will be swapping pages from disk, which takes time. So increased memory usage ends up slowing things down anyway. I'd try it the manager's way first, then when it doesn't work, try the better way, and once you get it working smoothly you can just explain to the manager that you tested both methods, and went with the one that works better.
 
Does this tiny ad smell okay to you?
a bit of art, as a gift, the permaculture playing cards
https://gardener-gift.com
reply
    Bookmark Topic Watch Topic
  • New Topic