• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Serialization and Large Files

 
Ranch Hand
Posts: 46
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I have this crazy problem I am working on. We've got a proram going that works for small and fair-to-middling sized files but when we try to run it on large files it crashes with an outOfMemory error. I traced the problem back to a section of code that writes to a very large data structure, specifically a vector of vectors of matrices (an imported structure). I'm doing a complicated formula on a whole lot of numbers so I have to do a lot of multiplying everything by everything else then sorting the results. Thus the vectors of vectors. But my vector of vectors is getting quite large and causing me problems no matter how much memory I allocate to to program when I run it.

So here's my question: Can I serialize this puppy out to a file bit by bit? What if I wrote out each inner vector as I was done with it? How would that even work? I know how to write one vector to its own little file and retrieve it again but I wouldn't know how to do that without making a whole lot of files and retrieving them all and putting them back into the larger structure again when I needed the data, which we already know is too large.

Any ideas about how to handle a large amount of data without using all your RAM would be very helpful.

Thanks,
Rebecca
 
Ranch Hand
Posts: 53
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
It sounds like you need a database to store your Vectors' data. Whether you store it in flat files as serialized objects, or you put the relevant data into a database, you need to have some sort of naming/identification scheme to figure out what is where. You could write each object to its own file, and keep a list in memory that lists what Vector is stored in what file-- a HashMap is good for this - the key is the "Name" of your Vector and the value is the path (File.getCanonicalPath()) of the output file.

Other general suggestions: use an ArrayList instead of a Vector, if possible, since it will take less memory. Are you using the java parameters -Xms1024m -Xmx1024m to increase the size of your Java Virtual Machine (use a smaller number if you do not have 1MB of RAM).

Hope this helps,
Joseph
 
Ranch Hand
Posts: 502
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Before you try anything exotic like swapping your memory structures to disk, did you try playing around with -Xmx option of the java command-line?

Basically. java has an upper limit of how much memory it can allocate.(I forget what's the default value for the upper limit). You can change the upper limit using the -Xmx option
 
Rebecca Witmer
Ranch Hand
Posts: 46
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Yes in JBuilder's Runtime Configuration > VM Parameters I have "-mx128m" That's up from the default 64. I tried having it be bigger but it just takes a longer time to crash.

So I think the best recommendation so far is to write each inner Vector to its own file then keep a list of where everything is when you want to get at it later, right?
 
Ranch Hand
Posts: 1873
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Rebecca,

Yup. It looks like you would have to go with the idea you are thinking to avoid lot of data loaded in memory at the same time..

Regards,
Maulin
 
Die Fledermaus does not fear such a tiny ad:
a bit of art, as a gift, the permaculture playing cards
https://gardener-gift.com
reply
    Bookmark Topic Watch Topic
  • New Topic