File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Java heap memory error

 
gee uma
Greenhorn
Posts: 2
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,

I need to process 10000 xml files and verify and insert the data into database.
I am loading all the files in the file object and iterating one by one. I am getting the memmory issue.
Please advise how to handle this?

thanks
gee
 
Jayesh A Lalwani
Rancher
Pie
Posts: 2756
32
Eclipse IDE Spring Tomcat Server
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Load them one by one instead of all 10K in one go
 
Salil Vverma
Ranch Hand
Posts: 255
Hibernate Oracle Spring
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hey Jayesh,

As you are processing too many files, you might want to make sure that at a time only certain number of files (probably 2-3) are loaded.

If you want to get optimum performance in this file loading & processing activity. I will suggest to load and process file using different threads. This way, in the mean time, when one thread would be processing the file other thread would be loading the other file. As soon as processing of one file is complete other file would have got loaded and available for processing.

To make sure that at a time only max certain number of files (probably 2-3) are loaded and loading and processing thread does not conflict ,you can use blocking queue.

 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13044
6
  • 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You left out very important information.

When you "process" a file do you parse it to a DOM or as a series of SAX events?

DOM processsing is of course slower and takes more memoy but would profit from the multiple Thread approach.

Are there some processing requirements that require more than one XML document be present at one time?

(Note that I am assuming that each "xml file" is a complete XML document - is that true?)

Bill
 
kri shan
Ranch Hand
Posts: 1453
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
If you use DOM parser, 10000 xml files needs huge memory.
 
gee uma
Greenhorn
Posts: 2
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks all of you.

One XMl file is complete xml document, for one record creation.
How to load one file at a time, i have around 10000 files in one folder.
Need to pick one by one file and process the file.
Currently using folder.listfiles, which loads all the files, which causing the memory issue.

Thanks
gee
 
Winston Gutkowski
Bartender
Pie
Posts: 9442
49
Eclipse IDE Hibernate Ubuntu
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
gee uma wrote:Currently using folder.listfiles, which loads all the files

Erm...no it doesn't. It returns you a List of the Files, which is quite different.

which causing the memory issue.

It is? Are you sure? The most files I've ever dealt with in a single directory is around 300,000 - which should be peanuts to a normal Java program. And believe me, it caused a LOT more problems than just OOM issues.

And if you are dealing with those sorts of quantities, you should be talking to your administrator, because it sounds like someone is using your file system as a database.

Also: You still haven't told us exactly what the problem is. Why don't you copy and paste the precise error you're getting? Just the first dozen or so lines will do.

Winston
 
rohit chavan
Ranch Hand
Posts: 132
Hibernate Java Oracle
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Winston Gutkowski wrote:

Also: You still haven't told us exactly what the problem is. Why don't you copy and paste the precise error you're getting? Just the first dozen or so lines will do.

Winston


Also, you can put your code here ( a snippet of how are you doing it) , and then it would be a lot easier to tackle the problem.
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic