aspose file tools*
The moose likes Performance and the fly likes Java heap memory error Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Performance
Bookmark "Java heap memory error" Watch "Java heap memory error" New topic
Author

Java heap memory error

gee uma
Greenhorn

Joined: Oct 12, 2013
Posts: 2
Hi,

I need to process 10000 xml files and verify and insert the data into database.
I am loading all the files in the file object and iterating one by one. I am getting the memmory issue.
Please advise how to handle this?

thanks
gee
Jayesh A Lalwani
Bartender

Joined: Jan 17, 2008
Posts: 2376
    
  28

Load them one by one instead of all 10K in one go
Salil Vverma
Ranch Hand

Joined: Sep 06, 2009
Posts: 255

Hey Jayesh,

As you are processing too many files, you might want to make sure that at a time only certain number of files (probably 2-3) are loaded.

If you want to get optimum performance in this file loading & processing activity. I will suggest to load and process file using different threads. This way, in the mean time, when one thread would be processing the file other thread would be loading the other file. As soon as processing of one file is complete other file would have got loaded and available for processing.

To make sure that at a time only max certain number of files (probably 2-3) are loaded and loading and processing thread does not conflict ,you can use blocking queue.


Regards
Salil Verma
William Brogden
Author and all-around good cowpoke
Rancher

Joined: Mar 22, 2000
Posts: 12785
    
    5
You left out very important information.

When you "process" a file do you parse it to a DOM or as a series of SAX events?

DOM processsing is of course slower and takes more memoy but would profit from the multiple Thread approach.

Are there some processing requirements that require more than one XML document be present at one time?

(Note that I am assuming that each "xml file" is a complete XML document - is that true?)

Bill
kri shan
Ranch Hand

Joined: Apr 08, 2004
Posts: 1373
If you use DOM parser, 10000 xml files needs huge memory.
gee uma
Greenhorn

Joined: Oct 12, 2013
Posts: 2
Thanks all of you.

One XMl file is complete xml document, for one record creation.
How to load one file at a time, i have around 10000 files in one folder.
Need to pick one by one file and process the file.
Currently using folder.listfiles, which loads all the files, which causing the memory issue.

Thanks
gee
Winston Gutkowski
Bartender

Joined: Mar 17, 2011
Posts: 7795
    
  21

gee uma wrote:Currently using folder.listfiles, which loads all the files

Erm...no it doesn't. It returns you a List of the Files, which is quite different.

which causing the memory issue.

It is? Are you sure? The most files I've ever dealt with in a single directory is around 300,000 - which should be peanuts to a normal Java program. And believe me, it caused a LOT more problems than just OOM issues.

And if you are dealing with those sorts of quantities, you should be talking to your administrator, because it sounds like someone is using your file system as a database.

Also: You still haven't told us exactly what the problem is. Why don't you copy and paste the precise error you're getting? Just the first dozen or so lines will do.

Winston

Isn't it funny how there's always time and money enough to do it WRONG?
Articles by Winston can be found here
rohit chavan
Ranch Hand

Joined: Oct 08, 2010
Posts: 132

Winston Gutkowski wrote:

Also: You still haven't told us exactly what the problem is. Why don't you copy and paste the precise error you're getting? Just the first dozen or so lines will do.

Winston


Also, you can put your code here ( a snippet of how are you doing it) , and then it would be a lot easier to tackle the problem.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Java heap memory error