I need to process 10000 xml files and verify and insert the data into database.
I am loading all the files in the file object and iterating one by one. I am getting the memmory issue.
Please advise how to handle this?
As you are processing too many files, you might want to make sure that at a time only certain number of files (probably 2-3) are loaded.
If you want to get optimum performance in this file loading & processing activity. I will suggest to load and process file using different threads. This way, in the mean time, when one thread would be processing the file other thread would be loading the other file. As soon as processing of one file is complete other file would have got loaded and available for processing.
To make sure that at a time only max certain number of files (probably 2-3) are loaded and loading and processing thread does not conflict ,you can use blocking queue.
If you use DOM parser, 10000 xml files needs huge memory.
Joined: Oct 12, 2013
Thanks all of you.
One XMl file is complete xml document, for one record creation.
How to load one file at a time, i have around 10000 files in one folder.
Need to pick one by one file and process the file.
Currently using folder.listfiles, which loads all the files, which causing the memory issue.
gee uma wrote:Currently using folder.listfiles, which loads all the files
Erm...no it doesn't. It returns you a List of the Files, which is quite different.
which causing the memory issue.
It is? Are you sure? The most files I've ever dealt with in a single directory is around 300,000 - which should be peanuts to a normal Java program. And believe me, it caused a LOT more problems than just OOM issues.
And if you are dealing with those sorts of quantities, you should be talking to your administrator, because it sounds like someone is using your file system as a database.
Also: You still haven't told us exactly what the problem is. Why don't you copy and paste the precise error you're getting? Just the first dozen or so lines will do.
Isn't it funny how there's always time and money enough to do it WRONG?
Artlicles by Winston can be found here