File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes Servlets and the fly likes Issue in reading large file on server, Servlet request time out Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Spring in Action this week in the Spring forum!
JavaRanch » Java Forums » Java » Servlets
Bookmark "Issue in reading large file on server, Servlet request time out" Watch "Issue in reading large file on server, Servlet request time out" New topic
Author

Issue in reading large file on server, Servlet request time out

yogesh goswami
Greenhorn

Joined: May 26, 2011
Posts: 6
I am trying to upload large text file. Text file size would be around 100 MB (around 1500000 lines). I have to process each line from the file and have to store it into DB.
I am reading file on server(in a servlet ) using below code.


----- Above code works fine if I process file with around 10000 records, but when I process large file I am getting out of memory issue. I want to split large file into small file on server , store it in temp directory and wanted to load each file but when I am splitting the file sometimes half of the last line in file will present in one file and remaining line will present in another file first line(Example – File1-> ‘How are you’, File1->’How’ File2->’Are You’).
Also processing of file take lot of time and servlet request is getting timed out.

Please suggest any solution for this problem. If possible please give some example.
Thanks in advance for your help.
Yogesh…
Amit Ghorpade
Bartender

Joined: Jun 06, 2007
Posts: 2716
    
    6

Firstly from the design perspective, I think that the file upload servlet should not be doing the DB update, have a separate utility class for that to keep things simple.
If your problem is only the time-out, putting a small AJAX to simply META-REFRESH the page should solve the problem.


SCJP, SCWCD.
|Asking Good Questions|
Amit Ghorpade
Bartender

Joined: Jun 06, 2007
Posts: 2716
    
    6

Also take a look at http://www.coderanch.com/t/496463/Servlets/java/EE-Session-Expires-Large-File
yogesh goswami
Greenhorn

Joined: May 26, 2011
Posts: 6
Thanks Amit. I will try with 'AJAX META-REFRESH'

Could you please suggest something to split file on server, I am able to split files but issue is , file splitting using size instead of ’ No. of lines’ and because of that last line is also splitting((Example – File1-> ‘How are you’, File1->’How’ File2->’Are You’)).
I am placing file splitting code please suggest if anything can be done?

Jayesh A Lalwani
Bartender

Joined: Jan 17, 2008
Posts: 2402
    
  28

You don't really need to split file. You are adding unnescary complication to the code. You can read lines in batches (of let's say 1000 lines each) process them, then read the next batch
William Brogden
Author and all-around good cowpoke
Rancher

Joined: Mar 22, 2000
Posts: 12803
    
    5
Are you required to use a browser on the client side?

If not, a custom Java client would be much more flexible - for example it could send batches of 100 lines and keep track of how many lines have been processed so that you could resume processing if the connection was broken.

Seems to me the chance of a connection problem in the course of sending 100MB with DB processing on the server must be pretty substantial.

Bill
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Issue in reading large file on server, Servlet request time out