This week's book giveaway is in the Clojure forum.
We're giving away four copies of Clojure in Action and have Amit Rathore and Francis Avila on-line!
See this thread for details.
Win a copy of Clojure in Action this week in the Clojure forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Issue in reading large file on server, Servlet request time out

 
yogesh goswami
Greenhorn
Posts: 6
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am trying to upload large text file. Text file size would be around 100 MB (around 1500000 lines). I have to process each line from the file and have to store it into DB.
I am reading file on server(in a servlet ) using below code.


----- Above code works fine if I process file with around 10000 records, but when I process large file I am getting out of memory issue. I want to split large file into small file on server , store it in temp directory and wanted to load each file but when I am splitting the file sometimes half of the last line in file will present in one file and remaining line will present in another file first line(Example – File1-> ‘How are you’, File1->’How’ File2->’Are You’).
Also processing of file take lot of time and servlet request is getting timed out.

Please suggest any solution for this problem. If possible please give some example.
Thanks in advance for your help.
Yogesh…
 
Amit Ghorpade
Bartender
Posts: 2851
10
Fedora Firefox Browser Java
  • 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Firstly from the design perspective, I think that the file upload servlet should not be doing the DB update, have a separate utility class for that to keep things simple.
If your problem is only the time-out, putting a small AJAX to simply META-REFRESH the page should solve the problem.
 
Amit Ghorpade
Bartender
Posts: 2851
10
Fedora Firefox Browser Java
 
yogesh goswami
Greenhorn
Posts: 6
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks Amit. I will try with 'AJAX META-REFRESH'

Could you please suggest something to split file on server, I am able to split files but issue is , file splitting using size instead of ’ No. of lines’ and because of that last line is also splitting((Example – File1-> ‘How are you’, File1->’How’ File2->’Are You’)).
I am placing file splitting code please suggest if anything can be done?

 
Jayesh A Lalwani
Rancher
Pie
Posts: 2756
32
Eclipse IDE Spring Tomcat Server
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You don't really need to split file. You are adding unnescary complication to the code. You can read lines in batches (of let's say 1000 lines each) process them, then read the next batch
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13048
6
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Are you required to use a browser on the client side?

If not, a custom Java client would be much more flexible - for example it could send batches of 100 lines and keep track of how many lines have been processed so that you could resume processing if the connection was broken.

Seems to me the chance of a connection problem in the course of sending 100MB with DB processing on the server must be pretty substantial.

Bill
 
I agree. Here's the link: http://aspose.com/file-tools
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic