This is actually a web application so it will be difficult to ask clients to zip all the files and upload.
I am looking for some mechanism that will make the upload process much faster. Expecting almost up to 1GB files also from clients.
Your server code doesn't have any control over the upload process. If the client uploads a 1-GB file then the network will transfer that gigabyte of data to your server code in whatever way it sees fit.
But look at what your server code is doing. Hopefully it isn't writing that gigabyte of data to a temporary file and then copying it to some other location; that would make the upload even longer.
You could also scrap the code you are using now and write an applet to assist with the uploading. This applet could upload the data to your server, gzipping it as it goes (as per James Ward's suggestion).
Joined: Apr 30, 2009
I do understand why you are suggesting to go with Applet. I will try to give you my upload process overview
1) Should use HTTPS protocol.
2) Each file should be scanned with UV scan in web server before moving into the right unix box.
With the above constraints i have to upload the files from my web application and move these files to right unix box after they are properly scanned for virus.
a) Presently i am using struts2 which inturn uses commons fileupload to upload the file from browser.
b) This is actually coping file in to the tmp locaiton in server and i am scanning files from that location and sending the files to right unix box using the JSch API.
Possible improvements i am looking in the following areas
p) Is there any mechanism that will improve the file upload part in the browser [i know that it is totally client side and the server does't have any control]
q) As you guys suggested gzipping might help me some extent.[I am yet to work on this]
r) Making seperate process for each file scanning might improve[using multiple threads]
Let me know if anyone have any suggestions to imporve the above solution or better idea that will improve performance for the above problem with the given contraints. No restrictions on the technologies as long as they are open sources[java].