HttpClient file upload code review

[Soumyajit Hazra]

Hello Ranchers,
Using help from the various useful discussions posted in this community I successfully used HttpClient API to automate some task. I need to create a java process that will perform login to a secured site then navigate to a particular page which contains one file component to upload a file. Now my java process will pick files from a specified directory and upload all the files my logging into the site instead of logging into the site via browser and uploading files one by one. Below is the code I am using for file upload from my java class

 private void uploadFiles(HttpClient client){
		ArrayList<File> listFile=this.getFileList();     // getting list of file from a directory specified
		for(File file:listFile){
			try{			
				PostMethod filePost = new PostMethod("<some url>/upload.asp");
				//filePost.getParams().setBooleanParameter(HttpMethodParams.USE_EXPECT_CONTINUE,true);
				Part[] parts = {new FilePart(file.getName(), file)};
				filePost.setRequestEntity(new MultipartRequestEntity(parts,filePost.getParams()));
				int status = client.executeMethod(filePost);
				logger.info("File Upload Status for file "+file.getName()+" is "+status+"("+filePost.getStatusLine()+")");
				filePost.releaseConnection();
				
			}catch(Exception ie){
				ie.printStackTrace();
			}
		}		
	}  

I need some advice regarding the above code like
1. What exactly the line filePost.getParams().setBooleanParameter() doing?
2. As I am uploading multiple files is it ok to create a PostMethod object every time I enter the loop and to release it at the end or I can create one then after all the files uploaded then to release the connection?
3. As I have very limited access to the secured site I have created some webpages of my own and deployed it in Tomcat and it did worked fine but in actual server when I tried to upload a single file using the code after uploading the file name it displays that is uploaded is octet-strea. I am completely clueless about this error. In this type of situation mainly what type of information about the remote server I need to get from the remote server admin?

[Martijn Verburg]

1.) It's setting a parameter that will have some effect when you perform the POST operation, lookup the javadoc to see what HttpMethodParams.USE_EXPECT_CONTINUE does for you

2.) Have you tried both options.....?

3.) So logger.info("File Upload Status for file "+file.getName()+" is "+status+"("+filePost.getStatusLine()+")"); is returning you octect-stream?

[Soumyajit Hazra]

Thanks for the reply. I checked the doc for HttpMethodParams.USE_EXPECT_CONTINUE but not understand properly what it saying.
Yes I have tried both the options. If I create a post object once and then loop through and at last release the connection then only one file is being uploaded not all the files. No no after uploading when some one manually login to the site there one option is present to check what are the files that has been uploaded, there in the file name section it shows the name as octect-stream but I am uploading test1.txt file and expecting that name. From my java code I am only getting the status code 200 and from that I conclude that file is uploaded.

[Martijn Verburg]

1.) There is a nice explanation here on use expect continue.

2.) Great! You answered your own question , just make sure you carefully close you resources each iteration.

3.) Not sure about this one, you need to find out where the server expects to get the file name from. Or are you directly viewing the file via a web browser?

[Soumyajit Hazra]

Thanks for the link. I'll check your 3rd point and get back to the discussion. One question I want to clarify , from my java code apart from the status code check can I do any thing else to check whether the file is uploaded or not. The reason of asking this is I don't have the credential to access the site. I need to place some request to some admin they will check and then let me know the result that what is happened actually. This is taking time.

Thank you Martijn .

[Martijn Verburg]

Assuming you're allowed to view where that file is you can use HTTPClient to check for its existence, it sounds like you're not allow to have access though?

[Soumyajit Hazra]

Yes that's the saddest part..

[Soumyajit Hazra]

As I discussed from my earlier post that when I am trying to upload a file using my code the file name changed to octet-stream where as the actual file name is test1.txt. Manually when I am logging into the site via browser and uploads the same file it shows the same name test1.txt in the site. I am running my java code from unix box. Is this can be a reason for the name change?

[Soumyajit Hazra]

Hi Ranchers
From past 2 days I am not able to solve the issue. Any file I am uploading via my java code the site shows the name as octet-stream.
Please help.

[Soumyajit Hazra]

Still I am stuck in this issue.

[Soumyajit Hazra]

I observe few thing regarding this file upload process that I need to discuss, hope that may give some clue to solve the issue. The code what I have posted in my first post, when used shows the uploded file name as "octet-strea". Now I have changed few more line in the code

    final FilePart filePart=new FilePart(file.getName(), file);
				filePart.setContentType("multipart/form-data");
				filePart.setName(file.getName());
				Part[] parts = {filePart};
				filePost.setRequestEntity(new MultipartRequestEntity(parts,filePost.getParams()));

After that it is showing the file name as "form-dat". So basically it is taking the content type as the file name and first time when I have not given the content type it took the default as text/octet-stream and "octet-strea" as the name. The setName() is not working at all or I am using it in a wrong way? One more thing is that the site is built on asp.net not using jsp or servlet. Can any one give some suggestion.

[Martijn Verburg]

OK, so it sounds like you're sending the right file name now, is it just the file that's not arriving or?

[Soumyajit Hazra]

Once you upload a file there is a link to check uploaded file list. In that list it shows file name,file size,user etc. There it shows the name as form-dat.

[Soumyajit Hazra]

Just to add that I checked the code locally by creating some jsp files trying to simulate the application and there every thing works fine. Is it due to the asp.net issue?

[Martijn Verburg]

That's possible yes, the receiving web service could be reading information from the wrong parts of the stream, I'd speak to them to find out

[Soumyajit Hazra]

Thanks for your reply Martijn.

the receiving web service could be reading information from the wrong parts of the stream

So is it the HttpClient component what we are using doing something wrong or the asp.net pages that receiving my request messing it up?

[Martijn Verburg]

Well since you've tested HTTPClient against your own 'server' you know it works right? Another trick is to use a Firefox plugin to analyse the request going out (you can inspect the header etc.)

[Soumyajit Hazra]

Yes I have tried that too. The request header what I am getting from browser as follows,

(Request-Line)	POST /Secure/Tools/VisionUpload/,DanaInfo=msdwideas.msdwis.com+Upload.asp HTTP/1.1
Host	access.msdwis.com
User-Agent	Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0
Accept	text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language	en-us,en;q=0.5
Accept-Encoding	gzip,deflate
Accept-Charset	ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive	300
Connection	keep-alive
Referer	https://access.msdwis.com/Secure/Tools/VisionUpload/,DanaInfo=msdwideas.msdwis.com+upload.asp
Cookie	DSID=73e58bda13656cad33dbacd1c1e88c3b; DSFirstAccess=1287404839; DSLastAccess=1287404951; DSCK:VisionToolTips=On%3B%20%20expires%3DThu%2C%2015%20Oct%202020%2012%3A29%3A41%20GMT%3B%20%20path%3D/Secure/Tools/VisionUpload%3B%20%20domain%3Dmsdwideas.msdwis.com%3B%20
Content-Type	multipart/form-data; boundary=---------------------------281123069513169
Content-Length	502

and the response header

    
(Status-Line)	HTTP/1.1 200 OK
Pragma	No-Cache
Cache-Control	No-Cache, Private
Set-Cookie	DSCK:VisionToolTips=0; path=/; expires='Sat, 03-jan-1970 00:00:00 GMT'
Set-Cookie	DSLastAccess=1287404968; path=/; Secure
Content-Type	text/html
content-disposition	filename="Upload.asp"
Accept-Ranges	none
Vary	Accept-Encoding, User-Agent
Connection	Keep-Alive
Keep-Alive	timeout=15
Transfer-Encoding	chunked

And from my code the request header is

  INFO *********Header info starts********
 INFO Referer-https://access.msdwis.com/Secure/Tools/VisionUpload/,DanaInfo=msdwideas.msdwis.com+upload.asp
 INFO User-Agent-Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0
 INFO Host-access.msdwis.com
 INFO Cookie-$Version=0; DSID=596d32510d32032eef400a05a4868c7d; $Path=/; DSFirstAccess=1286886217; $Path=/; DSLastAccess=1286886219; $Path=/
 INFO Content-Length-257
 INFO Content-Type-multipart/form-data; boundary=4bOqFQa5l5OAN8wqWliltE0MR0ZKzDbB4qXb
  

Response

    INFO *********Response Header info starts********
(Status-Line)	HTTP/1.1 200 OK
 INFO Pragma-No-Cache
 INFO Cache-Control-No-Cache
 INFO Content-Type-text/html
 INFO Content-Disposition-filename="Upload.asp"
 INFO Accept-Ranges-none
 INFO Cache-Control-Private
 INFO Vary-Accept-Encoding, User-Agent
 INFO Set-Cookie-DSLastAccess=1286886222; path=/; Secure
 INFO Transfer-Encoding-chunked
 INFO *********Header info ends********

[Martijn Verburg]

What does the Javadoc for filePart.setName(file.getName()); say? How does this translate when you send that info to the server? How is the server expected to get that value out?

[Soumyajit Hazra]

The doc states the following info only,

public void setName(String name)
Sets the part name.
Parameters:
name -

[Soumyajit Hazra]

I have changed the setContentType to

filePart.setContentType(file.getName());

, I know that is not right but now I am getting the file name properly displayed. Only thing is that it displays only the first 8 character of whatever I give in the content type. So my file name is test1.txt and it displays test1.tx . But manually uploading via the site giving full name of the file.

[Soumyajit Hazra]

Using the following piece of code,

  final FilePart filePart=new FilePart(file.getName(),file,"multipart/form-data",null);
				Part[] parts = {filePart};
				filePost.setRequestEntity(new MultipartRequestEntity(parts,filePost.getParams()));  

I am getting the following log when log level set to DEBUG. It seems that the file test1.text reaches the server.

 DEBUG >> "Content-Type: multipart/form-data; boundary=_t8K0pGhXKD8hXeAnJF8QO3aDgrRUVgg9M6b[\r][\n]"
DEBUG enter HttpConnection.print(String)
DEBUG enter HttpConnection.write(byte[])
DEBUG enter HttpConnection.write(byte[], int, int)
DEBUG enter HttpConnection.writeLine()
DEBUG enter HttpConnection.write(byte[])
DEBUG enter HttpConnection.write(byte[], int, int)
DEBUG >> "[\r][\n]"
DEBUG enter EntityEnclosingMethod.writeRequestBody(HttpState, HttpConnection)
DEBUG enter PostMethod.hasRequestContent()
DEBUG enter EntityEnclosingMethod.hasRequestContent()
DEBUG enter EntityEnclosingMethod.getRequestContentLength()
DEBUG enter PostMethod.hasRequestContent()
DEBUG enter EntityEnclosingMethod.hasRequestContent()
DEBUG getLengthOfParts(Parts[])
DEBUG enter length()
DEBUG enter lengthOfData()
DEBUG enter sendStart(OutputStream out)
DEBUG enter sendDispositionHeader(OutputStream out)
DEBUG enter sendDispositionHeader(OutputStream out)
DEBUG enter sendContentTypeHeader(OutputStream out)
DEBUG enter sendTransferEncodingHeader(OutputStream out)
DEBUG enter sendEndOfHeader(OutputStream out)
DEBUG enter sendEnd(OutputStream out)
DEBUG enter lengthOfData()
DEBUG enter HttpConnection.getRequestOutputStream()
DEBUG enter send(OutputStream out)
DEBUG enter sendStart(OutputStream out)
DEBUG >> "--"
DEBUG >> "_t8K0pGhXKD8hXeAnJF8QO3aDgrRUVgg9M6b"
DEBUG >> "[\r][\n]"
DEBUG enter sendDispositionHeader(OutputStream out)
DEBUG enter sendDispositionHeader(OutputStream out)
DEBUG >> "Content-Disposition: form-data; name="
DEBUG >> """
DEBUG >> "test1.txt"
DEBUG >> """
DEBUG >> "; filename="
DEBUG >> """
DEBUG >> "test1.txt"
DEBUG >> """
DEBUG enter sendContentTypeHeader(OutputStream out)
DEBUG >> "[\r][\n]"
DEBUG >> "Content-Type: "
DEBUG >> "multipart/form-data"
DEBUG >> "; charset="
DEBUG >> "ISO-8859-1"
DEBUG enter sendTransferEncodingHeader(OutputStream out)
DEBUG >> "[\r][\n]"
DEBUG >> "Content-Transfer-Encoding: "
DEBUG >> "binary"
DEBUG enter sendEndOfHeader(OutputStream out)
DEBUG >> "[\r][\n]"
DEBUG >> "[\r][\n]"
DEBUG enter sendData(OutputStream out)
DEBUG enter lengthOfData()
DEBUG >> "test"
DEBUG enter sendEnd(OutputStream out)
DEBUG >> "[\r][\n]"
DEBUG >> "--"
DEBUG >> "_t8K0pGhXKD8hXeAnJF8QO3aDgrRUVgg9M6b"
DEBUG >> "--"
DEBUG >> "[\r][\n]"
DEBUG Request body sent
DEBUG enter HttpConnection.flushRequestOutputStream()
DEBUG enter HttpMethodBase.readResponse(HttpState, HttpConnection)
DEBUG enter HttpMethodBase.readStatusLine(HttpState, HttpConnection)
DEBUG enter HttpConnection.readLine()
DEBUG enter HttpParser.readLine(InputStream, String)
DEBUG enter HttpParser.readRawLine()
DEBUG << "HTTP/1.1 200 OK[\r][\n]"
   

File name at the site now again I am getting form-dat. Any idea Martijn?

[Martijn Verburg]

Not really no, have you talked to the ASP people at the other end?

[Soumyajit Hazra]

Not sure whether I can reach them. I'll try. But i am totally lost why the content type it is taking as file name. That too only 8 char , "form-dat".

[Soumyajit Hazra]

It's really frustrating time for me. To summarize the problem again for some help. I am using HttpClient to upload files to a site (build using asp)
but once I upload the file the file name what it displays in the next webpage is not right. It follows a pattern,
If I don't use any content type the name it shows is octet-strea, when I specify content type as multipart/form-data it shows form-dat and if I specify text/plain (the right one to be used here) it shows plai .
That means it takes second half of the content-type and removes the last character. I have tried the two different version of code and both gives the same result

private void uploadFiles(HttpClient client,String referer){
		ArrayList<File> listFile=this.getFileList();
		commonProperties=getCommonProperties();
		for(File file:listFile){
			try{
				MultipartPostMethod filePost = new MultipartPostMethod(commonProperties.getUploadUrl());
				filePost.addRequestHeader("User-Agent","Mozilla/4.0 " +
						"(compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.5.30729; .NET CLR 3.0.30618; .NET4.0C; AskTB5.4)");
				filePost.addRequestHeader("Referer",referer);
				filePost.addRequestHeader("Connection","keep-alive");
				filePost.addRequestHeader("Keep-Alive","300");
				client.setConnectionTimeout(8000);
				//filePost.addParameter ("File1", file.getName (),file);
				//FilePart part1 = new FilePart ( file.getName (), file);  				
				filePost.addPart(new FilePart(file.getName(),file,"text/plain","ISO-8859-1" ));
				
				logger.info("File entity :"+file+" added to post into url:"+filePost.getURI());
				int status = client.executeMethod(filePost);				
				logger.info("File Upload Status for file "+file+" is "+status+"("+filePost.getStatusLine()+")");
				filePost.releaseConnection();
			}catch(Exception ie){
				logger.info(ie);
				ie.printStackTrace();
			}
		}
	}    

The above code is deprecated. And the following code too I checked

  private HashMap<File,Exception> uploadFiles(HttpClient client,String referer){
		ArrayList<File> listFile=this.getFileList();
		commonProperties=getCommonProperties();
		HashMap<File,Exception> exceptionMap=new HashMap<File,Exception>();		
				
		for(File file:listFile){
			try{			
				PostMethod filePost = new PostMethod(commonProperties.getUploadUrl());
				
				filePost.addRequestHeader("User-Agent","Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9) Gecko/2008052906 Firefox/3.0");
				filePost.addRequestHeader("Referer",referer);
				filePost.addRequestHeader("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,;q=0.8"); //change required
				filePost.addRequestHeader("Connection","keep-alive");
				filePost.addRequestHeader("Keep-Alive","300");				
				//filePost.getParams().setBooleanParameter(HttpMethodParams.USE_EXPECT_CONTINUE,true);
				final FilePart filePart=new FilePart(file.getName(),file,"text/plain",null);
				Part[] parts = {filePart};
				filePost.setRequestEntity(new MultipartRequestEntity(parts,filePost.getParams()));
				logger.info("File entity :"+parts[0].getName()+" added to post into url:"+filePost.getURI());
				this.showHeaders(filePost);				//for debug purpose only
				int status = client.executeMethod(filePost);				
				this.showResponseHeaders(filePost);		//for debug purpose only
				logger.info("File Upload Status for file "+file+" is "+status+"("+filePost.getStatusLine()+")");
				filePost.releaseConnection();
				
			}catch(Exception ie){
				exceptionMap.put(file,ie);
				ie.printStackTrace();
			}
		}		
		return exceptionMap;
	}	  

Hard luck.