Multi Files Transfer from client machines to server machines using socket communication in java

[Riyas Hameed]

Hi All,

if any one have the code to transfer files from client to server , Could you please [share] the code for multi file transfering from client to server.
Expecting your valuable response ..

My requirement is transfering all files from particular folder in client machine to target folder in server machine, I have the code but its transferring only one file to server machine after that connection disconnect. plese guide me to complete my requirement. (Java Socket Communication)

Thanks in Advance..

[Henry Wong]

https://coderanch.com/how-to/java/UseTheForumNotEmail

[Riyas Hameed]

Henry Wong wrote:https://coderanch.com/how-to/java/UseTheForumNotEmail

Sorry for that. I was looking in thread but i didnt get a solution for Multi file transfer using socket communication in java, could you please guide , My code is transferring only one file at a time..


I am new to this concept. for file transfering which is the best mechanism in java , either Socket communication or FTP?
please advice me the best concept to do the file transfer mechanism .

Expecting your valuable reply.
Thanks in Advance .

[Jeff Verdegan]

Riyas Hameed wrote:

Henry Wong wrote:https://coderanch.com/how-to/java/UseTheForumNotEmail

Sorry for that. I was looking in thread but i didnt get a solution for Multi file transfer using socket communication in java, could you please guide , My code is transferring only one file at a time..

If you know how to do one file, then you know how to do multiple files.

I am new to this concept. for file transfering which is the best mechanism in java , either Socket communication or FTP?

There is no single "best" mechanism. The best one for you in this particular situation is the one that best meets your requirements in this particular situation.

[Riyas Hameed]

Jeff Verdegan wrote:

Riyas Hameed wrote:

Henry Wong wrote:https://coderanch.com/how-to/java/UseTheForumNotEmail

Sorry for that. I was looking in thread but i didnt get a solution for Multi file transfer using socket communication in java, could you please guide , My code is transferring only one file at a time..

If you know how to do one file, then you know how to do multiple files.

I am new to this concept. for file transfering which is the best mechanism in java , either Socket communication or FTP?

There is no single "best" mechanism. The best one for you in this particular situation is the one that best meets your requirements in this particular situation.

Thanks for your reply, but I am getting connection refused after one file transferred to server, I am looking on that.

Here My code is :

Client part :

	for (int itor = 0; itor < numFiles; itor++)
				{        
					String fileName = files[itor];
					System.out.println("Client  itor" + itor);
					
					System.out.println("transfer File name is : " + fileName); 
					File sentFile = new File("D:/myfilelisting/"+fileName); 
					len = sentFile.length();  
					System.out.println("Transferring Length *****"+len); 
					out.println(len);
					out.println(sentFile);   
					//SENDFILE       
					bis = new BufferedInputStream(new FileInputStream("D:/myfilelisting/"+fileName)); 
					bos = new BufferedOutputStream(socket.getOutputStream( )); 
					byteArray = new byte[1000000];  
					count = 0;    
					while ( (num = bis.read(byteArray)) >0 )
					{         //num = bis.read(byteArray);   
						bos.write(byteArray,0,num);  
						}     
					bos.close(); 
					bis.close(); 
					System.out.println("file done: " + itor);
					}    
				done = true;  
				} catch (IOException ee){

Server part:

                       String length2 = inm.readLine(); 
					System.out.println("server side file length ********"+length2); 
					String filename = inm.readLine();
					String myfile = filename.substring(17);	
					
					System.out.println("Filename in server ***** = " + filename);
					out.println("ACK from server side : Filename received = " + filename);
					//RECIEVE and WRITE FILE
					byte[] receivedData = new byte[1000000];
					bis = new BufferedInputStream(incoming.getInputStream());			
					bos = new BufferedOutputStream(new FileOutputStream("D:/TargetFolder/"+myfile));
					//long length = (long)Integer.parseInt(length2);
					//length++;   
					while ( (num = bis.read(receivedData)) > 0)
					{        
						bos.write(receivedData,0,num);
						System.out.println("Writing the contents in the file from server ****");
					}       //System.out.println(counter);  
					bos.close(); 
					bis.close(); 

kindly advise me , what mistake i did in my code ..

[Jeff Verdegan]

In the future, please UseCodeTags so your code will be readable.


You're closing your socket's OutputStream after each file. You need to either:

1) Keep the socket and its streams open throughout the entire loop over all files.

OR

2) Create a new socket connection for each file, and close it in a finally block at the end of transferring that file.

I'd recommend #2. If you go with #1, you'll have to develop an additional protocol to delineate the files. That's not necessarily difficult, but since you're having these difficulties so far, it's probably best to start simple.

[Rob Spoor]

Riyas Hameed wrote:

					while ( (num = bis.read(byteArray)) >0 )

					while ( (num = bis.read(receivedData)) > 0)

0 is a valid return code for InputStream.read(byte[]). It means that currently no data is available, but in the future there still may be. -1 is the only indicator that there will not be any data anymore, so change the > into >=.

[Riyas Hameed]

Thanks a lot to all, I will do the same , Iam new to this forum going forward i will ad my code in code tag. I will do what you have mentioned and come back to you if i am facing any issue.

[Riyas Hameed]

Sir ,Please see the below code .

[Riyas Hameed]

Thanks a lot , As per your words, I did the changes , Its working fine..I jus added one more while loop in server part

while (true){
			try 
			{ 
				Socket incoming = socket.accept(); 
				try {  
					try{     
						if (!(file.exists()))
						{       
							file.mkdir(); 
						}      
						InputStream inStream = incoming.getInputStream();
						OutputStream outStream = incoming.getOutputStream();
						System.out.println("New Client Connected with id  from "+incoming.getInetAddress().getHostName()+"..." );
						System.out.println("Input stream *******************" + inStream.toString());
						
						BufferedReader inm = new BufferedReader(new InputStreamReader(inStream));
						
						PrintWriter out = new PrintWriter(outStream, true /* autoFlush */);
						 message = inm.readLine();
						System.out.println("Mesage Reached to Server"+message);
						
						String length2 = inm.readLine(); 
						System.out.println("server side file length ********"+length2); 
						String filename = inm.readLine();
						
						String myfile = filename.substring(17);	
						
						System.out.println("Filename in server ***** = " + myfile);
						//out.println("ACK from server side : Filename received = " + filename);
						//RECIEVE and WRITE FILE
						byte[] receivedData = new byte[1000000];
						bis = new BufferedInputStream(incoming.getInputStream());			
						bos = new BufferedOutputStream(new FileOutputStream("c:/TargetFolder/"+myfile));

}
}

Now its working fine, This client application will run more than 500 Machines.So how can i make the best perofrmance .please suggest me to do this part. Thanks in Advacne.

[Riyas Hameed]

Riyas Hameed wrote:Thanks a lot , As per your words, I did the changes , Its working fine..I jus added one more while loop in server part.

Now its working fine, This client application will run more than 500 Machines.So how can i make the best perofrmance .please suggest me to do this part. Thanks in Advacne.

[Jeff Verdegan]

Riyas Hameed wrote:Now its working fine, This client application will run more than 500 Machines.So how can i make the best perofrmance .please suggest me to do this part. Thanks in Advacne.

For parallel I/O such as this, the two main general approaches to optimize performance are 1) buffering, and 2) multithreading. You're already doing #1, although you may find that tweaking your buffer size up or down helps somewhat, so the main thing now is to use mulitple threads.

If I understand your scenario correctly you have many clients receiving files from a server and storing them on their local file systems. So you'll want multithreading in two places. 1) On the server, you'll want multiple threads to handle clients' requests. You'll have to experiment to find the optimal number. It could be anywhere between a few threads and a few hundred. 2) On the client, you might want two threads--one for reading from the server and one for writing to the file. Or that might not help at all. Impossible to say without trying it in the specific situation in which it will be used.

Other than that, the only thing you can do is see if the app is meeting your performance requirements, and if not, use a profiler to find bottlenecks.

[Riyas Hameed]

Jeff Verdegan wrote:

Riyas Hameed wrote:Now its working fine, This client application will run more than 500 Machines.So how can i make the best perofrmance .please suggest me to do this part. Thanks in Advacne.

For parallel I/O such as this, the two main general approaches to optimize performance are 1) buffering, and 2) multithreading. You're already doing #1, although you may find that tweaking your buffer size up or down helps somewhat, so the main thing now is to use mulitple threads.

If I understand your scenario correctly you have many clients receiving files from a server and storing them on their local file systems. So you'll want multithreading in two places. 1) On the server, you'll want multiple threads to handle clients' requests. You'll have to experiment to find the optimal number. It could be anywhere between a few threads and a few hundred. 2) On the client, you might want two threads--one for reading from the server and one for writing to the file. Or that might not help at all. Impossible to say without trying it in the specific situation in which it will be used.

Other than that, the only thing you can do is see if the app is meeting your performance requirements, and if not, use a profiler to find bottlenecks.

Thank a lot for your support , In my requirement , All clients will transfer the files to the centralized server and save in server location.

So I have to create two threads in server side one for reading and one for writing ? please correct me If i am wrong. I have implemented thread concept in server side , some time some of the bytes are missing while writing in server side , now i am facing the issue . can i share my code here ??

[Jeff Verdegan]

Riyas Hameed wrote:
Thank a lot for your support , In my requirement , All clients will transfer the files to the centralized server and save in server location.

So I have to create two threads in server side one for reading and one for writing ? please correct me If i am wrong. I have implemented thread concept in server side

If all 500 clients are sending their files at the same time, I would start with something like 10-20 threads reading from clients, putting into queues (1 queue per destination file) and 1-5 threads reading from the queues and writing to files. I'd probably use ThreadPoolExecutors and ArrayBlockingQueues or LinkedBlockingQueues.

Since I don't know your system or your requirements at all, this is just starting point. You'll have to do some research into the relevant classes, and do some testing and tweaking on your setup to see what works best.

, some time some of the bytes are missing while writing in server side , now i am facing the issue

Then there's a bug in your code.

. can i share my code here ??

Probably the whole thing will be too much for anybody to want to read. You'll have better luck if you can create an SSCCE that shows the relevant parts of your code and ideally demonstrates the problem, without any extra stuff that's not directly related to the problem.

[Riyas Hameed]

Jeff Verdegan wrote:

Riyas Hameed wrote:
Thank a lot for your support , In my requirement , All clients will transfer the files to the centralized server and save in server location.

So I have to create two threads in server side one for reading and one for writing ? please correct me If i am wrong. I have implemented thread concept in server side

If all 500 clients are sending their files at the same time, I would start with something like 10-20 threads reading from clients, putting into queues (1 queue per destination file) and 1-5 threads reading from the queues and writing to files. I'd probably use ThreadPoolExecutors and ArrayBlockingQueues or LinkedBlockingQueues.

Since I don't know your system or your requirements at all, this is just starting point. You'll have to do some research into the relevant classes, and do some testing and tweaking on your setup to see what works best.

, some time some of the bytes are missing while writing in server side , now i am facing the issue

Then there's a bug in your code.

. can i share my code here ??

Probably the whole thing will be too much for anybody to want to read. You'll have better luck if you can create an SSCCE that shows the relevant parts of your code and ideally demonstrates the problem, without any extra stuff that's not directly related to the problem.

Thank you very much, yes I have to go through all other thing which you have mentioned in the previous message . I am not sure but atleast minimum 50 to 100 Machine will send a file to server at a time . i will discuss with them and come back to you ..

[Riyas Hameed]

Riyas Hameed wrote:

Jeff Verdegan wrote:

Riyas Hameed wrote: Thank a lot for your support , In my requirement , All clients will transfer the files to the centralized server and save in server location. So I have to create two threads in server side one for reading and one for writing ? please correct me If i am wrong. I have implemented thread concept in server side

If all 500 clients are sending their files at the same time, I would start with something like 10-20 threads reading from clients, putting into queues (1 queue per destination file) and 1-5 threads reading from the queues and writing to files. I'd probably use ThreadPoolExecutors and ArrayBlockingQueues or LinkedBlockingQueues. Since I don't know your system or your requirements at all, this is just starting point. You'll have to do some research into the relevant classes, and do some testing and tweaking on your setup to see what works best.

, some time some of the bytes are missing while writing in server side , now i am facing the issue

Then there's a bug in your code.

. can i share my code here ??

Probably the whole thing will be too much for anybody to want to read. You'll have better luck if you can create an SSCCE that shows the relevant parts of your code and ideally demonstrates the problem, without any extra stuff that's not directly related to the problem.

Thank you very much, yes I have to go through all other thing which you have mentioned in the previous message . I am not sure but atleast minimum 50 to 100 Machine will send a file to server at a time . i will discuss with them and come back to you ..

Here is my code for read and write in file :

Server side : read the data from the file which have sent from client and writ in a file .

                                                          
                                                                                       byte[] receivedData = new byte[1000000];
					bis = new BufferedInputStream(incoming.getInputStream());			
					bos = new BufferedOutputStream(new FileOutputStream("c:/TargetFolder/"+myfile));
					 
                                                                                        while ( (num = bis.read(receivedData)) > 0)
					{        
						bos.write(receivedData,0,num);	
					
											
					} 

Client code :

                                                                                         bis = new BufferedInputStream(new FileInputStream("c:/myfilelisting/"+fileName)); 
					bos = new BufferedOutputStream(socket.getOutputStream( )); 
					byteArray = new byte[50000000];  
					count = 0;    
					while ( (num = bis.read(byteArray)) >0 )
					{         //num = bis.read(byteArray);   
						bos.write(byteArray,0,num);  
						System.out.println("Messge from server>>>>>>>>>>>>>>>>>>>" + inm.readLine());
						}

Some of the file bytes are missing while writing to server . please advise me to solve this . Thanks in Advance

[Paul Clapham]

If you read the documentation for the "read" method you're using, you will see that it returns -1 when it gets to the end of the stream. But you aren't checking for that. You are also considering 0 as a reason to stop reading.

But when the "read" method returns 0, that just means it has no bytes for you just at the moment. It doesn't mean that there will never be any more bytes. So fix your end-of-stream tests.

[Jeff Verdegan]

Paul Clapham wrote:If you read the documentation for the "read" method you're using, you will see that it returns -1 when it gets to the end of the stream. But you aren't checking for that. You are also considering 0 as a reason to stop reading.

But when the "read" method returns 0, that just means it has no bytes for you just at the moment. It doesn't mean that there will never be any more bytes. So fix your end-of-stream tests.

This was already mentioned by Rob Spoor many replies and 4 days ago. Perhaps this time it will sink in. :-)

[Paul Clapham]

You're right. And not only that, Rob described exactly what code changes should be done. And that was the only thing in his post. So yeah, there was really no excuse for
not noticing his post.

[Rob Spoor]

Sometimes people are focusing so much on what one person said that what others say doesn't sink in. This wouldn't be the first time when a post of mine had to be brought under the attention again because it was overlooked at first.

[Jeff Verdegan]

I think it's happened to us all a number of times. You get in a thread with multiple respondents and multiple issues to be addressed and the OP gets overwhelmed.

[Riyas Hameed]

Rob Spoor wrote:

Riyas Hameed wrote:

					while ( (num = bis.read(byteArray)) >0 )

					while ( (num = bis.read(receivedData)) > 0)

0 is a valid return code for InputStream.read(byte[]). It means that currently no data is available, but in the future there still may be. -1 is the only indicator that there will not be any data anymore, so change the > into >=.

Rob, I tried already what you said in the above conversation, even though i got the same error only, extremely sorry, forget to noticed that one.

[Jeff Verdegan]

Riyas Hameed wrote:

Rob Spoor wrote:

Riyas Hameed wrote:

					while ( (num = bis.read(byteArray)) >0 )

					while ( (num = bis.read(receivedData)) > 0)

0 is a valid return code for InputStream.read(byte[]). It means that currently no data is available, but in the future there still may be. -1 is the only indicator that there will not be any data anymore, so change the > into >=.

Rob, I tried already what you said in the above conversation, even though i got the same error only, extremely sorry, forget to noticed that one.

The last code you posted shows:

 while ( (num = bis.read(byteArray)) >0 )  

Which is still the same mistake you were making in the beginning, and does not incorporate Rob's (and Paul's) advice. Fix that, and then if you're still having trouble, post your latest code, or an SSCCE that accurately represents it.

[Riyas Hameed]

Riyas Hameed wrote:

Rob Spoor wrote:

Riyas Hameed wrote:

					while ( (num = bis.read(byteArray)) >0 )

					while ( (num = bis.read(receivedData)) > 0)

0 is a valid return code for InputStream.read(byte[]). It means that currently no data is available, but in the future there still may be. -1 is the only indicator that there will not be any data anymore, so change the > into >=.

Rob, I tried already what you said in the above conversation, even though i got the same error only, extremely sorry, forget to noticed that one.

[Riyas Hameed]

Riyas Hameed wrote:

Riyas Hameed wrote:

Rob Spoor wrote:

Riyas Hameed wrote:

					while ( (num = bis.read(byteArray)) >0 )

					while ( (num = bis.read(receivedData)) > 0)

0 is a valid return code for InputStream.read(byte[]). It means that currently no data is available, but in the future there still may be. -1 is the only indicator that there will not be any data anymore, so change the > into >=.

Rob, I tried already what you said in the above conversation, even though i got the same error only, extremely sorry, forget to noticed that one.

Thanks a lot for your support, Now it is working fine . I have changed the condition to till -1. Thanks for your time ..