File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes Java in General and the fly likes Check ftp download files finish Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Java in General
Bookmark "Check ftp download files finish" Watch "Check ftp download files finish" New topic
Author

Check ftp download files finish

michael yue
Ranch Hand

Joined: Nov 20, 2003
Posts: 204
Hi,

I am building a program to ftp some files which has a few parts. The download can happen at anytime. How can I detect whether I have finished download the few files besides from using threading? Any api that is suitable for this? Thanks
Henry Wong
author
Sheriff

Joined: Sep 28, 2004
Posts: 18510
    
  40

I am assuming that you are referring to the receiving end, as the sending side can easily tell when it is finished...


IMHO, you have a couple of ways that it could work...

- You could monitor the modification times. If the file hasn't been modified in the last minute or so, then you can assume that the transfer completed.

- You could modify the sender to send a small marker file. For example, it transfers a file call "ABC.TXT", when it is done, it transfers another file called, ".MYPROG.MARKER.FILE.ABC.TXT". You receiver should then just check for the appearance of ".MYPROG.MARKER.FILE.*" and assume the file it is referring to, is completed.

Henry


Books: Java Threads, 3rd Edition, Jini in a Nutshell, and Java Gems (contributor)
michael yue
Ranch Hand

Joined: Nov 20, 2003
Posts: 204
Hi
Thanks for the reply but actually I need to determine that all 10 files has been downloaded successfully. Meaning I have to detect the point where all the 10 files has been ftp in sucessfully. Telling the sender to send in confirmation files when finish downloading is not impossible but troublesome as this may involve an additional 10 files to send from the other side. Thanks
Henry Wong
author
Sheriff

Joined: Sep 28, 2004
Posts: 18510
    
  40

Same trick will work, you can uses a marker file, or check the modification times on the 10 files.

Using marker files is probably easier, as you can use one file to signal the sending of all 10 files.

Henry
Stefan Wagner
Ranch Hand

Joined: Jun 02, 2003
Posts: 1923

Doesn't the File-Transfer-Protocoll define a way to tell when finished?
When I download a very small file, I don't have to wait a minute or so. Firefox immediately knows when ready.
It also knows appromaxitive how long the download will take and exactly, how big the file will be - that might be a starting point.

And I don't receive MARKER.ABC files all the time (and how to know I finished receiving them? get a MARKER.ABC.MARKER.ABC file )


http://home.arcor.de/hirnstrom/bewerbung
Henry Wong
author
Sheriff

Joined: Sep 28, 2004
Posts: 18510
    
  40

Originally posted by Stefan Wagner:
Doesn't the File-Transfer-Protocoll define a way to tell when finished?
When I download a very small file, I don't have to wait a minute or so.


The FTP client knows when a file transfer has finished. The FTP server knows when a file transfer has finished. Unfortunately, in this case, it is neither the ftp client or server -- it is a java program sitting on the server side, waiting for files to appear.

Henry
Paul Clapham
Bartender

Joined: Oct 14, 2005
Posts: 18541
    
    8

Actually the way I read the original post, the question is whether the client can tell when the download is complete. But it's obvious that an FTP client can tell when it has finished downloading a file, so there must be more to the question than that. I don't see why it would be difficult for this program to have a few boolean flags somewhere, and set one for each download. When all the flags are set, all the downloads are done.
michael yue
Ranch Hand

Joined: Nov 20, 2003
Posts: 204
Hi

Sorry for the vague question. I need to do a client program to check whether the 10 files is completely obtained so that I can trigger another program to make a report based on the data on those 10 files. So I need a program that can check from time to time.

Thanks
Ulf Dittmer
Marshal

Joined: Mar 22, 2005
Posts: 41096
    
  44
As Paul said, the client knows exactly when the download is done. Can't it start that other program directly, or through a shell script?


Ping & DNS - my free Android networking tools app
Henry Wong
author
Sheriff

Joined: Sep 28, 2004
Posts: 18510
    
  40

I was under the impression (incorrectly) that this was a third program that was not involved in the transfer. If this is the ftp client, then detecting completion should be part of the API. (of the ftp library that you are using)

Henry
Samir Jyoti
Greenhorn

Joined: Jan 01, 2006
Posts: 8
Guys,

Same Problem here. FTP is wrtting some files and a timed process is reading the file. I need to ensure if the file have been completely downloaded (FTP is over).

How did you solve your problem.

- Samir
Samir Jyoti
Greenhorn

Joined: Jan 01, 2006
Posts: 8
there are a few posts at:

http://forum.java.sun.com/thread.jspa?forumID=4&threadID=423493


most of them suggest a control file or a to-try-to-rename operation. I can't get control files so the best option for me was to-try-to-rename the file; untill it allows.

The problem with that is that it works fine on windows, but fails on unix environments; there is even a bug in java relating to this:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4017593

so where do we go from here?

- Samir
Paul Clapham
Bartender

Joined: Oct 14, 2005
Posts: 18541
    
    8

If you can't get the people sending the file to cooperate, here's what I would do: when you first see the file on your system, wait one hour and then process it. If they don't like the one-hour delay, suggest to them that a more reliable method (like sending a control file) would enable you to speed the process up.
Samir Jyoti
Greenhorn

Joined: Jan 01, 2006
Posts: 8
come one everyone! I am sure this is a common problem and there must be smart and fail-free mechnism to detect. Please suggest a solution other than the control file solution.
Stefan Wagner
Ranch Hand

Joined: Jun 02, 2003
Posts: 1923

I think I still don't get it.
Are you downloading these files (then you could do it with java, and know when finished).
Or is there someone uploading files?
Paul Clapham
Bartender

Joined: Oct 14, 2005
Posts: 18541
    
    8

Samir, if you believe there is a simple and reliable solution then by all means go out and find it and post it here. I have never found any such solution. And I have participated in quite a few discussions of the topic (including the one you linked to) and none of them ever produced anything practical. But a simple solution must be easy to find, so please do.

What we usually agree with our business partners is that we will do an FTP upload to a file whose name they are not looking for, and when the upload is complete then we will rename the file to something they are looking for. And likewise for their uploads to us. They all agree to do that because they've had the problem too.
Henry Wong
author
Sheriff

Joined: Sep 28, 2004
Posts: 18510
    
  40

For me, using a small file to mark the completion worked best. I have also used the rename the file upon completion technique -- which worked well too.

Checking the modification times, and then wait another X minutes, worked just okay. The problem with this, is sometime failures leave partial files.


Of course, this whole discussion depends on the fact that we are writing application that is neither the FTP client or server.

Henry
Henry Wong
author
Sheriff

Joined: Sep 28, 2004
Posts: 18510
    
  40

Originally posted by Stefan Wagner:
I think I still don't get it.
Are you downloading these files (then you could do it with java, and know when finished).
Or is there someone uploading files?


I think... because this is such a common problem that this thread has been somewhat hijacked.

For the original poster, yes, he should be able to tell since he is the FTP client. For everyone else, we are still working it out ...

Henry
Jonathon Knight
Greenhorn

Joined: May 19, 2006
Posts: 1
Hey guys,

I too had this problem - on an AIX platform. I managed to solve it (so far..?) like this..

When the sender is sending the file to your machine and your Java app needs to check this inbound file to see if it's "finished", the best way i found was to try and "read" the file. Basically if it's still transferring, it will throw a FileNotFoundException which you catch and handle and essentially "loop" until the error doesn't happen.

Trying the built in File.canRead() or File.canWrite() functions are useless -- they always seem to return a "true" which isn't what you want for an FTP transfer check. The reason being on UNIX is the swap file -- yuo can still rename and move a file around while it is in mid-FTP transfer so the built in "java.io.File" commands that seem obvious don't work.

I figured out that if you try to physically "read" the first line of data in the file while it is in FTP transfer, it throws the FileNotFoundException. When the transfer is done, it doesnt throw the error, meaning it's cool for you to do what you need to the file. here's a snippet of what I did:




The problem with this now is what happens if you get a file that has 0 bytes in it - well, it loops forever I need to put another check into this. BTW, it *was* working flawlessly for the past 3 years until a 3rd party sent us an unexpected file with 0 bytes. It shouldnt have happened but it highlighted the flaw, so I am correcting it now.

But the main issue is, that if you try to do a br.readLine() on a file that is in mid-transfer, it fails. So just handle the exception (ie, do nothing) and keep looping until the exception ISNT thrown.

Hope maybe this helps - it's worked wonders for me as I couldnt find any solution to this issue on AIX.

Cheers,

Jon
 
Consider Paul's rocket mass heater.
 
subject: Check ftp download files finish
 
Similar Threads
downloading all the files in a folder through ftp
ftp files Problem
WinXP On Linux
protocols quary: need help
FTP UPLOAD