Win a copy of Re-engineering Legacy Software this week in the Refactoring forum
or Docker in Action in the Cloud/Virtualization forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

File race condition

 
John Landon
Ranch Hand
Posts: 228
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,
I have a process COPY that copies files from dir A to dir B.
I have to write a program that grabs files from dir B, processes them and moves them to dir C.
My question is what happens if process COPY have not finished writing the file but my new process already grabed it. So it will be incomplete??..
Any ideas?

Thanks.
 
Paul Clapham
Sheriff
Pie
Posts: 20758
30
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yes, that will be trouble if you do that. So arrange your file creation protocol so that doesn't happen. Don't allow the writing task to write in a place where the reading task is looking.

So for example you could have the writing task write files into directory X, then move them into A when they are complete. (If X and A are on the same drive then that just changes pointers in the file system, so you can treat it as atomic.)

Or you could have the reading task only read files with a certain property (such as the extension not being ".notdoneyet"). Then the writing task writes files with that extension, and renames them when it's finished.

Don't bother with clever solutions like waiting for the file size to stop changing, or putting locks on the files. I've tried them and they never worked reliably for me.
 
Campbell Ritchie
Sheriff
Posts: 48404
56
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I would suggest you try it, and tell us what happens. I suspect that until the file has been completely written, you will suffer FileNotFoundExceptions. But I am not sure.
 
John Landon
Ranch Hand
Posts: 228
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
is there a way to check if the file is in use?
 
John Landon
Ranch Hand
Posts: 228
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Paul Clapham wrote:Yes, that will be trouble if you do that. So arrange your file creation protocol so that doesn't happen. Don't allow the writing task to write in a place where the reading task is looking.

So for example you could have the writing task write files into directory X, then move them into A when they are complete. (If X and A are on the same drive then that just changes pointers in the file system, so you can treat it as atomic.)

Or you could have the reading task only read files with a certain property (such as the extension not being ".notdoneyet"). Then the writing task writes files with that extension, and renames them when it's finished.

Don't bother with clever solutions like waiting for the file size to stop changing, or putting locks on the files. I've tried them and they never worked reliably for me.


how do I check that? "when they are complete"
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic