File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
Win a copy of Clojure in Action this week in the Clojure forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Using Threads to insert files into DB

 
Mike sams
Greenhorn
Posts: 2
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,

Files will be generated in a folder during run time and using Threads, I need to insert each file in the DB .
Using threads I have to do this.Without missing any files, I have to insert all the files into DB.

Can anyone kindly suggest me how I can do this using Threads.

Thanks,
 
William P O'Sullivan
Ranch Hand
Posts: 859
Chrome IBM DB2 Java
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You will definitely need some sort of GateKeeper or Traffic Cop.

If Thread#1 is processing File#, you do not want Thread#2 to pick up File#1 as well.

Threads are great for asynchronous processing etc.. but you do need to think a little more
about your deliverable and how to architect then build it.

WP
 
Martin Vajsar
Sheriff
Pie
Posts: 3747
62
Chrome Netbeans IDE Oracle
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Welcome to the Ranch, Mike!

In your case, a producer-consumer design pattern might be used (google it up if you haven't encountered that yet). You'd have one producer thread, which would be responsible for locating new files and putting them into queue to be picked up by consumers. Since you have only once consumer, you can keep records of which files have been processed without need for any synchronization. Consumers would pick the file name from the queue and process them. You could use a synchronized queue (such as ArrayBlockingQueue, but there are others) to do all the hard synchronization work.

I can see two potential problems:

1) I assume that the files are being generated by an external process. You need to make sure you don't start processing a file before the external process is done with it. It might take some time to write the file; if you start processing it too early, you might get exceptions (if the file is locked, for example), or - worse - you might end up uploading incomplete file into the database.

2) Is your infrastructure capable of uploading several files at once? Database and network could present a bottleneck to your solution, by uploading more than one file at once you might actually decrease your overall performance. In the producer-consumer architecture this can actually be easily adjusted by creating a limited number of consumers (eg. only one).
 
Mike sams
Greenhorn
Posts: 2
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Martin,

Thank you very much for your suggestion .I am going through the producer-consumer design pattern .
After reading the file that is generated by the external process , I want to write to the DB and later want to delete the file from the folder .
At any point of time, the program should read only one file and only after writing to the DB, the program should read the second file.

If you possible can you please give me a small example of this as it would help me greatly.

Thank you very much.

 
Martin Vajsar
Sheriff
Pie
Posts: 3747
62
Chrome Netbeans IDE Oracle
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The Ranch is NotACodeMill (<-- click). Try to write the code yourself and if you get stuck, feel free to ask specific question about any problem you encounter.

However, just note that if you want to process only one file at a time, you don't need to make your code multithreaded at all.
 
I agree. Here's the link: http://aspose.com/file-tools
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic