Meaningless Drivel is fun!*
The moose likes Threads and Synchronization and the fly likes Using Threads to insert files into DB Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Threads and Synchronization
Bookmark "Using Threads to insert files into DB" Watch "Using Threads to insert files into DB" New topic

Using Threads to insert files into DB

Mike sams

Joined: Apr 24, 2012
Posts: 2

Files will be generated in a folder during run time and using Threads, I need to insert each file in the DB .
Using threads I have to do this.Without missing any files, I have to insert all the files into DB.

Can anyone kindly suggest me how I can do this using Threads.

William P O'Sullivan
Ranch Hand

Joined: Mar 28, 2012
Posts: 860

You will definitely need some sort of GateKeeper or Traffic Cop.

If Thread#1 is processing File#, you do not want Thread#2 to pick up File#1 as well.

Threads are great for asynchronous processing etc.. but you do need to think a little more
about your deliverable and how to architect then build it.

Martin Vajsar

Joined: Aug 22, 2010
Posts: 3606

Welcome to the Ranch, Mike!

In your case, a producer-consumer design pattern might be used (google it up if you haven't encountered that yet). You'd have one producer thread, which would be responsible for locating new files and putting them into queue to be picked up by consumers. Since you have only once consumer, you can keep records of which files have been processed without need for any synchronization. Consumers would pick the file name from the queue and process them. You could use a synchronized queue (such as ArrayBlockingQueue, but there are others) to do all the hard synchronization work.

I can see two potential problems:

1) I assume that the files are being generated by an external process. You need to make sure you don't start processing a file before the external process is done with it. It might take some time to write the file; if you start processing it too early, you might get exceptions (if the file is locked, for example), or - worse - you might end up uploading incomplete file into the database.

2) Is your infrastructure capable of uploading several files at once? Database and network could present a bottleneck to your solution, by uploading more than one file at once you might actually decrease your overall performance. In the producer-consumer architecture this can actually be easily adjusted by creating a limited number of consumers (eg. only one).
Mike sams

Joined: Apr 24, 2012
Posts: 2
Hi Martin,

Thank you very much for your suggestion .I am going through the producer-consumer design pattern .
After reading the file that is generated by the external process , I want to write to the DB and later want to delete the file from the folder .
At any point of time, the program should read only one file and only after writing to the DB, the program should read the second file.

If you possible can you please give me a small example of this as it would help me greatly.

Thank you very much.

Martin Vajsar

Joined: Aug 22, 2010
Posts: 3606

The Ranch is NotACodeMill (<-- click). Try to write the code yourself and if you get stuck, feel free to ask specific question about any problem you encounter.

However, just note that if you want to process only one file at a time, you don't need to make your code multithreaded at all.
It is sorta covered in the JavaRanch Style Guide.
subject: Using Threads to insert files into DB
Similar Threads
How to read the DESede encrypted value using a Java Program
Process the multiple records in a file by Producer/consumer concept using Multithreading
PreparedStatement and sysdate
Techniques for getting auto generated key for ms access?