I have a timer schedule(r) which schedules the task once in every 10 mins. But again that task creates some other threads to process the data in its run method.
My question is, if the threads which are created in the run method of task takes more than 10 mins to finish, what happens to the scheduler? is it going to execute the run method again(or schedule the next task)? or will the scheduler (timer) wait till it finishes the run method of previous task before it schedules next task?
in other words, if run method of Task takes more than 10 mins and schedule time is 10 mins then what happens? will the scheduler schedule another task or waits for the previous task run method to be finished?
[Ugender]: My question is, if the threads which are created in the run method of task takes more than 10 mins to finish, what happens to the scheduler?
If you are creating threads in the run() method of a TimerTask, then they are most likely completely independent of the task. Unless you use Thread.join() or other techniques to make one thread wait for another. Most likely, you are not actually creating new threads in the run method, and we should overlook this comment. But if you are creating new threads there, you should probably say some more about how the threads are being used. Does the run() method need to wait for any results from the child threads, or can run() just ignore the child threads?
"I'm not back." - Bill Harding, Twister
Joined: Nov 14, 2005
Jim, Thanks for your response.
Here is what I have to do. for every 10 mins, my task should read the data (its going to be large file) and process it. So now I can read that file in the Task run method. but, to process that data I am using multi threaded (Reader- workers)program. using the reader thread, I should read the file (certain number of records) and put those records in to the queues(vectors) to be processes. once reader is done placing the data in queues, then workers threads come in and process. Once workers done, again reader thread start reading the data from the large file and places in the queues...so this will goes on till the large file is done.
so after next schedule, my task run gets the data into the large file again which will be processed by previous reader-workers threads. this is cycle...
Now my solution is in the Task run method, create reader and workers threads and use the join to make sure those threads finishes before Task run method ends so that way I am controlling my threads creation (I dont have keep on create new thread) and every thread needs a database connection so I dont want mess my DB server as well.
Hope this design will serve my purpose.
Please put any ideas you have to make it better.
Joined: Jan 30, 2000
That sounds good. You may be able to make some improvements to allow greater concurrency (and therefore less processing time, hopefully).
[Ugender]: once reader is done placing the data in queues, then workers threads come in and process.
It sounds like maybe the workers don't start at all until all the reading is done. I would probably start the workers just before I started the reader putting data onto the queues. That way workers can start processing as soon as the first data is available from the file; they don't have to wait until the whole file has been read.
[Ugender]: Now my solution is in the Task run method, create reader and workers threads and use the join to make sure those threads finishes before Task run method ends so that way I am controlling my threads creation (I dont have keep on create new thread) and every thread needs a database connection so I dont want mess my DB server as well.
It sounds like the run() method creates all the threads and then waits for them all to complete. Does it also have to create a new DB connection for each new thread? I would look into using a thread pool (e.g. JDK 5's Executors.newFixedThreadPool()) and a DataSource configured for connection pooling, so you can re-use existing resources rather than creating them anew each time.
Also, by using join() this way, that means that if even one worker is still busy processing data, you can't start reading data for the next job. An alternative might be to remove the join() code, and instead use a BlockingQueue (e.g. LinkedBlockingQueue) for the data. I'd try to make sure the capacity of the queue is at least large enough for run of the reader - so for a single run, the queue will work just like it does now. But if it takes more than ten minutes to process each run, the queue will gradually grow until it reaches its capacity. When that happens, the reader (in the Task's run() method) will block until the queue has more space available; in this way, further input will slow down until the readers catch up.
Here I'm assuming that when things get busy, you still need to process all the data that has been read. If that's not the case, and you only need to process the most recent data from the file, then alternate implementations are possible. You might simply clear the queue again at the start of each read cycle. Or for that matter, the current solution using join() would work well there, I think. [ October 13, 2007: Message edited by: Jim Yingst ]
Joined: Nov 14, 2005
Thanks a lot for your time and valuable inputs.
Yes, I started with what you suggested about the reader thread should start as soon as it finds space in queues instead of waiting till the end of queues getting empty. But in the initial stage, we wanted to test all the data is getting processed so now we are waiting on all the worker threads to finish before reader thread starts. that way probably better chance to see which worker thread waiting and kind of record making it wait. once we get more confidence on the program, then we move to the first idea as you suggested.
As per the DB connection, yes I am using the connection pool so that should take care of the connection sharing thing.
I too wanted to use Executor and all new packages in jdk 5, but sadly I have to stick to jdk 1.4.2.
I do more research on blocked queue method that you are suggesting. As of now, I don't exactly know how to implement it.