We are working on a multi-threaded server based application. Currently it is written to:
1. Listen for an MQ message. The listner runs on it's own (explicitly called thread at startup), and sleeps until the process is killed.
2. OnMessage calls a message processor class that gets called on it's own explicitly called thread. (Listner thread spawns child threads)
3. During this message processing, we need to read/write to file system directories. Some threads will be writing to the same directories, and sometimes files, as other threads, at times.
4. Right before calling the message processor, and right after the message processor class is complete, a message is sent to another queue which a log listner is listening to. This log listner spawns new threads for each message it needs to log, and inserts the information into an oracle database table.
My questions are:
1. Should I synchronize the blocks where each thread needs to read/write to a directory, and insert/update the db table? Or, will the next thread automaticly wait for the previous thread to finish?
2. Will threads that die due to exception cause others to not execute?
3. If so, how can I make sure that threads that encounter an exception don't prevent other threads that are spawned aftef from completing their work?
For #1 and #2 ... Do you do anything to prevent a flood of incoming messages from overloading you with threads? I'd suggest a fixed number of threads that read messages from the queue or a single reader and a fixed size thread pool. The queueing system is already a great buffer to hold incoming messages until you can get to them, so reading them and queuing them in memory in a thread pool seems wasteful.
For #3 ... Two threads writing to the same file could be trouble for sure. I'm not sure what will happen. Maybe the first one starts writing and the second one fails to get an open for write. If so the second could wait a second and try again. Or you might try to build a mechanism with an object representing each file and code synchronized on that object. I'd be reluctant to try that - many ways to go wrong. The database writes should be no problem if you can use transactions to assure ACID operations.
For #4 ... A thread per message on logging also sounds like overkill. You're offloading logging to another thread to help your main processes run quickly; don't set up a zillion threads to compete with the main process! If the log queue listener is single threaded and gets behind some times, what's the worst thing that happens? Queue depth goes up a bit and the log is seconds out of date. The log IO is likely to be the slowest part of that operation, so if you synchronize all those threads to prevent overlapping writes you'll be nearly as slow as a single thread.
Any of that help?
A good question is never answered. It is not a bolt to be tightened into place but a seed to be planted and to bear more seed toward the hope of greening the landscape of the idea. John Ciardi
Joined: Jan 23, 2004
Yes, that is all very helpful.
We do not do anything to prevent a flood of imcoming message on threads. You mentioned a fixed number of threads; is that not the same as a thread pool? If not, how do I go about setting that value? If so, when you say
queuing them in memory in a thread pool seems wasteful
, do you mean I should leave it as is?
For the file manipulation, can I use wait()/notify() without using synchroziation? Or is that also not recommended?
I think you are right about the log queue threads. I will change that.
Joined: Jan 29, 2003
A fixed number of listeners would be my first choice. Each would be on a long-lived thread that starts at startup and runs all day long. Messages wait in the message queue until a listener is ready to pick them up.
A fixed number of handler threads in a pool would be my second choice. If the listeners pull messages faster than the handlers can process them they'll pile up in the thread pool's collection of runnables. I'd rather have them pile up outside my JVM memory in the message queue where they'll survive a hard shutdown on the server.
An uncontrolled number of threads might be ok if you're confident the message senders won't bury you. That fits the whole "do the simple thing until you can prove it doesn't work" idea.
It would be nice to have some monitoring on all this so you can see how fast msgs are coming in, how many threads you have going, etc. We use a very expensive tool called Wily Introscope for that kind of thing.
Joined: Jun 06, 2006
Apart from the suggestions offered by the others ...
Why would you want to make your code so difficult for others to debug i.e by spawning so many threads?
Simplify things ..... assign one thread per functionality... This way you can also configure how many threads you want per functionality etc ( Assumption:- different functionality means less sync problems across threads...Have different log files(Use some standard logger) ,different directories to write to)
>>Should I synchronize the blocks where each thread needs to read/write to a directory, and insert/update the db table? ------ Database tables have a low level lock where the entire table gets locked for one insert... So you dont have to worry if you are making inserts ... but if you are making updates based on the current snapshot of the table ... yu have a problem...
>> Will threads that die due to exception cause others to not execute? Handle exceptions appropriately localised to threads and make sure they dont kill the parent theread that is spawning them [ June 06, 2006: Message edited by: Lakshmi Ashok ]
Joined: Jan 23, 2004
For the file writing part, we aren't writing to logs for this requirement. We are writing uploaded static files to web server docroots. In this case, sometimes the process will be writing to the same directory, and possibly the same file, if 2 or more users are publishing pages. Here, I was wondering if this block of writing to the file system should be syncronized.
Also, can anyone advise what maybe be happening in this issue I'm having?
One of the listeners resides on a server without WebSphere. It only has JRE 1.4.2 installed. We start the listener by calling a startup java class with a shell script. Once started, it seems to get the first few messages, but then it doesn't get others. However, if I restart the process, it will pick up all the waiting messages that it wasn't picking up before the restart. Any ideas of what could be causing the listener not to pickup messages?
Thank you for all your suggestions. Now I don't feel so alone & abandoned with this project!