aspose file tools*
The moose likes Java in General and the fly likes Processing lots of JMS messages Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Java in General
Bookmark "Processing lots of JMS messages" Watch "Processing lots of JMS messages" New topic
Author

Processing lots of JMS messages

Mark Herschberg
Sheriff

Joined: Dec 04, 2000
Posts: 6037
This questions involves J2EE technology threading, and performance issues. So I'm posting here.
We hava a Java client which is reciving lots of JMS messages (on a topic) from the server. It receives anywhere from 3-30 messages per second. The messages may take, say 100ms each to process (roughly).
We've noticed that the client lags behind the server. That is, when the server stops sending messages, the client is still actively processing them for seconds to minutes afterwards.
We are using BEA Weblogic 7.0 as the JMS prodiver, running on Win2000. The clients are running JDK 1.4.1 on Win2000 and on XP. They are reasonably fast machines.
We think what is happening is that the thread in which the JMS reciver runs (i.e. the thread running the onMessage method), then gets delayed processing the message and so won't listen for the next JSM message until it is infinished.
What is the best way to handle this situation?
1) Spawn a new thread
We could have the onMessage method spawn a new thread (probably a SwingWorker thread since it ultimately does update the GUI), as soon as it gets the message. This runs in the background and frees the receiver's thread to listen for the next message.
2) Create multiple receivers
IIRC, I can set it up so I have multiple instances all listening on the same topic. These can basically take turns pulling items off the topic.
3) Other
Any other ideas?
I recognize that there may be an issue where we simply recieve more information then we can process in a second. I'm not yet certain that is the case. If it is, I'll have to find a way to better filter the information either client side or server side, but this base problem will still remain.
--Mark
Michael Morris
Ranch Hand

Joined: Jan 30, 2002
Posts: 3451
My first instinct would be to go with option 1, but I don't think I would necesarilly want to spawn one thread per message which would mean using a thread pool dedicated to the task. You could just keep a collection of pending messeages in the thread pool manager and as worker threads became idle assign them a message to process. That should make onMessage() relatively quick and it should have no trouble keeping up with incoming messages.


Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius - and a lot of courage - to move in the opposite direction. - Ernst F. Schumacher
Jim Yingst
Wanderer
Sheriff

Joined: Jan 30, 2000
Posts: 18671
My thoughts were pretty much what Michael just said. But my knowledge of both WebLogic and JMS is pretty limited - it wouldn't surprise me if it turned out that you're not supposed to spawn new threads yourself, or maybe the system is already doing this for you somewhere.
We think what is happening is that the thread in which the JMS reciver runs (i.e. the thread running the onMessage method), then gets delayed processing the message and so won't listen for the next JSM message until it is infinished.
If you're not certain (and I certainly am not) it may be worthwhile to test this assumption. Perhaps you can just log the entry and exit times when onMessage() is executed, to see if each message is indeed being processed serially or not. Maybe also use Thread.currentThread().toString() to see what you can learn.
Hmmm, I just googled for docs on this, and came up with
http://www.weblogic.com/docs45/classdocs/API_jms.html#3serverpool
which sounds as though WebLogic provides a framework for multiple threads to process messages concurrently. I suspect that this would be preferable to spawning your own threads.


"I'm not back." - Bill Harding, Twister
Mark Herschberg
Sheriff

Joined: Dec 04, 2000
Posts: 6037
Sorry, I should have been clearer. Yes, we did test this, and discovered that when the work function is sleep() the next message isn't processed until it's done sleeping.
A thread pool! Yes, much better idea then constantly spawning new threads. :-p
Thanks for the link Jim. Unfortunately that looks like it's designed for server side. Since we're using EJBs, I'm assuming (maybe I'm being naive) that the multiple MDB instances are effectively a thread pool.
Um, it's been a while since I've made a thread pool. I'm sure I can look it up, but one question that comes to mind, is it ok to use SwingWorker threads in a thread pool? Actually, I guess I don't need SwingWorker threads. That's really just a simple way to keep a task out of the GUI thread, right? If I'm using a thread pool, I'm already doing that. (Does this sound right?)
--Mark
[ July 08, 2003: Message edited by: Mark Herschberg ]
Michael Morris
Ranch Hand

Joined: Jan 30, 2002
Posts: 3451
Here's a simple thread pool implementation. I adapted it from Complete Java 2 Certification Study Guide .

To start it up just pass in the number of threads in the pool to the ThreadPoolManager constructor and start it in a new thread. You'll also probably want to call setDaemon(true) on its thread. You'll need to make a Runnable class to process your messages and do the work in its run() method. Then as messages come in just call threadPoolManager.addJob(messageRunner).
Stan James
(instanceof Sidekick)
Ranch Hand

Joined: Jan 29, 2003
Posts: 8791
Doug Lea's util.concurrent package has a thread pooler, too. He literally wrote the book on threading, and has had a lot of input to a future version of the JDK that will have classes very similar to these.
BTW: That business about the client not keeping up and messages queuing up is why we use a queuing messaging system in the first place, no? Of course it is pretty serious if the client never catches up and just gets further and further behind! And you absolutely want your client to run its very best and not waste time sleeping.
Let us know how the thread pool works out for your overall throughput!


A good question is never answered. It is not a bolt to be tightened into place but a seed to be planted and to bear more seed toward the hope of greening the landscape of the idea. John Ciardi
Mark Herschberg
Sheriff

Joined: Dec 04, 2000
Posts: 6037
Thank you everyone who replied.
I tried using the thread pool, so that as soon as a message comes in, it's placed in a runnable task and put into the thread pool. That didn't seem to help.
What was really wierd were the numbers I was getting from OptimizeIt. The profiler would only show 30-50% of the processor being actively used. I couldn't tack the missing CPU time (which obviously makes it hard to optimize :-p ).
I then went back to the good old days of performance debugging and started using System.currentTimeMillis(). Each message took slightly longer then a second (1020-1170ms). When I drilled down further using similar techniques, I tracked the issue to a sleeping thread. I use the JClass LiveTable, which is very similar to a JTable. When we get a message, we do the following: set the table cell with the new data, set the font to bold, wait one second, set the table font to plain. In short, we "flash" the table cell using the bold font.
I was having a problem getting this to work. Their tech support reminded me of Swing threading issues. basically, I had to update the table in a non-GUI event thread. I thought I was ok, because JMS spawns it own thread upon receiving a message. Still, I created new threads for it as follows.

This was taking all the time. If I took out the sleep, the speed was fine, as now each message took 1000ms less.
I then tried combining both color changers into one thread, which sleeps, as below.

This didn't improve performance any.
Then I had the idea to use the thread pool for this work, as opposed to for processing the whole message. This worked.
It seemed like when I was sleeping either in the JMS thread, or the thread created by SwingUtilities.invokeLater(), the scheduler didn't swap out the thread to let others run. Either that or things got queued up somehow. Can anyone explain this to me?
Thanks again for the all the help.
--Mark
Jim Yingst
Wanderer
Sheriff

Joined: Jan 30, 2000
Posts: 18671
It sounds as though you were putting all those sleep(1000) calls into the same thread, so they had no choice but to run consecutively. In your first code sample it's not clear how many threads might have been involved in calling that code, but from the behavior you describe it seems there was only one. In the second code, it doesn't actually matter how many threads there might have been originally, because you've put the sleep(1000) into the Runnable that is invoked by SwingUtils.invokeLater(). That means that Runnable will eventually be executed by the event handling thread, of which there is only one. invokeLater means you don't necessarily care when the invocation begins, but it's still going to be done by the one and only event processing thread. So if you get 500 messages, you may finish processing them in a few seconds - but the event handling queue now has approximately 500 pending one-second tasks. Which will execute one at a time.
To help understand what's going on, it may be helpful to put some logging before and after each sleep:

I believe you will see all sleeps were on the same thread - in the case of your second code, it's the event handling thread. In the first case, I dunno.
I think the key is that you want to make sure that each 1-second wait occurs in a separate thread. The set bold / set plain will need to go the the event thread, because that's the only way you're supposed to change the state of most Swing components. But the sleep(1000) part should absolutely be handled by a separate thread for each message. (Unless you need to limit the thread pool - but that will certainly impact things.

This is probably what you are doing now, but I wanted to make sure it's clear in case you're not.
A more complex but scalable solution might use just one thread to keep track of all the 1-second waits concurrently. You'd have a startoff method which is called from any thread - this immedately makes the cell bold, and schedules the cell for "plaining" later by adding an entry to the end of a list, indicating the cell number and the time at which that cell should be restored to plain text. Then you've got another thread that just keeps working its way through that list, always removing an entry from the front (the oldest entry) and waiting until the designated time has arrived, then setting the designated cell to plain. Once it finishes with that entry, discard and grab a new entry from the front of the list, and wat for that one.
Stan James
(instanceof Sidekick)
Ranch Hand

Joined: Jan 29, 2003
Posts: 8791
I just read a Robert Martin chapter (you guys are all sick of hearing me say that - I'll finish the book one day and stop) about the Active Object pattern. It's a very cute way of putting Command objects in a queue and having a processor pull them out of queue and execute them. You could put one command on the queue to bold, another to wait and another to unbold. A quick google on "active object pattern" got a zillion hits. See if it makes sense.
Other approach: Swing has Timer object just for things like blinking. The runnable that you feed to invokeLater might be able to set bold and then use Timer to fire the unbold event later.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Processing lots of JMS messages