aspose file tools*
The moose likes JBoss/WildFly and the fly likes JMS messages not consumed after consuming some messages Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Java 8 in Action this week in the Java 8 forum!
JavaRanch » Java Forums » Products » JBoss/WildFly
Bookmark "JMS messages not consumed after consuming some messages" Watch "JMS messages not consumed after consuming some messages" New topic
Author

JMS messages not consumed after consuming some messages

M. Phaneendra Vijay
Greenhorn

Joined: Jun 10, 2009
Posts: 26
I am using JBoss 5.1 JMS Queues. After processing certain no. of messages, say after a 50,000 messages, the messages in the queue are not consumed. Then the delivering count stays the same and the messaging count goes on increasing. I get around 30 to 40 messages per second. I searched it on google and did the following
1) close the queueSession and queueConnection.
2) changed to mysql db from hsqldb.
3) increased the transaction timeout.

Even after doing all these, my messages are still getting stuck in the queue.

The application is like this.

A c++ application sends the messages over socket. The java converter application receives the messages and puts them in a blocking queue. And some 20 threads access the blocking queue (receivedQ) and puts them in RHInputQueue (Queue). The BL application uses an MDB (ProcessRequest) to consume those messages from RHInputQueue, does some manipulations on the message and puts that message in the RHOutputQueue (Queue). The converter application then uses another MDB to consume those messages in the RHOutputQueue and puts them in another blocking queue. And one thread accesses that blocking queue (processedQ) and sends them over the socket to the c++ application.

Here [Attached] is the thread dump from the jboss jmx console after it stopped processing the messages from the RHInputQueue

Thanks in advance.


[Download threaddump.zip] Download

Paul Clapham
Bartender

Joined: Oct 14, 2005
Posts: 18155
    
    8

Here's an entry from our FAQ explaining why asking people to download and unzip something to look at a text file which you could have pasted into your post is a bad idea: PostTextNotScreenshots
Peter Johnson
author
Bartender

Joined: May 14, 2008
Posts: 5779
    
    7

Adding a 4000 line thread dump to the text of your post is a sure way to have people ignore your post entirely. I have, instead, turned it into an attachment.

Having said that, I am not willing to dig through those 4000 lines, even attached. You need to pare the thread dump down to the few threads that are not simply waiting for user input. Threads waiting for user input usually have a short stack trace (less than a dozen lines), and usually have a wait of some kind as the first line. That leaves the thread dumps that are lengthy (many lines in the stack trace). You need to filter out the ones that are actually doing work, and the best way to do that is to take 2 or 3 thread dump about 10 to 15 seconds apart. Look for threads that seem to be doing the same thing in each dump.

Having said that, unless you find that the message readers are hung (possibly in infinite loops), then I doubt that the thread dump will reveal the cause of the hanging messages.


JBoss In Action
Peter Johnson
author
Bartender

Joined: May 14, 2008
Posts: 5779
    
    7

Actually, Paul, blame me not the OP. But I still say that a 4000 line text file is not something that works well as part of the text of a post.
M. Phaneendra Vijay
Greenhorn

Joined: Jun 10, 2009
Posts: 26
Sorry for posting such a huge message and thanks for the reply.

I have checked the thread dump and removed the threads that are in RUNNABLE, WAITING and TIMED_WAITING states.
I found that the thread that calls the onMessage is blocked and it is locked on an address. What does it imply and how should I proceed further?

Thanks.
Peter Johnson
author
Bartender

Joined: May 14, 2008
Posts: 5779
    
    7

But did you take multiple thread dumps spaced 10-15 seconds apart to see if those threads changed? Some of the threads appear to be caught in the middle of doing work, others are waiting on a lock. Are they still doing the same things in 10 seconds?

Also, it looks like your onMessage method also sends a message. Hopefully it is not sending the message back to the same queue...
Paul Clapham
Bartender

Joined: Oct 14, 2005
Posts: 18155
    
    8

Peter Johnson wrote:Actually, Paul, blame me not the OP. But I still say that a 4000 line text file is not something that works well as part of the text of a post.


Absolutely. I agree. However I didn't look at the file before I posted that comment.
M. Phaneendra Vijay
Greenhorn

Joined: Jun 10, 2009
Posts: 26
The thread dump that I have checked, is for the last time (Oct 2nd) that the messages were not consumed from the queue (RHInputQueue). It is actually a production system. Till now, all the messages are consumed from the queue (RHInputQueue) properly. When it stops consuming, I will then take some 3 to 4 thread dumps with a gap of 10 to 15 seconds and analyze that.

And yes, the onMessage method processes the incoming message from RHInputQueue and sends it to another queue (RHOutputQueue).
M. Phaneendra Vijay
Greenhorn

Joined: Jun 10, 2009
Posts: 26
I have checked the thread dumps with a gap of 15 seconds. I don't see anywhere, where the thread is actually blocked or in deadlock.
Could you please help?

I am unable to attach files. It is giving an error saying "Files with the extension .zip are not allowed as attachment in the message."
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: JMS messages not consumed after consuming some messages
 
Similar Threads
Persistent Messages, What heppens if a system is started!
How to kick start the DLQ
A question about clustering and jms
Jboss | MDB| Messages nt consumed
JMS queries