Win a copy of Design for the Mind this week in the Design forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

JMS messages not consumed after consuming some messages

 
M. Phaneendra Vijay
Greenhorn
Posts: 26
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am using JBoss 5.1 JMS Queues. After processing certain no. of messages, say after a 50,000 messages, the messages in the queue are not consumed. Then the delivering count stays the same and the messaging count goes on increasing. I get around 30 to 40 messages per second. I searched it on google and did the following
1) close the queueSession and queueConnection.
2) changed to mysql db from hsqldb.
3) increased the transaction timeout.

Even after doing all these, my messages are still getting stuck in the queue.

The application is like this.

A c++ application sends the messages over socket. The java converter application receives the messages and puts them in a blocking queue. And some 20 threads access the blocking queue (receivedQ) and puts them in RHInputQueue (Queue). The BL application uses an MDB (ProcessRequest) to consume those messages from RHInputQueue, does some manipulations on the message and puts that message in the RHOutputQueue (Queue). The converter application then uses another MDB to consume those messages in the RHOutputQueue and puts them in another blocking queue. And one thread accesses that blocking queue (processedQ) and sends them over the socket to the c++ application.

Here [Attached] is the thread dump from the jboss jmx console after it stopped processing the messages from the RHInputQueue

Thanks in advance.
Filename: threaddump.zip
Description:
File size: 18 Kbytes
[Download threaddump.zip] Download Attachment
 
Paul Clapham
Sheriff
Pie
Posts: 20958
31
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Here's an entry from our FAQ explaining why asking people to download and unzip something to look at a text file which you could have pasted into your post is a bad idea: PostTextNotScreenshots
 
Peter Johnson
author
Bartender
Posts: 5852
7
Android Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Adding a 4000 line thread dump to the text of your post is a sure way to have people ignore your post entirely. I have, instead, turned it into an attachment.

Having said that, I am not willing to dig through those 4000 lines, even attached. You need to pare the thread dump down to the few threads that are not simply waiting for user input. Threads waiting for user input usually have a short stack trace (less than a dozen lines), and usually have a wait of some kind as the first line. That leaves the thread dumps that are lengthy (many lines in the stack trace). You need to filter out the ones that are actually doing work, and the best way to do that is to take 2 or 3 thread dump about 10 to 15 seconds apart. Look for threads that seem to be doing the same thing in each dump.

Having said that, unless you find that the message readers are hung (possibly in infinite loops), then I doubt that the thread dump will reveal the cause of the hanging messages.
 
Peter Johnson
author
Bartender
Posts: 5852
7
Android Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Actually, Paul, blame me not the OP. But I still say that a 4000 line text file is not something that works well as part of the text of a post.
 
M. Phaneendra Vijay
Greenhorn
Posts: 26
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Sorry for posting such a huge message and thanks for the reply.

I have checked the thread dump and removed the threads that are in RUNNABLE, WAITING and TIMED_WAITING states.
I found that the thread that calls the onMessage is blocked and it is locked on an address. What does it imply and how should I proceed further?

Thanks.
 
Peter Johnson
author
Bartender
Posts: 5852
7
Android Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
But did you take multiple thread dumps spaced 10-15 seconds apart to see if those threads changed? Some of the threads appear to be caught in the middle of doing work, others are waiting on a lock. Are they still doing the same things in 10 seconds?

Also, it looks like your onMessage method also sends a message. Hopefully it is not sending the message back to the same queue...
 
Paul Clapham
Sheriff
Pie
Posts: 20958
31
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Peter Johnson wrote:Actually, Paul, blame me not the OP. But I still say that a 4000 line text file is not something that works well as part of the text of a post.


Absolutely. I agree. However I didn't look at the file before I posted that comment.
 
M. Phaneendra Vijay
Greenhorn
Posts: 26
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The thread dump that I have checked, is for the last time (Oct 2nd) that the messages were not consumed from the queue (RHInputQueue). It is actually a production system. Till now, all the messages are consumed from the queue (RHInputQueue) properly. When it stops consuming, I will then take some 3 to 4 thread dumps with a gap of 10 to 15 seconds and analyze that.

And yes, the onMessage method processes the incoming message from RHInputQueue and sends it to another queue (RHOutputQueue).
 
M. Phaneendra Vijay
Greenhorn
Posts: 26
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I have checked the thread dumps with a gap of 15 seconds. I don't see anywhere, where the thread is actually blocked or in deadlock.
Could you please help?

I am unable to attach files. It is giving an error saying "Files with the extension .zip are not allowed as attachment in the message."
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic