This week's giveaway is in the EJB and other Java EE Technologies forum.
We're giving away four copies of EJB 3 in Action and have Debu Panda, Reza Rahman, Ryan Cuprak, and Michael Remijan on-line!
See this thread for details.
The moose likes Other Open Source Projects and the fly likes ActiveMQ 5.2.0 + REST + HTTP POST = java.lang.OutOfMemoryError Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of EJB 3 in Action this week in the EJB and other Java EE Technologies forum!
JavaRanch » Java Forums » Products » Other Open Source Projects
Bookmark "ActiveMQ 5.2.0 + REST + HTTP POST = java.lang.OutOfMemoryError" Watch "ActiveMQ 5.2.0 + REST + HTTP POST = java.lang.OutOfMemoryError" New topic
Author

ActiveMQ 5.2.0 + REST + HTTP POST = java.lang.OutOfMemoryError

Richard Loth
Greenhorn

Joined: May 01, 2009
Posts: 4
First off, I am a newbie when it comes to JMS & ActiveMQ.

I have been looking into a messaging solution to serve as middleware for a message producer that will insert XML messages into a queue via HTTP POST. The producer is an existing system written in C++ that cannot be modified (so Java and the C++ API are out).

Using the "demo" examples and some trial and error, I have cobbled together a working example of what I want to do (on a windows box).

The web.xml I configured in a test directory under "webapps" specifies that the HTTP POST messages received from the producer are to be handled by the MessageServlet.

I added a line for the text app in "activemq.xml" ('ow' is the test app dir):

<webAppContext contextPath="/ow" resourceBase="${activemq.base}/webapps/ow" logUrlOnStart="true"/>

I created a test script to "insert" messages into the queue which works well.

The problem I am running into is that it as I continue to insert messages via REST/HTTP POST, the memory consumption and thread count used by ActiveMQ continues to rise (It happens when I have timely consumers as well as slow or non-existent consumers).

When memory consumption gets around 250MB's and the thread count exceeds 5000 (as shown in windows task manager), ActiveMQ crashes and I see this in the log:

Exception in thread "ActiveMQ Transport Initiator: vm://localhost#3564" java.lang.OutOfMemoryError: unable to create new native thread

It is as if Jetty is spawning a new thread to handle each HTTP POST and the thread never dies.

I did look at this page:

http://activemq.apache.org/javalangoutofmemory.html

and tried <memoryUsage limit="1 mb"/> but that didn't fix the problem (although I didn't fully understand the implications of the change either).

Does anyone have any ideas?

Thanks!

- Bruce Loth

PS - I included the "test message producer" python script/java program below for what it is worth. I created batches of 100 messages and continued to run the script manually from the command line while watching the memory consumption and thread count of ActiveMQ in task manager.




Paul Clapham
Bartender

Joined: Oct 14, 2005
Posts: 18120
    
    8

Well, after reading that several times it occurred to me that what you were posting there was the client code, and your problem is that the crash is on the server side. Am I right?

It's true that if you don't terminate your connections to JMS queues properly, then you'll get this "out of memory" problem which really isn't "out of memory", it's "out of threads". It happened to me too. But if the server isn't doing that right, then there isn't much you can do from the client side to fix it. You would be better off trying to get the server code fixed.
Richard Loth
Greenhorn

Joined: May 01, 2009
Posts: 4
Paul - The code I posted was just to show folks what I had tested. I was hoping that this was a "configuration" problem. I don't think that there is anything wrong with the code I used for the test ... it is pretty straight forward, simple, and was solely intended for testing concepts and the scalability of ActiveMQ. Yeah, it is ActiveMQ that crashes.

If I have to get in a futz with ActiveMQ server code then I'll probably jump ship to some other solution. Incidentally, I feel like the problem is w/ Jetty. If I write a message producer using JAVA/ & JMS, ActiveMQ works like a champ (I threw 200,000 messages at it w/out any problems). I have looked at the source for MessageServlet but I really don't want to spent the time screwing with it or Jetty.

- Bruce
Paul Clapham
Bartender

Joined: Oct 14, 2005
Posts: 18120
    
    8

Richard Loth wrote:Incidentally, I feel like the problem is w/ Jetty.


Based on what you've posted, I would tend to agree with you. And I'm not convinced that you can blame yourself for missing something in the configuration -- there shouldn't be a configuration option to close sessions properly or not to close them properly.
Richard Loth
Greenhorn

Joined: May 01, 2009
Posts: 4
OK ... I coded a small servlet that will process an HTTP POST message and deployed it to webapps directory for the instance of Jetty that is included with ActiveMQ 5.2.0. The servlet doesn't do anything other than read the body of the message I send and echo it back. I can hit that servlet with 1,500 rapid, sequential HTTP POST request messages and the memory consumption and thread count do not budge. So the problem might be with the MessegeServlet class that I was using. This servlet is included with ActiveMQ and implements a REST-ful API to messaging.

I'll have to take a look at the source for the servlet but I am tempted to simply write my own to interface with the ActiveMQ broker. I don't need most of the functionality that the MessageServlet provides and at least I'll understand my own work better.
Jim McCabe
Greenhorn

Joined: Mar 24, 1999
Posts: 8
I don't think this is just Jetty. For example see this link for someone who saw the same thing in JBoss:

http://osdir.com/ml/java.activemq.user/2005-06/msg00257.html

From these memory/thread leak reports, and from the lack of documentation, I get the impression that the MessageServlet was a quick proof-of-concept and is not being used in any real production environments.
Richard Loth
Greenhorn

Joined: May 01, 2009
Posts: 4
Jim McCabe wrote:From these memory/thread leak reports, and from the lack of documentation, I get the impression that the MessageServlet was a quick proof-of-concept and is not being used in any real production environments.


That could well be the case. I ended up writing my own small web application (servlet + helper classes) from scratch that receives messages as HTTP POST request messages and inserts them into an ActiveMQ message queue. The application is deployed to the embedded Jetty servlet container and my experiences with ActiveMQ 5.2.0 and Jetty have been positive thus far. The problems with runaway memory consumption and an ever increasing thread count have not manifested despite rigorous testing.

I glanced briefly at the source code for the "MessageServlet" that is provided with the "examples" included in the ActiveMQ 5.2.0 download but never tried to deconstruct it and have no idea why it exhibits the behavior that it does.

- RBL

PS - Btw, I posted the original question to two other messages groups (Stack Overflow & the ActiveMQ discussion board) and got zero responses. Makes me wonder how folks are using ActiveMQ ... I guess it isn't the way I am using it.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: ActiveMQ 5.2.0 + REST + HTTP POST = java.lang.OutOfMemoryError
 
Similar Threads
Should Threads performing independent tasks sleep?
puzzled with non-static synchronized method accessing static field
producer/consumer problem
I'm trying to send XML to a servlet but the input is always null
JMS: Problem with Consumer Program