ActiveMQ 5.2.0 + REST + HTTP POST = java.lang.OutOfMemoryError

[Richard Loth]

First off, I am a newbie when it comes to JMS & ActiveMQ.

I have been looking into a messaging solution to serve as middleware for a message producer that will insert XML messages into a queue via HTTP POST. The producer is an existing system written in C++ that cannot be modified (so Java and the C++ API are out).

Using the "demo" examples and some trial and error, I have cobbled together a working example of what I want to do (on a windows box).

The web.xml I configured in a test directory under "webapps" specifies that the HTTP POST messages received from the producer are to be handled by the MessageServlet.

I added a line for the text app in "activemq.xml" ('ow' is the test app dir):

<webAppContext contextPath="/ow" resourceBase="${activemq.base}/webapps/ow" logUrlOnStart="true"/>

I created a test script to "insert" messages into the queue which works well.

The problem I am running into is that it as I continue to insert messages via REST/HTTP POST, the memory consumption and thread count used by ActiveMQ continues to rise (It happens when I have timely consumers as well as slow or non-existent consumers).

When memory consumption gets around 250MB's and the thread count exceeds 5000 (as shown in windows task manager), ActiveMQ crashes and I see this in the log:

Exception in thread "ActiveMQ Transport Initiator: vm://localhost#3564" java.lang.OutOfMemoryError: unable to create new native thread

It is as if Jetty is spawning a new thread to handle each HTTP POST and the thread never dies.

I did look at this page:

http://activemq.apache.org/javalangoutofmemory.html

and tried <memoryUsage limit="1 mb"/> but that didn't fix the problem (although I didn't fully understand the implications of the change either).

Does anyone have any ideas?

Thanks!

- Bruce Loth

PS - I included the "test message producer" python script/java program below for what it is worth. I created batches of 100 messages and continued to run the script manually from the command line while watching the memory consumption and thread count of ActiveMQ in task manager.

def foo():
    import httplib, urllib

    body = "<?xml version='1.0' encoding='UTF-8'?>\n \
    <ROOT>\n \
        [snip: xml deleted to save space]
    </ROOT>"

    headers = {"content-type": "text/xml",
               "content-length": str(len(body))}

    conn = httplib.HTTPConnection("127.0.0.1:8161")
    conn.request("POST", "/ow/message/RDRCP_Inbox?type=queue", body, headers)
    response = conn.getresponse()
    print response.status, response.reason
    data = response.read()
    conn.close()
## end method definition


## Begin test code
count = 0;

while(count < 100):
    # Test with batches of 100 msgs
    count += 1
    foo()

import java.io.*;
import java.net.*;

public class RestSender {

private static final String xmlstr = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\n" +
"<Root>\n" +
    [snip: XML deleted to save space]
"</Root>";

    public RestSender()
    {
    }

    public static void sendXML(int totalMsgs) throws Exception
    {
        int count = 0;
        try {
            while (count <= totalMsgs) {
                count++;
                // Send data
                URL u = new URL("http://127.0.0.1:8161/ow/message/RDRCP_Inbox?type=queue"); 
                HttpURLConnection conn = (HttpURLConnection) u.openConnection();
                conn.setDoOutput(true);
                OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
                wr.write(xmlstr);
                wr.flush();
                
                // Get the response
                BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
                String line;
                while ((line = rd.readLine()) != null) {
                    // Process line...
                }
                wr.close();
                rd.close();
                conn.disconnect();
            } // while
        }
        catch ( Exception e ) {
            e.printStackTrace();
        }
    }

    public static void main(String[] args) 
    {
        final int totalMsgs = 100;
        int count = 1;
        try {
            while (count <= totalMsgs) {
                count++;
                sendXML(totalMsgs);
            }
        }
        catch ( Exception e ) {
            e.printStackTrace();
        }
    }
}

[Paul Clapham]

Well, after reading that several times it occurred to me that what you were posting there was the client code, and your problem is that the crash is on the server side. Am I right?

It's true that if you don't terminate your connections to JMS queues properly, then you'll get this "out of memory" problem which really isn't "out of memory", it's "out of threads". It happened to me too. But if the server isn't doing that right, then there isn't much you can do from the client side to fix it. You would be better off trying to get the server code fixed.

[Richard Loth]

Paul - The code I posted was just to show folks what I had tested. I was hoping that this was a "configuration" problem. I don't think that there is anything wrong with the code I used for the test ... it is pretty straight forward, simple, and was solely intended for testing concepts and the scalability of ActiveMQ. Yeah, it is ActiveMQ that crashes.

If I have to get in a futz with ActiveMQ server code then I'll probably jump ship to some other solution. Incidentally, I feel like the problem is w/ Jetty. If I write a message producer using JAVA/ & JMS, ActiveMQ works like a champ (I threw 200,000 messages at it w/out any problems). I have looked at the source for MessageServlet but I really don't want to spent the time screwing with it or Jetty.

- Bruce

[Paul Clapham]

Richard Loth wrote:Incidentally, I feel like the problem is w/ Jetty.

Based on what you've posted, I would tend to agree with you. And I'm not convinced that you can blame yourself for missing something in the configuration -- there shouldn't be a configuration option to close sessions properly or not to close them properly.

[Richard Loth]

OK ... I coded a small servlet that will process an HTTP POST message and deployed it to webapps directory for the instance of Jetty that is included with ActiveMQ 5.2.0. The servlet doesn't do anything other than read the body of the message I send and echo it back. I can hit that servlet with 1,500 rapid, sequential HTTP POST request messages and the memory consumption and thread count do not budge. So the problem might be with the MessegeServlet class that I was using. This servlet is included with ActiveMQ and implements a REST-ful API to messaging.

I'll have to take a look at the source for the servlet but I am tempted to simply write my own to interface with the ActiveMQ broker. I don't need most of the functionality that the MessageServlet provides and at least I'll understand my own work better.

[Jim McCabe]

I don't think this is just Jetty. For example see this link for someone who saw the same thing in JBoss:

http://osdir.com/ml/java.activemq.user/2005-06/msg00257.html

From these memory/thread leak reports, and from the lack of documentation, I get the impression that the MessageServlet was a quick proof-of-concept and is not being used in any real production environments.

[Richard Loth]

Jim McCabe wrote:From these memory/thread leak reports, and from the lack of documentation, I get the impression that the MessageServlet was a quick proof-of-concept and is not being used in any real production environments.

That could well be the case. I ended up writing my own small web application (servlet + helper classes) from scratch that receives messages as HTTP POST request messages and inserts them into an ActiveMQ message queue. The application is deployed to the embedded Jetty servlet container and my experiences with ActiveMQ 5.2.0 and Jetty have been positive thus far. The problems with runaway memory consumption and an ever increasing thread count have not manifested despite rigorous testing.

I glanced briefly at the source code for the "MessageServlet" that is provided with the "examples" included in the ActiveMQ 5.2.0 download but never tried to deconstruct it and have no idea why it exhibits the behavior that it does.

- RBL

PS - Btw, I posted the original question to two other messages groups (Stack Overflow & the ActiveMQ discussion board) and got zero responses. Makes me wonder how folks are using ActiveMQ ... I guess it isn't the way I am using it.