Help. So I'm writing a network application that receives live streaming data from another server. The other server (not in my control) sends it via serialized objects and, I believe, my only way of reading these is to use the ObjectInputStream and ObjectOutputStream. My issue is that I'm getting a large number of objects over time and I notice two things
a) there seems to be some sort of bottleneck. the time it takes for the CPU to process an object takes longer.
b) i eventually hit the OOM error. increasing the memory for the application using Xmx only prolongs the problem.
my code looks something like this:
This is very simple code. The usedMemory() just uses the Runtime.getRuntime() class.The main part is inside the A() function where it does obj = inputStream.readObject(). If I run the code with that line commented out, I notice that my heap size always stays the same (pretty obvious). But if I keep that line, the memory size keeps increasing over time. I stripped the code from all the functionality linked to processing that object. Even then, the size keeps increasing.
This is the OOM memory I get:
I believe that the bottleneck I referred to earlier probably started happening when Java tries to re-size the HashTable (?).
People have suggested closing and re-opening the stream but I can't re-initialize it each time. I'll lose data and time.
Is there any way I can use a different I/O object that's much more efficient at this?
The ObjectOutputStream maintains a table mapping objects written into the stream to a handle. The first time an object is written to a stream, its contents are written into the stream; subsequent writes of the object result in a handle to the object being written into the stream. This table maintains references to objects that might otherwise be unreachable by an application, thus, resulting in an unexpected situation of running out of memory. A call to the ObjectOutputStream.reset() method resets the object/handle table to its initial state, allowing all previously written objects to be elgible for garbage collection. See handle.
No my actual code did do stuff with the object. What I'm saying is that, even with all that removed, and just setting it to null, the heap size just keeps increasing. Doing a System.gc() call doesn't seem to help either. If I can't cut down the heap size when I'm not even doing anything with the object, how can I expect to solve this problem after attaching functionality to it.
I tried something similar to your code, a server which writes a Date to a socket and a client which reads the date from the socket. I used the JConsole application included with the JDK to monitor the client. As you can see from this 20-minute run, there is no trend in heap consumption (keep in mind that the scale of the graph is .8 to 2 Mb, so we're talking about miniscule variations in heap)
The code for the client side looks much like your own:
So there's got to be something else going on in your code. Can you show us a stand-alone example which demonstrates the behavior you are seeing?
Joined: Feb 03, 2009
First I gotta say this forum is awesome. You guys are really helpful (especially Joe) and quick.
So I used the code above and changed the socket information to mine. Before my program got to this method, it initialized the socket, input/output streams and sent commands to the output stream for logging in.
I don't know how to use JConsole, but I saved the output of the free/total memory. This is the output I got after a few minutes (<5 minutes). The total memory keeps increasing over time. This screws my latency (can't see that in the output) because, I believe, JVM is trying to allocate more memory and storing/retrieving objects is taking a while.
Right now (10-15 mins later), total memory is at 33,415,168.
ryan atw wrote:First I gotta say this forum is awesome. You guys are really helpful (especially Joe) and quick.
Hey, how often do you call A()? Is there a chance that you are creating numerous instances and they're all hanging around, blocked on the socket waiting for input? That would account for the memory being used. Try adding an instance counter and printing creation and destruction of that anonymous class.
Joined: Feb 03, 2009
I won't be able to run/test the code until next Monday (no streaming data until then) so I'll show you what my code looks like instead.
You are leaving some code out, like initializing the Socket and ObjectOutputStream. I tried using your code for A() in my TimeClient posted above. The JConsole session for that looked exactly like the one I posted for my previous client. No memory leak.
You can dummy up a server to provide you with some data pretty easily. My server is all of 47 lines of code.
Joined: Feb 03, 2009
That's what I have in my login function. I can't reveal much code because I'm using a proprietary API from another vendor. My guess is that it's their server code that's the fault?