This week's giveaway is in the Android forum.
We're giving away four copies of Android Security Essentials Live Lessons and have Godfrey Nolan on-line!
See this thread for details.
The moose likes I/O and Streams and the fly likes ObjectInputStream creating OutOfMemoryError Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Android Security Essentials Live Lessons this week in the Android forum!
JavaRanch » Java Forums » Java » I/O and Streams
Bookmark "ObjectInputStream creating OutOfMemoryError" Watch "ObjectInputStream creating OutOfMemoryError" New topic
Author

ObjectInputStream creating OutOfMemoryError

Ryan Atwood
Greenhorn

Joined: Feb 03, 2009
Posts: 15
Help. So I'm writing a network application that receives live streaming data from another server. The other server (not in my control) sends it via serialized objects and, I believe, my only way of reading these is to use the ObjectInputStream and ObjectOutputStream. My issue is that I'm getting a large number of objects over time and I notice two things
a) there seems to be some sort of bottleneck. the time it takes for the CPU to process an object takes longer.
b) i eventually hit the OOM error. increasing the memory for the application using Xmx only prolongs the problem.

my code looks something like this:


This is very simple code. The usedMemory() just uses the Runtime.getRuntime() class.The main part is inside the A() function where it does obj = inputStream.readObject(). If I run the code with that line commented out, I notice that my heap size always stays the same (pretty obvious). But if I keep that line, the memory size keeps increasing over time. I stripped the code from all the functionality linked to processing that object. Even then, the size keeps increasing.

This is the OOM memory I get:


I believe that the bottleneck I referred to earlier probably started happening when Java tries to re-size the HashTable (?).

People have suggested closing and re-opening the stream but I can't re-initialize it each time. I'll lose data and time.

Is there any way I can use a different I/O object that's much more efficient at this?

Thank you.
Joe Ess
Bartender

Joined: Oct 29, 2001
Posts: 8843
    
    7

#3 Why is OutOfMemoryError thrown after writing a large number of objects into an ObjectOutputStream?

The ObjectOutputStream maintains a table mapping objects written into the stream to a handle. The first time an object is written to a stream, its contents are written into the stream; subsequent writes of the object result in a handle to the object being written into the stream. This table maintains references to objects that might otherwise be unreachable by an application, thus, resulting in an unexpected situation of running out of memory. A call to the ObjectOutputStream.reset() method resets the object/handle table to its initial state, allowing all previously written objects to be elgible for garbage collection. See handle.


Java Serialization FAQ


"blabbing like a narcissistic fool with a superiority complex" ~ N.A.
[How To Ask Questions On JavaRanch]
Ryan Atwood
Greenhorn

Joined: Feb 03, 2009
Posts: 15
Thanks for the reply.

I do have the reset code at Line 11 and it doesn't seem to solve the increasing heap or the slow processing times.
Ernest Friedman-Hill
author and iconoclast
Marshal

Joined: Jul 08, 2003
Posts: 24183
    
  34

Is this your real code? You're resetting and flushing that ObjectOutputStream, but not actually using it for anything. You never write any data to it, and the objects are simply being discarded.

In any case, the stack trace shows an OutOfMemoryError while reading an object. Perhaps that object graph is simply too large to fit in memory. What do you know about the form of the data itself?


[Jess in Action][AskingGoodQuestions]
Ryan Atwood
Greenhorn

Joined: Feb 03, 2009
Posts: 15
No my actual code did do stuff with the object. What I'm saying is that, even with all that removed, and just setting it to null, the heap size just keeps increasing. Doing a System.gc() call doesn't seem to help either. If I can't cut down the heap size when I'm not even doing anything with the object, how can I expect to solve this problem after attaching functionality to it.

Thanks
Joe Ess
Bartender

Joined: Oct 29, 2001
Posts: 8843
    
    7

I tried something similar to your code, a server which writes a Date to a socket and a client which reads the date from the socket. I used the JConsole application included with the JDK to monitor the client. As you can see from this 20-minute run, there is no trend in heap consumption (keep in mind that the scale of the graph is .8 to 2 Mb, so we're talking about miniscule variations in heap)


The code for the client side looks much like your own:

So there's got to be something else going on in your code. Can you show us a stand-alone example which demonstrates the behavior you are seeing?
Ryan Atwood
Greenhorn

Joined: Feb 03, 2009
Posts: 15
First I gotta say this forum is awesome. You guys are really helpful (especially Joe) and quick.

So I used the code above and changed the socket information to mine. Before my program got to this method, it initialized the socket, input/output streams and sent commands to the output stream for logging in.




I don't know how to use JConsole, but I saved the output of the free/total memory. This is the output I got after a few minutes (<5 minutes). The total memory keeps increasing over time. This screws my latency (can't see that in the output) because, I believe, JVM is trying to allocate more memory and storing/retrieving objects is taking a while.

Right now (10-15 mins later), total memory is at 33,415,168.

Thanks!
Ryan Atwood
Greenhorn

Joined: Feb 03, 2009
Posts: 15
And sure enough.. I got this a while later:

Joe Ess
Bartender

Joined: Oct 29, 2001
Posts: 8843
    
    7

ryan atw wrote:First I gotta say this forum is awesome. You guys are really helpful (especially Joe) and quick.


Thanks
Hey, how often do you call A()? Is there a chance that you are creating numerous instances and they're all hanging around, blocked on the socket waiting for input? That would account for the memory being used. Try adding an instance counter and printing creation and destruction of that anonymous class.
Ryan Atwood
Greenhorn

Joined: Feb 03, 2009
Posts: 15
I won't be able to run/test the code until next Monday (no streaming data until then) so I'll show you what my code looks like instead.


Joe Ess
Bartender

Joined: Oct 29, 2001
Posts: 8843
    
    7

You are leaving some code out, like initializing the Socket and ObjectOutputStream. I tried using your code for A() in my TimeClient posted above. The JConsole session for that looked exactly like the one I posted for my previous client. No memory leak.
You can dummy up a server to provide you with some data pretty easily. My server is all of 47 lines of code.
Ryan Atwood
Greenhorn

Joined: Feb 03, 2009
Posts: 15


That's what I have in my login function. I can't reveal much code because I'm using a proprietary API from another vendor. My guess is that it's their server code that's the fault?
Joe Ess
Bartender

Joined: Oct 29, 2001
Posts: 8843
    
    7

ryan atw wrote: I can't reveal much code because I'm using a proprietary API from another vendor. My guess is that it's their server code that's the fault?


I can't say because I can't read the code and test it!
It would be a stretch to say that the server code is creating a memory leak in your client code.
Ryan Atwood
Greenhorn

Joined: Feb 03, 2009
Posts: 15
That's true.. but honestly that is my code. The parts I'm hiding are creating these custom objects, and sending them across the socket during login.

Thanks anyways.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: ObjectInputStream creating OutOfMemoryError
 
Similar Threads
Logging problem log4j:ERROR
Error in serialization java.io.InvalidClassException:
maven -- java.lang.OutOfMemoryError: Java heap space
[JSF2] Warning message during redeploy
Outofmemory Error in ObjectInputStream