• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Paul Clapham
  • Ron McLeod
  • Tim Cooke
  • Junilu Lacar
Sheriffs:
  • Rob Spoor
  • Devaka Cooray
  • Jeanne Boyarsky
Saloon Keepers:
  • Jesse Silverman
  • Stephan van Hulst
  • Tim Moores
  • Carey Brown
  • Tim Holloway
Bartenders:
  • Jj Roberts
  • Al Hobbs
  • Piet Souris

Can exhaustion of non-memory resources cause OutOfMemoryError?

 
Ranch Hand
Posts: 71
3
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
In pre Java 7 code where non-memory resources (like file handles, socket connections) are not automatically closed (via the try-with-resources statement), what happens if I forget to clean up non-memory resources? Can exhaustion of non-memory resources cause OutOfMemoryError? I'd imagine that if they're called "non-memory", they'd have nothing to do with memory.
 
Saloon Keeper
Posts: 13369
295
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Exactly. They don't have anything to do with memory. You clean them up, because if you don't, you may have a hard time reusing them, or other strange things may occur.

For instance, if you're writing to an output file using a buffered stream, and you forget to close the stream before the program finishes running, your output may actually never be written to file.

Another example is that a database might disallow you to open a new connection if you've not closed the last one.

 
Tess Jacobs
Ranch Hand
Posts: 71
3
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Stephan van Hulst wrote:Exactly. They don't have anything to do with memory.


Thanks for that, but I'm still confused about something.

What happens if I use FileInputStream objects to read a large number of files. Won’t these files be read into memory? If after reading the files, I set the FileInputStream objects to null (so that the objects are eventually garbage collected) but I forget to close the streams, what happens to the memory occupied by the files? Isn’t there a chance that reading a large number of files without closing the streams used to read them can lead to OutOfMemoryError?
 
Bartender
Posts: 689
17
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Every object you create will occupy some non-zero amount of memory. But in the case of the ones you mentioned that amount of memory will be small. You will almost certainly run into other problems long before you run out of memory. For example I believe operating systems will place a limit on how many file handles a process can have open, so if you don't close them then you might run out.

Specifically with the FileInputStream, closing the stream does not cause the file data to be garbage collected. The input stream passed the data to the calling code.
 
Stephan van Hulst
Saloon Keeper
Posts: 13369
295
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
No, because "reading files" doesn't actually use up memory. Allocating memory uses up memory. When you call a method (puts variables on the stack), or when you create a new object (reserves memory on the heap), is when you risk memory issues.

When you read a byte from a file, you must already have a variable to put it into. All you do when reading the byte, is change the value of your variable. After you close the stream, the variables or objects will still be there, if you didn't let them go out of scope.

You close streams so that their underlying source or sink can be reused, or the stream can add some special information or perform other operations that need to be done when they are no longer used.

You also don't usually have to assign null to variables to have them garbage collected, they will eventually get garbage collected if you let them go out of scope.
 
Tess Jacobs
Ranch Hand
Posts: 71
3
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks for clarifying that.
 
Marshal
Posts: 26909
82
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Have a look at this link: OutOfMemoryError creating a thread. In other words, it's possible to get OutOfMemoryError in situations which don't actually involve running out of memory.
 
Tess Jacobs
Ranch Hand
Posts: 71
3
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I’m guessing that a thread is another example of a non-memory resource since it is a path of execution.

The IBM article says that threads do actually consume memory (native memory, not heap space).

The IBM article seems to support my original suspicion that exhaustion of a non-memory resource can cause OutOfMemoryError. Because even though it is called a non-memory resource, it does consume memory.

I read another IBM article here which says that:

Running out of Java™ heap isn't the only cause of a java.lang.OutOfMemoryError. If native memory runs out, OutOfMemoryErrors that your normal debugging techniques won't be able to solve can occur.

 
Bartender
Posts: 10780
71
Hibernate Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tess Jacobs wrote:The IBM article seems to support my original suspicion that exhaustion of a non-memory resource can cause OutOfMemoryError. Because even though it is called a non-memory resource, it does consume memory.


Can I suggest that you're focusing on the wrong thing?

I suppose if you're debugging someone else's code, and it's throwing an OOM, then it might be of some use to know exactly when it can be thrown; but surely the important thing is not to let it happen in the first place.

So if you are debugging someone else's code, I think I'd start out by checking that it closes every resource it opens.

An ounce of prevention is worth a pound of cure.

Also:

If after reading the files, I set the FileInputStream objects to null (so that the objects are eventually garbage collected)...


Or not.

IMO, setting objects to null is a very bad way to "clean up", and just as prone to "forgetting" as "forgetting to close". It also suggests that either:
(a) Your design isn't sufficiently modular.
or:
(b) You're hanging on to object references for far too long.

Simple lesson: Don't forget. If you open a Closeable, close it. There is only one exception I know of to that rule: closing a stream or Scanner based on System.in.

HIH

Winston
 
Tess Jacobs
Ranch Hand
Posts: 71
3
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Winston Gutkowski wrote:Can I suggest that you're focusing on the wrong thing?



Thanks Winston. I agree that it's important to close a non-memory resource when you're done with it, but I also think it's important to understand the consequence of not closing it. From what I've gathered, a memory leak is not a likely consequence; a different kind of resource leak is more likely.
 
Marshal
Posts: 74344
334
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Winston Gutkowski wrote: . . . There is only one exception I know of to that rule: closing a stream or Scanner based on System.in. . . .

I would expand that to Formatters/output streams pointing to System.out and System.err

I personally think you should use System.in in a utility class, so you only need to use it in one location.
 
Stephan van Hulst
Saloon Keeper
Posts: 13369
295
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Actually, I think there are many exceptions. I've occasionally needed different FilterInputStream instances to read the same underlying stream, for different data types. When you're done with them, closing them will close the underlying stream and then your other streams won't function any longer.

Another example is when you're passing a stream to a method that should read or write to it. The method may know exactly how to interpret the stream, using its own wrapper stream. When it's done, it shouldn't close the stream, because it doesn't know what the caller has planned for the remainder of the input/output.
 
Winston Gutkowski
Bartender
Posts: 10780
71
Hibernate Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Stephan van Hulst wrote:Actually, I think there are many exceptions.


You're absolutely right.

Another example is when you're passing a stream to a method that should read or write to it. The method may know exactly how to interpret the stream, using its own wrapper stream. When it's done, it shouldn't close the stream, because it doesn't know what the caller has planned for the remainder of the input/output.


Ah, now I reckon that's slightly different. Something has to be in control of the stream that knows when to close it. We have immutable collections; why not uncloseable streams? It shouldn't be that difficult to wrap a Closeable in an object that simply fails to forward a close() request (or that throws an Exception if you do), and if you don't want a method closing your stream, even by accident, pass one of those.

Winston
 
Stephan van Hulst
Saloon Keeper
Posts: 13369
295
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Except unmodifiable collections are necessary to guarantee the validity of an object. I don't really see the need for an uncloseable stream. It should be the norm that if a method opens a stream (rather than wrapping an existing one), it closes it. If a method gets one passed to it, it does I/O and then it leaves the stream be.
 
Greenhorn
Posts: 2
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Many of the answers already cover the topic. But indeed, besides the regular heap memory there is a native memory and resources the JVM needs to perform the task at hand. Consuming those resources can also result in OutOfMemoryErrors - such as the java.lang.OutOfMemoryError: Requested array size exceeds VM limit, java.lang.OutOfMemoryError: Unable to create new native thread or java.lang.OutOfMemoryError: Metaspace in Java 8.

There are also other interesting corner cases, where the GC overhead hits the predetermined limits and/or finalizer queues growing beyond the limit where the cleaning thread can keep up with the work, but these would need to go into more details.
 
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
To add to the valuable information already provided:

Yes, JVM can absolutely run out of memory for either a) Java Heap where your objects live b) Native memory.

When OOM occurs, various JVMS (i.e Oracle HotSpot, IBM, HP, Jrockit etc) behave differently. With Oracle Hotspot, one thing you can look for is the 'Detail Message' that gets printed with every OutOfMemory error. If the 'Detail Message' has things like 'Native Memory','Native method','Native code' etc, you are running into native memory issue.

Native memory is utilized by Operating System. So, you will need operating system tools to diagnose this issue. On the same note, a Java heap dump will be of no use but rather you will need a dump of the 'process' from OS perspective.

It can be difficult to pin point the issue. But check your application to see if it uses any native code (JNI) to start.

Also look for any java command line parameter (-X<option) specifically for native memory that can help.

The tool jstack that comes with JDK can throw some light as it can print the stack trace of native methods (run jstack when you are experiencing out of memory errors)

Read one of my articles for some additional info (may not be directly related to your issue)

http://bit.ly/1mZoV1K (5 not so easy ways to monitor the Heap Usage of your Java Application)

Good luck.

>
 
You showed up just in time for the waffles! And this tiny ad:
Building a Better World in your Backyard by Paul Wheaton and Shawn Klassen-Koop
https://coderanch.com/wiki/718759/books/Building-World-Backyard-Paul-Wheaton
reply
    Bookmark Topic Watch Topic
  • New Topic