*
The moose likes Performance and the fly likes Use of memory == less? GC : Soft ref Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of The Java EE 7 Tutorial Volume 1 or Volume 2 this week in the Java EE forum
or jQuery UI in Action in the JavaScript forum!
JavaRanch » Java Forums » Java » Performance
Bookmark "Use of memory == less? GC : Soft ref" Watch "Use of memory == less? GC : Soft ref" New topic
Author

Use of memory == less? GC : Soft ref

nishwas mahindra
Ranch Hand

Joined: Dec 23, 2002
Posts: 104
hi
i am writing a search engine.I m loading pointers to words in index files in memory.When requests are less then it works fine.For some time it serves properly.But after some time as number of requests increases lack of memory prohibits it to process any requests.And I have to restart.Is there any solution for this memory issue.How can I free my memory at runtime.For fast retrieval of results I have to keep the pointers to words inside memory.
Garbage collection is not the feasible solution I think.
Can soft references be of any use in this scenario?
Even if I use soft references will it create any problem if arbruptly any reference to object gets garbage collected?
Thanks in advance.
Bipin
Peter den Haan
author
Ranch Hand

Joined: Apr 20, 2000
Posts: 3252
Originally posted by bipin deshpande:
Can soft references be of any use in this scenario?
Not really. You will find that they are garbage collected much sooner than you want. The solution to your problem is probably caching -- a simple LRU cache is likely to do fine. Rather than allowing your data set to grow indefinitely, you cache up to a fixed number of documents or words (I don't know which, your problem isn't fully clear to me). When you need to put new content in your cache, you simply junk the least recently used item(s) to make room. A LinkedList can keep track of the items.
- Peter
nishwas mahindra
Ranch Hand

Joined: Dec 23, 2002
Posts: 104
I don't have any ranking system because i can not prioritize my docs.
Don Kiddick
Ranch Hand

Joined: Dec 12, 2002
Posts: 580
Originally posted by Peter den Haan:
A LinkedList can keep track of the items.
- Peter

I find the LinkedHashMap class (since 1.4) makes writing LRU caches a cynch. Might be worth looking at.
Tom.
nishwas mahindra
Ranch Hand

Joined: Dec 23, 2002
Posts: 104
i don't know much abt linkedhashmap class.Is it useful instead of hashtable ?
what advantages it has over hashtable?

what do u mean by cynch?
Peter den Haan
author
Ranch Hand

Joined: Apr 20, 2000
Posts: 3252
LinkedHashMap is a flavour of HashMap which maintains a LinkedList of its key values. Tom rightly drew attention to this class, because it can actually implement an LRU algorithm out of the box using its three-argument constructor: new LinkedHashMap(size, 0.75, true) will keep its LinkedList in order of last access.
It will even facilitate the removal of the oldest entry to keep the size constant; all you have to do is subclass this class and override the removeEldestEntry method. An example is given in the javadoc.
LinkedHashMap is really a ready to go LRU cache kit.
- Peter
Peter den Haan
author
Ranch Hand

Joined: Apr 20, 2000
Posts: 3252
Originally posted by bipin deshpande:
iIs it useful instead of hashtable ?
And please, don't use Hashtable in new code if you can avoid it. Or Vector or Enumeration for that matter. Use the Collections classes instead (HashMap, ArrayList, LinkedList, etc, programming against the Map, List, etc interfaces wherever possible).
There is really no good reason to use Hashtable and friends, and a few very good ones not to.
- Peter
nishwas mahindra
Ranch Hand

Joined: Dec 23, 2002
Posts: 104
I have completed my application using jdk1.3. So if i have to use LinkedHashMap I have to start from scratch.
Peter den Haan
author
Ranch Hand

Joined: Apr 20, 2000
Posts: 3252
Why would you have to start from scratch? You could either hop to 1.4, or implement a little LRU cache by hand. Neither should be remotely as stressful as starting from scratch.
- Peter
Mark Herschberg
Sheriff

Joined: Dec 04, 2000
Posts: 6037
Originally posted by Peter den Haan:
And please, don't use Hashtable in new code if you can avoid it. Or Vector or Enumeration for that matter. Use the Collections classes instead (HashMap, ArrayList, LinkedList, etc, programming against the Map, List, etc interfaces wherever possible).
There is really no good reason to use Hashtable and friends, and a few very good ones not to.
- Peter


Um, so what are the good reasons not to? I know they created the collections interface, but never paid much attention to it.
--Mark
Peter den Haan
author
Ranch Hand

Joined: Apr 20, 2000
Posts: 3252
Originally posted by Mark Herschberg:
Um, so what are the good reasons not to [use Vector, Hashtable and friends]?
  • The interfaces and classes of the Collections framework form a logical, coherent whole that does everything Vector and Hashtable do and much, much, much more. Vector and Hashtable sit outside it and are simply so much more to remember without any benefit. You could of course go the other way and use Vector and Hashtable exclusively and ignore the Collections framework, but you would lose out on some of the most significant tools in the Java toolbox.
  • Sun's attempt to retrofit Vector and Hashtable with the Collections interfaces has made their APIs bloated and inconsistent. What shall I use today? An Enumeration or an Iterator? Code developed by different developers is likely to be inconsistent.
  • If your code isn't multi-threaded, Vector and Hashtable perform worse because of their unnecessary synchronization.
  • If your code is multi-threaded, the synchronization in Vector and Hashtable is probably in the wrong place and you'll have to synchronize explicitly anyway. The duplicate synchronization again decreases performance.
  • Far worse than this, the fact that they are synchronized lures developers without much experience in concurrent programming into a false sense of security about the thread-safety of their application.
  • - Peter
    [ January 09, 2003: Message edited by: Peter den Haan ]
    Rod Macpherson
    Greenhorn

    Joined: Jan 12, 2003
    Posts: 10
    Originally posted by bipin deshpande:
    hi
    i am writing a search engine.I m loading pointers to words in index files in memory....

    GC definitely not a solution: you are running out of memory and by definition the GC could not find enough objects on the heap to free. Forcing GC is only going to insult its intelligence
    Memory mapped files will provide direct access to your index files with automatic paging. Excellent book with examples:
    http://www.oreilly.com/catalog/javanio/
    The suggestions to data structures with "weak" references are also excellent but require you provide a reloading strategy if the reference evaporates: that may require random access of your persistent data. Good ideas nevertheless.
    Rod Macpherson
    Greenhorn

    Joined: Jan 12, 2003
    Posts: 10
    Originally posted by bipin deshpande:
    I have completed my application using jdk1.3. So if i have to use LinkedHashMap I have to start from scratch.

    Sounds like you have to restructure your app to solve this memory problem anyway: starting from scratch not required. If you can you definitely should switch to JDK 1.4.
    nishwas mahindra
    Ranch Hand

    Joined: Dec 23, 2002
    Posts: 104
    Thanks to all.
    I think i should go for jdk1.4.
    Mark Herschberg
    Sheriff

    Joined: Dec 04, 2000
    Posts: 6037
    Thank you Peter. After reading your message I looked into the collections framework. I've been using it for two weeks now and am starting to appreciate the benefits (sorting became very easy!). Thanks for showing me the light.
    --Mark
     
    I agree. Here's the link: http://aspose.com/file-tools
     
    subject: Use of memory == less? GC : Soft ref