• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Hot to deal with OutOfMemoryError

 
Ranch Hand
Posts: 413
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Whenever I am building my hashMap from a short text file then it runs ok
But when I try to do the same from a 781 KB file then I get this error
Can any one tell me remedy for it
Thanks in advance
 
Ranch Hand
Posts: 1873
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Gaurav
It depends on lot of factors.
- You can try to write the code to consider less memory usage. e.g. instead using String s = s1 + s2 100000 times use StringBuffer and all
- and some other little things to lessen your memory requirements
- run JVM with -Xms option to increase maximum heap size
BUT if you really have very large dataset then at somepoint you MAY run out options
Please post your code if possbile to see if there is anything easy to do before increasing memory size or claim that "its too much data" ..
Regards
Maulin
 
Gaurav Chikara
Ranch Hand
Posts: 413
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

This was my code but only diff is that inpur file regex.txt is of size 780 K and contains lot of gibberish characters.I am trying to make a search engine which will store all entries of a particular keyword in a rextr file
in HashMap in form of key value pairs where key is keyword and value is all occureneces of it separated by comma
Thanks in advance
[ Jess added UBB [code] tags to preserve whitespace, check 'em out! ]
[ April 20, 2004: Message edited by: Jessica Sant ]
 
Ranch Hand
Posts: 82
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi.
Well, as stated before, you ARE using many Strings and String operations. It will be worthwhile to see what you can change to StringBuffers.
Also, as stated before, I'd check the JVM's maximum heap size flag, which is, btw, -Xmx not -Xms.
Nimo.
 
Ranch Hand
Posts: 77
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
The problem does not lie in the Java options, nor does it have anything to do with the fact that you use many string operations - you can safely do that. The problem lies in the design. If you have a good development environment, try to run the program in a debugger and see what happens. To give an example:
readLine() reads random input lines, but the program treats them as if they were words. Actually, it is far worse: The program treats them as if they were regular expressions. If the program reads e.g. a line containing only an "e", it will try to match "e" against every input line. And every "e" will match even if it is in the middle of a word. That's a lot of matches. Now imagine how many matches will occur if the program reads a blank line.
The explosive number of matches combined with the fact that the program stores every match in the directionary causes the OutOfMemoryException.
To say it in another way, what the program does is this:
  • 1) For each line l in the file, do this:
  • 1--a) Store the line in the dictionary
  • 1--b) For each previously read line m in the dictionary, do this:
  • 1--b--x) If m matches (~= contains) l, record that l was seen at the current file position


  • What I guess it should do is this:
  • 1) For each word w in the file, do this:
  • 1--a) Record that w was seen at the current file position


  • With this simpler structure, the program will be shorter, faster, and will no doubt be able to organize files of several megabytes without needing special java options.
    I assume the first step you need to take is find a clean way to break the input into words. You can use a StringTokenizer for this purpose. You can also use regexps for this, but if you choose to do it, you should only need one generic regular expression that will match any word.
    Hope this helps!
    [ April 21, 2004: Message edited by: Nicky Bodentien ]
     
    reply
      Bookmark Topic Watch Topic
    • New Topic