I am facing a major memory problem on a research project that I am working on right now. The project is designing a graphmaster for an Artificial Intelligent chatterbot. The AI bot requires patterns to be loaded in memory for answering questions posed to it by users.
So for efficient pattern searching I used the B* Tree to hash patterns and store the answers to those patterns at those nodes.
Now comes the problem. The entire pattern database is around 25 MB. So I need to load the whole database in memory. However I am able to load only around 4 MB data. After that the JVM crashes complaining about insufficient memory. I tried to perform a lot of optimizations relating to the structure of the B* tree, but without success. Any idea what can be done in this particular scenario. Without loading those patterns in memory the AI bot cannot work..
Have you looked into increasing the amount of memory available to Java as controlled by command line parameters? See the tools.html file in your tooldocs (you did download and install the Java Docs, right?) That 25MB of text is going to need a LOT more memory since 1) Java strings use 2bytes/char 2) you are creating lots of objects Bill
Can you challenge that "in memory" requirement? What user cares if it's in memory? Well, maybe a professor making an assignment. Real people care about rapid access to up-to-date information. Can you load only n levels of the tree below the current node? That would give very fast access to the next couple branches. Then maybe discard parts and load lower parts on another thread? Think of a sliding window in memory over the whole tree on disk. Not saying that's a great idea ... just saying think outside the box (or the memory in the box) and see where you go.
A good question is never answered. It is not a bolt to be tightened into place but a seed to be planted and to bear more seed toward the hope of greening the landscape of the idea. John Ciardi