Efficient way of handling data in memory

[shan raj]

Hi All,

The client request datas are stored in multiple files.
My requirement is to handle the data in efficient way to process the client request. For this I have loaded the data in memory using java object during the startup of the server process. But it can not be scaled for larger information in memory.

To make it efficient I thought of storing the infrequently requested java object in file by

 implements Serializable   

interface. So that whenever in need it loads the data to memory and serve the response.

I am not sure will this approach scale or not.

Any one please validate the above approach or prove me solution to do alternate approach.

Thanks
Shan

[Nitesh Kant]

What you are doing is called caching and persistence is one of the aspects of a cache solution.

From what you have described, it looks like you have huge user data in memory. In order to avoid memory problems, you want to remove less frequently used data from memory and store in a persistence store.

If this is what is your problem statement then a persistence enabled caching solution is the correct way. EHCache is one of the freely available cache library, it may be worth using it.

[Ryan Beckett]

I am confused by your question. Maybe I can clear some things up for you.

The client request datas are stored in multiple files.

1. Persistence requires Serialization. Serialization converts an object to a byte stream using writeObject() and readObject() and stores the bytes in a file. This is efficient, in comparison to storing an object with your own algorithm. That would be ridiculous.

To make it efficient I thought of storing the infrequently requested java object in file by

2. I think I understand your dilemma. For example, you have a properties file that stores user names and passwords, and you also have a file that stores a less requested object, say an object that represents a pool of conference rooms. The properties file is accessed many times to check logging information, and you don't need to store object states. It's fine to store the data is characters because the schema is simple. The conference room pool has many rooms which have states. Like I said storing them with your own algorithm is ridiculous, hence why Java provides you with Serialization.
Obviously, it would the quickest way to get some data.

3. As far as caching is concerned, it only matters in the context of data accessing. If your reading large amounts of data from a file, whether its many times or even once, its best to use a BufferedInputStream. Chunks of data will be read from the file instead of character by character. I'd rather access the hard drive and read a 1KB chunk once, than read a 16 bit character 8192 times.

I hope this helps.

[shan raj]

Thanks Nitesh Kant & Ryan Beckett.

Your understanding of my question is exactly correct. if there is no side effect I will use object serialization way for persisting the data in file.
In parallel I would check EHCache mechanism.

Between, if I use the Serializable to persist and retrieve the data like below

 
ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream("C:\\sample.txt"));
oos.writeObject(myObj);

1. Will it take more time compared to plain text file reading or any other drawbacks?
2. How to use BufferedInputStream here or the ObjectOutputStream itself take of buffered reading.

With Regards
Shan

[Campbell Ritchie]

A database connection? Would that help?