In a project we are working on, there is large amount of data in play at runtime. There are about 40 domain classes corresponding to different chunks of this data. We have a search module which is pretty heavy and needs to work with most of these objects throughout the user session. Existing application does all of this with objects in session memory. It loads data in all these objects, goes through various stages working with these and gives a final result. It is not that bad actually but because it needs all the data in memory so more RAM is required during search operations. There is not object caching or serialization going on. All data is in session memory.
We are in the process of re-designing this part of application. A part of our team has come up with idea of using HSQL db as intermediate storage for data. This means we will have de-normalized tables representing
intermediate results of search results instead of objects. In other words, suppose search does job X1, X2 and X3 before it is done. If X1 part works with 15 objects and comes up with intermediate result R1, this R1 will be flattened into a database table. These 15 objects will be discarded. X2 will fetch data from R1 tables to do its job and its result R2 will be flattened in database as R2. X3 will work with R3 and finally delete parts of R1, R2 and R3 data that is not needed for rest of the user session. Their thinking is:
1) This will reduce the complexity of search operation because it is easier to work with denormalized relational data instead of heirarchy of
java objects
2) This may reduce load on server because we will use some kind of caching with HSQLDB and it ill take care of serializing data not in use
Counter-argument by some other team members are:
1) We will still need object structure because when we have to use data from R1 and R2 we will have to keep it in objects
2) It may actually increase the load on server and need much more RAM because HSQLDB will consume a lot more memory than bare Java objects and will also be much slower (based on our little tests we performed)
In stead we are suggesting to use session scope for Spring beans (since we are migrating to Spring) and use caching like Apache cache or ehcache to take care of memory utilization.
What will you suggest in this case? If you have alternative suggestions please share.