• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Caching a Large List

 
Greenhorn
Posts: 2
jQuery Eclipse IDE Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I have a very large list(Array List) of objects - approximately 50,000 to 60,000 instances of a particular class (which in turn has approximately 9-10 instance variables including Lists, Hash Map).
I need to filter this list as per user request based on 3 parameters.
This list is the life of my application - I need it be available for usage throughout the lifetime of my application. Also keeping the performance standard in mind - I can not afford to hit Database repeatedly. Is there any way so that i can cache this list in an effective manner? I am not permitted to alter the DB schema.

If any further info is needed - please let me know.

Please help. Thanks in advance.
 
Marshal
Posts: 79177
377
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Debasish Mukherjee wrote:I have a very large list(Array List) of objects - approximately 50,000 to 60,000 . . .

that isn’t large. 60000000 would be large.

You should not have to alter the DB schema if you can create a query which will filter the database before you populate your list. That is almost certainly the fastest way to filter that List. And probably the best.
A few other suggestions:-
  • Create a new database and populate it from that List. Run a query to select according to your three parameters on that database.
  • Create a class (Foo) which encapsulates the 3 fields from that List. Populate a Map; I am not sure whether you are better with a Map<Foo, List<Something>> or a Map<Something, Foo>.
  • Create a Comparator<Something> which uses those three fields and put the contents of the List into a sorted set. You can find elements with a binary search.
  • I do not know whether any of those suggestions will help you.
     
    Campbell Ritchie
    Marshal
    Posts: 79177
    377
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator
    I think this question is too difficult for “beginning”, so I shall move it.
     
    Bartender
    Posts: 10780
    71
    Hibernate Eclipse IDE Ubuntu
    • Likes 1
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator

    Debasish Mukherjee wrote:I have a very large list(Array List) of objects - approximately 50,000 to 60,000 instances of a particular class (which in turn has approximately 9-10 instance variables including Lists, Hash Map).
    I need to filter this list as per user request based on 3 parameters.
    This list is the life of my application - I need it be available for usage throughout the lifetime of my application. Also keeping the performance standard in mind - I can not afford to hit Database repeatedly. Is there any way so that i can cache this list in an effective manner? I am not permitted to alter the DB schema.


    First: If your list/table contains 50-60,000 rows, I highly doubt that database access is going to be that much of an overhead.

    Second: It's also highly likely that your database can sort and filter that list far quicker than you will ever do, because that's what they're designed to do.

    The only thing that caching the List might do is to reduce network traffic between the db and your app; however, there are several other ways of doing that, and I'd want to be absolutely sure that I was solving an actual problem - not a "perceived" one - before I wrote any extra caching layer.
    And the ONLY way to prove that is with metrics.

    W.A. Wulf wrote:More computing sins are committed in the name of efficiency (without necessarily achieving it) than for any other single reason — including blind stupidity.



    Winston
     
    reply
      Bookmark Topic Watch Topic
    • New Topic