Keeping 4 lakhs db records data into a array list of plain java bean or VO objects

Dear Friends,

Is it a good way to keep all 4 lakhs records from db into a array list. Will it affects the performance. My requirement is i need to migrate that data to a content repository. So i need to send them one by one by accessing those plain java bean or VO's present in the array list.

Please let me know what is the efficeint way, whether to read 1000 records once process them and then again read another 1000 or i can do all 4 lakhs at a time. Which way will me more performance oriented.

Please help me ASAP Thanks in advance.

Read them X records at a time and process them. 400,000 records processed into an arraylist can become bulky and inefficient. Trying to gulp down all the data at once is usually not the way to go

venkata vinay bysani wrote:Which way will me more performance oriented.

Performance is very subjective. It depends on what kind of processing you are doing.
If you have to iterate over the list processing each record one by one and you do not have memory constraints then nothing like keeping it simple.
However, if you have to search elements in the list and massage data then obviously a list with half a million records is an absolute no.

venkata vinay bysani wrote:
Is it a good way to keep all 4 lakhs records

Please use English words. Not all the members of this forum know what lakh is.

Hi Vinay,

why cant you have a [code] PL/SQL [/code] procedure block that does this processing? => Option1
(because db is intelligent enough to handle the performance-again depends on hw smart your pl/sql block is written

always remember being a developer one should not interact with your db directly => use any orm(for instance use
Hibernate/Ibatis/..because they take care of performance to some extent..) to interact with your db => option 2

one thing i would like to know.? why do you need access to 4000 db records ?

finally performance is a guideline , not a target.

why cant you have a PL/SQL procedure block that does this processing? => Option1
(because db is intelligent enough to handle the performance-again depends on hw smart your pl/sql block is written

True, though the down side of this approach is you push work into the database layer that can be expensive to distribute, instead of the application layer that can be cheap to distribute.

always remember being a developer one should not interact with your db directly => use any orm(for instance use
Hibernate/Ibatis/..because they take care of performance to some extent..) to interact with your db => option 2

Using an ORM for such large bulk operations is a bad idea. ORMs are not suitable for bulk data manipulation.

Processing n records at a time and then purging it to DB is a advised ,since holding so much of data in memory may eat up all the available memory and leads to application / server crash down.One thing when ever this type of processing is performed make sure its in off peak loads...