This week's giveaway is in the Android forum.
We're giving away four copies of Android Security Essentials Live Lessons and have Godfrey Nolan on-line!
See this thread for details.
The moose likes Spring and the fly likes Spring Batch to process 1500000 records in cache memory for each event Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Android Security Essentials Live Lessons this week in the Android forum!
JavaRanch » Java Forums » Frameworks » Spring
Bookmark "Spring Batch to process 1500000 records in cache memory for each event" Watch "Spring Batch to process 1500000 records in cache memory for each event" New topic
Author

Spring Batch to process 1500000 records in cache memory for each event

Ninad Kulkarni
Ranch Hand

Joined: Aug 31, 2007
Posts: 787

Hi All,

I am using Spring Batch for following requirement

I have requirement for our Project Asset Liability Mamagement (ALM)

1. One user initiated event will have 1500000 records to process.
2. For every seconds there will be around 1000 events for processing data.
3. We will get input data for processing each record from application.
4. We will read records from cache memory and then will process records and again will write into cache memory. This process is for single record, in same way we have to process 1500000 records.
3. I am using Spring Batch to do above job.

Can anybody give suggestion or comments to achive above job processing in less time with optimum performance.


SCJP 5.0 - JavaRanch FAQ - Java Beginners FAQ - SCJP FAQ - SCJP Mock Tests - Tutorial - JavaSE7 - JavaEE6 -Generics FAQ - JLS - JVM Spec - Java FAQs - Smart Questions
Ninad Kulkarni
Ranch Hand

Joined: Aug 31, 2007
Posts: 787

Hi All,

Please give comments and views on above.
Mark Spritzler
ranger
Sheriff

Joined: Feb 05, 2001
Posts: 17249
    
    6

Not really much to go on for anyone to answer. But I have some suggestions.

Or better yet, the docs of Spring Batch has those suggestions

http://static.springsource.org/spring-batch/reference/html/scalability.html

Remote Chunking and Partitioning is the basic answer.

Mark


Perfect World Programming, LLC - Two Laptop Bag - Tube Organizer
How to Ask Questions the Smart Way FAQ
Ninad Kulkarni
Ranch Hand

Joined: Aug 31, 2007
Posts: 787

Thanks Mark for your views and suggestion we have used same
 
Don't get me started about those stupid light bulbs.
 
subject: Spring Batch to process 1500000 records in cache memory for each event
 
Similar Threads
Spring Integration and Spring Batch
Help improve performance jdbcTemplate update
Data Loading Strategy
reading and writing large data(200000) from oracle
BatchUpdateException: [IBM][CLI Driver][DB2/LINUX] SQL0302N