Last week, we had the author of TDD for a Shopping Website LiveProject. Friday at 11am Ranch time, Steven Solomon will be hosting a live TDD session just for us. See for the agenda and registration link
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Paul Clapham
  • Ron McLeod
  • Jeanne Boyarsky
  • Tim Cooke
Sheriffs:
  • Liutauras Vilda
  • paul wheaton
  • Henry Wong
Saloon Keepers:
  • Tim Moores
  • Tim Holloway
  • Stephan van Hulst
  • Carey Brown
  • Frits Walraven
Bartenders:
  • Piet Souris
  • Himai Minh

Too many connections

 
Rancher
Posts: 43027
76
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
It's hard to say where the slowdown is coming from without knowing what the code is doding for each record. Are objects being created for each record? Is some work done locally before hitting the DB? How much memory are you allocating to the JVM? What happens with each record after the thread pool limit is exhausted? How is the code even dealing with 80000 records if there are only 30 threads?

I tried a couple of runs with 5 threads and each time the results were not comparable to when i took 30 threads.


The point I was trying to make was not whether the sweet spot is at 5 or 10 or 30 threads; the point was that using 20000 will have a major negative impact on performance.
 
Ranch Hand
Posts: 78
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Are objects being created for each record? Is some work done locally before hitting the DB?


Hardly any objects are being created. Most of it is math calculations inside the functions.
This is what happens:

If method execute is called and all the threads in the ExecutorService are being used, the Runnable will be placed in a queue and assigned to the first thread that completes its previous task.

How is the code even dealing with 80000 records if there are only 30 threads?


I dont know. Maybe thats why it takes so long.
But when i did the test with 10,000 records also my performance was bad:
Time elapsed = 1019.74 sec, CDR=10000, no of threads= 30 maxActive= 35

Even with 4000 records it wasnt a remarkable performance:
Time elapsed = 179.25sec, CDR=4000, no of threads= 30

And i did understand the point that you were trying to make about threads. I just wanted to check what would be optimal.
Increasing the thread size would only worsen the performance, what else could be the bottle neck?

 
Ulf Dittmer
Rancher
Posts: 43027
76
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

mayank gupta wrote:Hardly any objects are being created.


I beg to differ:

This looks like 80000 CreditCalculatorThread objects are being created. Depending on how much memory these objects consume, this could be a very substantial amount of memory. Depending on how much memory is allocated to the JVM, it could be spending most of its time doing garbage collection.

That's the reason I kept saying from the beginning: Don't allocate that many threads. Rewrite your worker threads so that each can work on multiple records sequentially.
 
Hey, check out my mega multi devastator cannon. It's wicked. It makes this tiny ad look weak:
Free, earth friendly heat - from the CodeRanch trailboss
https://www.kickstarter.com/projects/paulwheaton/free-heat
reply
    Bookmark Topic Watch Topic
  • New Topic