aspose file tools*
The moose likes JDBC and the fly likes Too many connections Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Soft Skills this week in the Jobs Discussion forum!
JavaRanch » Java Forums » Databases » JDBC
Bookmark "Too many connections" Watch "Too many connections" New topic
Author

Too many connections

Ulf Dittmer
Marshal

Joined: Mar 22, 2005
Posts: 42915
    
  68
It's hard to say where the slowdown is coming from without knowing what the code is doding for each record. Are objects being created for each record? Is some work done locally before hitting the DB? How much memory are you allocating to the JVM? What happens with each record after the thread pool limit is exhausted? How is the code even dealing with 80000 records if there are only 30 threads?

I tried a couple of runs with 5 threads and each time the results were not comparable to when i took 30 threads.

The point I was trying to make was not whether the sweet spot is at 5 or 10 or 30 threads; the point was that using 20000 will have a major negative impact on performance.
mayank gupta
Ranch Hand

Joined: Dec 21, 2008
Posts: 78
Are objects being created for each record? Is some work done locally before hitting the DB?

Hardly any objects are being created. Most of it is math calculations inside the functions.
This is what happens:

If method execute is called and all the threads in the ExecutorService are being used, the Runnable will be placed in a queue and assigned to the first thread that completes its previous task.
How is the code even dealing with 80000 records if there are only 30 threads?

I dont know. Maybe thats why it takes so long.
But when i did the test with 10,000 records also my performance was bad:
Time elapsed = 1019.74 sec, CDR=10000, no of threads= 30 maxActive= 35

Even with 4000 records it wasnt a remarkable performance:
Time elapsed = 179.25sec, CDR=4000, no of threads= 30

And i did understand the point that you were trying to make about threads. I just wanted to check what would be optimal.
Increasing the thread size would only worsen the performance, what else could be the bottle neck?

Ulf Dittmer
Marshal

Joined: Mar 22, 2005
Posts: 42915
    
  68
mayank gupta wrote:Hardly any objects are being created.

I beg to differ:

This looks like 80000 CreditCalculatorThread objects are being created. Depending on how much memory these objects consume, this could be a very substantial amount of memory. Depending on how much memory is allocated to the JVM, it could be spending most of its time doing garbage collection.

That's the reason I kept saying from the beginning: Don't allocate that many threads. Rewrite your worker threads so that each can work on multiple records sequentially.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Too many connections