aspose file tools*
The moose likes Threads and Synchronization and the fly likes New threading architecture leads to unexpected results Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Spring in Action this week in the Spring forum!
JavaRanch » Java Forums » Java » Threads and Synchronization
Bookmark "New threading architecture leads to unexpected results" Watch "New threading architecture leads to unexpected results" New topic
Author

New threading architecture leads to unexpected results

John Ryan
Ranch Hand

Joined: Mar 14, 2001
Posts: 124
Hi all,

We have redesigned an existing application and are now coming up with some unexpected results. This is a simplified explanation of what is going on, and maybe is somewhat vague, but any pointers would be appreciated.

Initially our application ran with a number of threads (usually 3) reading from a single socket. There were two message types which could be read from the socket - Message 1 which leads to an entry being written to a database and the sending of a response, Message 2 leads to our application forwarding the message to another application "B". The problem with this was that our application was not accepting/reading messages quickly enough - the queue on the socket was large and the length of time taken to handle messages led to a timeout in Appication B (with very nasty results)

So we changed the application architecture so that the threads reading from the socket no longer processed type 1 messages. We felt that this would increase the number of messages being read by the threads from the socket because the threads would not have to waste time writing to a database and sending a response. Instead type 1 messages are placed in an internal queue (a LinkedHashMap) and we have seperate data handling threads which write the entry to the database and send the response. This change has led to our application dramatically increasing the number of messages it can read from the socket (by a factor of 2). However the roundtrip time for type 1 messages has also increased exponentially. It seems that the type 1 messages just sit for too long in the internal queue. There does not seem to be a bottleneck in writing to or reading from this queue (as it is synchronized). So we are now stuck trying to figure out why our data handling threads cannot handle the types 1 messages enough?

All this might sound pretty vague but I am hoping that someone might spot a gaping hole in our logic
Cheers
John
[ October 20, 2004: Message edited by: John Ryan ]
David Harkness
Ranch Hand

Joined: Aug 07, 2003
Posts: 1646
If I understand you correctly, in summary you had three threads processing 2 types of messages from a socket. To increase throughput, you added two threads to handle type 1 messages (in the same JVM) and left the original three threads to handle only type 2 messages.

If that's the case, you've really only added more scheduling overhead as you've increased the number of threads without decreasing the amount of work. True, this can be a benefit if much of the time a thread spends is waiting for I/O or an external resource (writing to a database). Other threads can process work while those threads wait.

Perhaps you can add more threads to process type 2 messages since those are what you really want to address.
Alexandru Popescu
Ranch Hand

Joined: Jul 12, 2004
Posts: 995
You did not mention how you manage the threads that work with the database and also how you manage the database connection. This may represent a performance problem if not correct implemented.

./pope

just a guess :-(


blog - InfoQ.com
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
Did you take a look at the CPU load?


The soul is dyed the color of its thoughts. Think only on those things that are in line with your principles and can bear the light of day. The content of your character is your choice. Day by day, what you do is who you become. Your integrity is your destiny - it is the light that guides your way. - Heraclitus
John Ryan
Ranch Hand

Joined: Mar 14, 2001
Posts: 124
Originally posted by David Harkness:
If I understand you correctly, in summary you had three threads processing 2 types of messages from a socket. To increase throughput, you added two threads to handle type 1 messages (in the same JVM) and left the original three threads to handle only type 2 messages.

Yes this is correct. I notice I mixed up the message numbers somewhat in the original post and have edited this. The original 3 threads do not only "handle" type 2 messages. They only process type 2 messages but they also read type 1 messages from the socket and add them to the internal queue

Originally posted by David Harkness:

If that's the case, you've really only added more scheduling overhead as you've increased the number of threads without decreasing the amount of work. True, this can be a benefit if much of the time a thread spends is waiting for I/O or an external resource (writing to a database). Other threads can process work while those threads wait.

Perhaps you can add more threads to process type 2 messages since those are what you really want to address.


Well I think a significant amount of time was being spent on writing to the database so the fact that we are using seperate threads to write (Message type 1) to the database should ensure that our original 3 threads have more time to handle message type 2. Also our application is running on a Sun Solaris machine with 4 processors and from my understanding Java threads on Solaris are mapped to lightweight processes so adding extra threads should not introduce much scheduling overhead (is adding an extra thread not much the same as starting another process?)
John Ryan
Ranch Hand

Joined: Mar 14, 2001
Posts: 124
Originally posted by Ali Pope:
You did not mention how you manage the threads that work with the database and also how you manage the database connection. This may represent a performance problem if not correct implemented.

./pope

just a guess :-(


Hi Ali,
The threads that work with the database are set up as a ThreadGroup and the number started is configurable. We are using ConnectionPooling to manage our DB connections and have checked the delay in retrieving a connection and it is not an issue/cause of the problem.
John Ryan
Ranch Hand

Joined: Mar 14, 2001
Posts: 124
Originally posted by Ilja Preuss:
Did you take a look at the CPU load?


Yes each time we execute our tests we monitor the CPU load
Alexandru Popescu
Ranch Hand

Joined: Jul 12, 2004
Posts: 995
Originally posted by John Ryan:

The threads that work with the database are set up as a ThreadGroup and the number started is configurable. We are using ConnectionPooling to manage our DB connections and have checked the delay in retrieving a connection and it is not an issue/cause of the problem.


Good about connection pooling. But I am not sure about ThreadGroup? Did you mean thread pool instead?

./pope
Alexandru Popescu
Ranch Hand

Joined: Jul 12, 2004
Posts: 995
You can find some discussion about ThreadGroup vs ThreadPool on this forum in order to clarify the terms/usages.

./pope
John Ryan
Ranch Hand

Joined: Mar 14, 2001
Posts: 124
Originally posted by Ali Pope:
You can find some discussion about ThreadGroup vs ThreadPool on this forum in order to clarify the terms/usages.

./pope


Hi Ali,
Sorry I meant a ThreadPool - we have an object which holds an array of Thread objects. Number of threads that are created and started is configurable based on a parameter passed to the ThreadPool constructor
Alexandru Popescu
Ranch Hand

Joined: Jul 12, 2004
Posts: 995
Oke. Now as I saw you said you did some monitoring: where are the hotspots detacted? I think this is the point where you should dig.
Give us more details maybe someone will figure out.

./pope
David Harkness
Ranch Hand

Joined: Aug 07, 2003
Posts: 1646
To summarize where we're at, you have two incoming messages over a socket. A pool of threads (A) reads them in turn. Type 1 messages are placed in a queue for processing by another thread pool (B). Type 2 messages are handled completely by the thread that reads them from the socket. Finally, thread pool B pulls type 1 messages from the queue and writes them to a database.

And the problem is that messages are not being processed fast enough.

My first instinct would be to have designed it such that only one thread was reading from the socket and placing the messages into two separate queues (to be able to tune them separately). Then I would have two thread pools processing the queues.

Note that this is very similar to your design. The difference is that if all three reader threads are busy handling type 2 messages at once, the socket starts backing up. But I see a few advantages to this method:
  • Both types of messages are treated exactly the same up to the point where the actual handling of the message differs: they both get read and put into a queue.
  • Each queue and its respective thread pool can be tuned more easily.
  • Adding new types of messages just means a new queue.
  • You could have a single thread pool to handle message queues by applying an algorithm to choose which queue to poll based on queue priority and backlog.

  • I think at this point you really need to know what each reader thread is doing by adding logging if you haven't already. Figure out how long it takes each thread to complete its task (put a type 1 message on the queue, handle a type 2 message, handle a type 1 message). With that you should be able to tune your thread pools without making another architecture change.
    Warren Dew
    blacksmith
    Ranch Hand

    Joined: Mar 04, 2004
    Posts: 1332
        
        2
    John Ryan:

    We have redesigned an existing application and are now coming up with some unexpected results.... This change has led to our application dramatically increasing the number of messages it can read from the socket (by a factor of 2). However the roundtrip time for type 1 messages has also increased exponentially.

    Why is this unexpected?

    Previously, your server capacity was limited by the fact that when too many type 1 messages came in, the latency increased to the point where type 2 messages timed out. Having fixed that limitation, your capacity doubled - and now it's limited by how many type 1 messages you can process. To me, that sounds like the expected result.

    You don't say how many thread you have in your type 1 thread pool. If type 1 requests are processor limited, you might want to put in at least four threads into the pool to use all the processors, yielding often enough that the type 2 threads aren't blocked or running the type 1 threads at a lower priority. However, if the number of database calls, and perhaps disk access, are limiting you, you might need to speed up the database.
    John Ryan
    Ranch Hand

    Joined: Mar 14, 2001
    Posts: 124
    Originally posted by Warren Dew:

    Why is this unexpected?

    Previously, your server capacity was limited by the fact that when too many type 1 messages came in, the latency increased to the point where type 2 messages timed out. Having fixed that limitation, your capacity doubled - and now it's limited by how many type 1 messages you can process. To me, that sounds like the expected result.



    Warren I am inclined to agree with you. After looking at our results again yesterday evening I realised that our processor threads handling type 1 messages are able handle the same number of messages per second as our old application was able to handle (when the reader threads handled both type 1 and type 2 messages). The longer roundtrip time through the server is just the result of the fact that we take in more type 1 messages but still only have the capability to process them at the same rate. So maybe this is not an issue. We have solved the problem with the type 2 messages timing out, which was the main aim of the changes, so maybe we do not have a problem after all. I am not sure why it took me so long to see this but....

    Originally posted by Warren Dew:

    You don't say how many thread you have in your type 1 thread pool. If type 1 requests are processor limited, you might want to put in at least four threads into the pool to use all the processors, yielding often enough that the type 2 threads aren't blocked or running the type 1 threads at a lower priority. However, if the number of database calls, and perhaps disk access, are limiting you, you might need to speed up the database.


    In fact we do want the type 1 messages to run at a lower priority. By allowing the reader threads to handle the type 2 messages as soon as they are received and deferring type 1 messages to be handled by the processor threads I think we have made the type 2 messages higher priority
    John Ryan
    Ranch Hand

    Joined: Mar 14, 2001
    Posts: 124
    Originally posted by David Harkness:

    My first instinct would be to have designed it such that only one thread was reading from the socket and placing the messages into two separate queues (to be able to tune them separately). Then I would have two thread pools processing the queues.

    Note that this is very similar to your design. The difference is that if all three reader threads are busy handling type 2 messages at once, the socket starts backing up. But I see a few advantages to this method:
  • Both types of messages are treated exactly the same up to the point where the actual handling of the message differs: they both get read and put into a queue.
  • Each queue and its respective thread pool can be tuned more easily.
  • Adding new types of messages just means a new queue.
  • You could have a single thread pool to handle message queues by applying an algorithm to choose which queue to poll based on queue priority and backlog.

  • I think at this point you really need to know what each reader thread is doing by adding logging if you haven't already. Figure out how long it takes each thread to complete its task (put a type 1 message on the queue, handle a type 2 message, handle a type 1 message). With that you should be able to tune your thread pools without making another architecture change.

    Hi David,
    The reason we did not do somthing like this is because we want the type 2 messages to be handled with higher priority(to prevent the timeout that was occurring on the application which they are forwared to). If both type 1 and type 2 messages were placed into internal queues they would effectively have an equal priority and this is not what we want.....
    Alexandru Popescu
    Ranch Hand

    Joined: Jul 12, 2004
    Posts: 995
    John, David's proposal is a very scalable one and I would like to suggest using it. As regards the priority issue you are talking this is not the case. You can continue to increase the priority of message 2 processor threads.
    Moreover you can use a priority queue .

    ./pope
    David Harkness
    Ranch Hand

    Joined: Aug 07, 2003
    Posts: 1646
    Originally posted by John Ryan:
    If both type 1 and type 2 messages were placed into internal queues they would effectively have an equal priority and this is not what we want.....
    As Ali said, you can use a priority queue. As an example and sticking with two separate queues, when each thread in the handling pool grabs the next message to handle, it would apply some formula to choose a queue.The issue with having multiple reader threads also handling type 2 messages is that reading gets stalled whenever all reader threads are handling type 2 messages. Using a single reader thread that only reads ensure that the messages are placed into the queues as quickly as possible, given that adding a message to a queue is nearly instantaneous.

    This gives you greater flexibility in tuning your handling threads. As well, if you find a good priority queue you can avoid doing what I've done above to illustrate the technique. Normally you just provide an algorithm to calculate the priority of any given message and let the queue order them for you.

    This way you can have a single thread pool for handling all types of messages rather than one pool for each message type. The latter is suboptimal when you get a series of one type of message (a bunch of idle threads, not enough type X handling threads).
    Mr. C Lamont Gilbert
    Ranch Hand

    Joined: Oct 05, 2001
    Posts: 1170

    If you want certain threads to run with higher priority set them so. I would not use architecture to determine priority because that is not easily configurable.
    Henry Wong
    author
    Sheriff

    Joined: Sep 28, 2004
    Posts: 18896
        
      40

    Originally posted by CL Gilbert:
    If you want certain threads to run with higher priority set them so. I would not use architecture to determine priority because that is not easily configurable.


    I am not sure if I agree with this. For simple cases, playing with thread priorities or changing the number of threads may be best. However, if there is a complex need for ordering, or even some guarantee of ordering, then doing it by algorithm may be best. Keep the threads all at the same priority, and have them sort out the processing order based on the data.

    Henry


    Books: Java Threads, 3rd Edition, Jini in a Nutshell, and Java Gems (contributor)
    David Harkness
    Ranch Hand

    Joined: Aug 07, 2003
    Posts: 1646
    Originally posted by Henry Wong:
    if there is a complex need for ordering, or even some guarantee of ordering, then doing it by algorithm may be best. Keep the threads all at the same priority, and have them sort out the processing order based on the data.
    I agree. Even if the JVM and OS could guarantee that your thread priorities accurately affected how much "work" each thread could do per unit of time, the work they do may make this moot.

    For example, the work for type 1 messages (after being read from the socket as all messages are) seems to be
  • Pull the message from the queue (zero time)
  • Perform an insert into a database (blocking operation)
  • Throw the message away (zero time)

  • Say the insert takes an exagerated 1 second to complete and the other message types performed complex calculations that take about a second of dedicated CPU time, but you wanted to make type 1 messaeges higher priority. If you set type 1 threads to HIGH priority and type 2 to LOW but keep the number of threads equal (3 each), you'll end up with 3 threads waiting for I/O and 3 threads busy with type 2 messages.

    The result is that the both message types are processed at roughly the same rate as without priority settings. The priorities have little effect.

    Yes, the example is somewhat contrived, but I tried to keep it as close to John's case as I could. Priorities would probably work if the type of work each thread does is very similar.

    Thinking more about it, I'd say that prioritizing with an algorithm gives you control over the message-consumption rate while using thread priorities gives you control over the thread "workload," but that seems too nebulous to me without playing around with it. I'll be the first to admit that I haven't written any books about Java threading.

    Henry, the first edition book was one of the first cool Java books I got. I had never done any thread programming before Java, and your book made it amazingly easy to get started. Thanks and good luck with the new edition!
    Alexandru Popescu
    Ranch Hand

    Joined: Jul 12, 2004
    Posts: 995
    Originally posted by CL Gilbert:
    If you want certain threads to run with higher priority set them so. I would not use architecture to determine priority because that is not easily configurable.


    As Henry and David comments, prioritizing just using the setPriority is not always the solution. Involving priority queues is imo the way to solve this.

    ./pope
    John Ryan
    Ranch Hand

    Joined: Mar 14, 2001
    Posts: 124
    Hi guys,
    Thanks for all your replies. They have all been very useful and extremely educating. I agree that Davids soltion is probably the best way to go given its flexability and extensibility. Priority queues are something that I had not been aware of but I have now found some articles on the Sun web site that go into some details on how to implement them. I had previously ruled out varying the actual priority of the threads purely because I dont think I understand completely the behaviour that results from having threads of different priorities.

    My final question would be if you consider that our solution has suceeded in giving a higher priority to the type 2 messages??? - even though it isnt the best solution and could lead to backup of messages on the socket if the reader threads are all busy processing type 2 messages?

    Cheers,
    John
    Alexandru Popescu
    Ranch Hand

    Joined: Jul 12, 2004
    Posts: 995
    I think that Warren's comment best describe your current solution:

    Previously, your server capacity was limited by the fact that when too many type 1 messages came in, the latency increased to the point where type 2 messages timed out. Having fixed that limitation, your capacity doubled - and now it's limited by how many type 1 messages you can process. To me, that sounds like the expected result.


    ./pope
     
    wood burning stoves
     
    subject: New threading architecture leads to unexpected results