wood burning stoves*
The moose likes Tomcat and the fly likes Almost all threads are in Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Products » Tomcat
Bookmark "Almost all threads are in "P" status" Watch "Almost all threads are in "P" status" New topic
Author

Almost all threads are in "P" status

Mei Zhiguan
Greenhorn

Joined: Feb 05, 2012
Posts: 8
Hi,

I'm encountering a problem:
1. almost all threads are in "P" status, and occasionally there is 1 or 2 in "S" or "R" status,
2. and there is no slow query on MySQL side, because I can't catch any process with command "show full processlist",
3. while tomcat connection rate is merely less than 30req/s,
4. and tomcat CAN response correct result page in acceptable time(1 second)
System architecture: Tomcat*3 + Memcached*2 + MySQL-Proxy*2 + MySQL*2, Linux-Redhat
Software information: apache-tomcat-7.0.6, MySQL--5.5.11-1.rhel5.x86_64, jdk-6u23-linux-x64-rpm
Configuration:
tomcat server.xml:

which provides default maximum number of threads is 200

Screenshot: please refer to attachment.

Question:
1. is this situation(full of "P" status) problematic?
2. how to explain the situation?(waiting for Http request body? sounds unreasonable!)


[Thumbnail for Screenshot.png]

Rob Spoor
Sheriff

Joined: Oct 27, 2005
Posts: 19653
    
  18

If you check just below the table, you'll see this:
P: Parse and prepare request S: Service F: Finishing R: Ready K: Keepalive

You can read what these mean here.


SCJP 1.4 - SCJP 6 - SCWCD 5 - OCEEJBD 6
How To Ask Questions How To Answer Questions
Mei Zhiguan
Greenhorn

Joined: Feb 05, 2012
Posts: 8
Rob Spoor wrote:If you check just below the table, you'll see this:
P: Parse and prepare request S: Service F: Finishing R: Ready K: Keepalive

You can read what these mean here.


thank you for your quick reply.
I did read related information, including official documentation, but I just can't imagine why all of the threads are in the status of "Parse and prepare request", after all, there are merely <30 requests per second, and the service is quite responsive.

Please forget the "Max processing time: 9024 ms", because when it was around 1000ms, the situation is the same.

Anybody have ever met similar problem.
William Brogden
Author and all-around good cowpoke
Rancher

Joined: Mar 22, 2000
Posts: 12761
    
    5
How big are these requests anyway? How are they formatted?

Bill
Mei Zhiguan
Greenhorn

Joined: Feb 05, 2012
Posts: 8
William Brogden wrote:How big are these requests anyway? How are they formatted?

Bill


Request size:
I can roughly calculate the average size of requests by dividing Inbound throughput by request count, and the data is from Cacti
Average Request / Second: 84.85
Average Inbound throughput(bits/s): 135.93k(bits/s)
Average Request Size = 135.93 / 84.85 = 1.6k(bit)
= 1.6 * 1024 / 8 = 208(Byte/s)

Request format:

1. Every request is formatted with GET parameters and POST parameters, and below is a typical access log containing URL, which reflects GET parameters:
119.146.56.30 - - [01/Feb/2012:23:59:59 +0800] "POST /RecommendationService/recommend?user=07520560066&ppd=Unnd:PROG/565532@IKBTV.BMM.BMM&Page=page_stop_confirm HTTP/1.1" 200 907

2. POST parameter consists of:
method=seriesbasedrecommend
user=07580420366
ppd=Unnd:PROG/565532@IKBTV.BMM.BMM
spd=Unnd:PROG/565532@IKBTV.BMM.BMM
type=webdoc
tag=tech
pviewhistory=Unnd:PROG/565532@IKBTV.BMM (NOTE: at most 10, seperated by ",")
sviewhistory=Unnd:PROG/565532@IKBTV.BMM (NOTE: at most 10, seperated by ",")
fcount=10
wcount=5
charset=UTF-8

Any further information needed?


[Thumbnail for connectionrate.png]

[Thumbnail for throughput.png]

Mei Zhiguan
Greenhorn

Joined: Feb 05, 2012
Posts: 8

Hey, am I asking a wrong question at a wrong place? Isn't this a Forum on Tomcat?
I don't think this problem deserves no attention.
William Brogden
Author and all-around good cowpoke
Rancher

Joined: Mar 22, 2000
Posts: 12761
    
    5
The reason you are not hearing from me is because I am completely mystified by your problem. I can think of no reason that those simple requests could be keeping those requests occupied in the P state. I monitor my own server with the management app and I don't think I have ever hit it where a Thread was in the P state.

Bill
Rob Spoor
Sheriff

Joined: Oct 27, 2005
Posts: 19653
    
  18

Bill's not the only one who's mystified. I don't think the reason you're not getting any answers is because nobody wants to answer your question, I think it is because nobody can answer it.
Tim Holloway
Saloon Keeper

Joined: Jun 25, 2001
Posts: 15958
    
  19

And the reason you're not hearing from me is that I only look at this kind of stuff when there's a problem, so I don't keep up-to-date on what it all means. As far as I could tell, you're not actually experiencing performance or reliability issues, so I didn't worry about it.

I mean, for a small phenomenal fee, I could dig into your server and do a serious analysis and give you an answer, but in my experience, companies only pay phenomenal fees to people who come in and make trendy noises ("Cloud, Cloud, Cloud"), not to solve things that are only mildly annoying.

There's no conspiracy of silence here. When you don't get an answer on JavaRanch, it means we probably don't know. Since we're not being paid, we have no incentive to sound wiser than we are other than simply for ego inflation.


Customer surveys are for companies who didn't pay proper attention to begin with.
Mei Zhiguan
Greenhorn

Joined: Feb 05, 2012
Posts: 8

Thank you for your replies, and I will post the answer when I get it!
Mei Zhiguan
Greenhorn

Joined: Feb 05, 2012
Posts: 8
Problem RESOLVED !!!

Solution:
1. minimize keepAliveTimeout from default 20000 to 3000
2. maximize maxThreads from default 200 to 2000

Effect:
Most of threads come back to "R" or "S" status, and tomcat responses quite quickly. Be precisely:
1. busy threads of tomcat declined at least 60%;
2. respond time decreased at least 80%;
3. throughput increased around 10%;
4. heap size decreased around 10%.

Analysis:
Tomcat default configuration (maxThreads=200, keepAliveTimeout=3000) is for normal web application, which means that
1. 90% of requests are GET, the remains are POST and others;
2. 1 primary GET triggered by user is followed by a bunch of GET triggered by browser.
However, my application is:
1. 100% of requests are POST;
2. every POST requests are triggered by another server without any consequent request;
This determines that the 3000ms of keep-alive is nothing but a waste of time.

That's it. Not sure whether it is correct. Any comments?

Tim Holloway
Saloon Keeper

Joined: Jun 25, 2001
Posts: 15958
    
  19


2. every POST requests are triggered by server without any consequent request;


That's not right. Whether it's a GET or a POST, if it's HTTP, servers cannot "trigger" unsolicited responses. Some client somewhere had to issue that POST as an HTTP request.

The primary reason why POSTs are offending and not GETs is likely that you are carrying a lot of data in the POST request body. GETs tend to be self-limiting. The larger the request, the longer it's going to take to pipe it in.

There's also a hint that perhaps the application's per-request processing time is a bit long, but that would be have to be measured.

Mei Zhiguan
Greenhorn

Joined: Feb 05, 2012
Posts: 8
Tim Holloway wrote:

2. every POST requests are triggered by server without any consequent request;


That's not right. Whether it's a GET or a POST, if it's HTTP, servers cannot "trigger" unsolicited responses. Some client somewhere had to issue that POST as an HTTP request.

The primary reason why POSTs are offending and not GETs is likely that you are carrying a lot of data in the POST request body. GETs tend to be self-limiting. The larger the request, the longer it's going to take to pipe it in.

There's also a hint that perhaps the application's per-request processing time is a bit long, but that would be have to be measured.



Hmm... there's a misunderstanding about "triggered by server", I've updated previous post to "triggered by another server".
 
It is sorta covered in the JavaRanch Style Guide.
 
subject: Almost all threads are in "P" status
 
Similar Threads
Tomcat daemon
java.lang.OutOfMemoryError: Java heap space
Long running request returns no HTTP response
Popular training institutes in Chennai for LAMP @ CEGONSOFT
Hot Deployement in Tomcat