aspose file tools*
The moose likes Testing and the fly likes JMeter Stress Testing Logic Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Engineering » Testing
Bookmark "JMeter Stress Testing Logic" Watch "JMeter Stress Testing Logic" New topic
Author

JMeter Stress Testing Logic

Sabine Chemaly
Greenhorn

Joined: Dec 08, 2011
Posts: 16
Hi,
I'm new to JMeter.
What I need to do is stress testing a web application when 3000 users are registered.
1- In the Thread Group does 'Number of Threads' means number of concurrent users? If not what is it if we want to compare to real life scenario?

2- Let's say we have 1500 concurrent users, should I put 1500 as Number of Threads?
Or is it logical if I do it this way: I check with the company what's their target throughput from 1500 users and I check if we are able to reach it by taking any number of threads initially and increasing it to get to the target one. (This way we would have checked whether the customer is able to have the target throughput with a descent response time)

Any good advice will be appreciated.
Thank you in advance
Karthik Shiraly
Ranch Hand

Joined: Apr 04, 2009
Posts: 501
    
    5
Yes, each thread in jmeter thread group is like a virtual user simulating the actions of a real user.

When creating a test plan, it's important to keep in mind the actual use cases - that is, how do different users use the system.
Any system will have atleast a dozen and usually many more use cases, related to its domain.
Not all users will be performing the same use case.
And not all 3000 *registered* users will be using the system every second (registered is not the same as concurrent).
So you need to work out a representative usage scenario - that is, how many percent of users perform which use case and how often?
Usually, this input comes from the functional specs, the client and/or domain experts.

If it's a new product, take the figure you get from them with a pinch of salt, because the client may not have any idea either and is likely to produce a number out of thin air that sounds good to their ears, unless they have based it on some objective survey or study.

But if it's not a new product, then they can provide such data from their access or analytics logs.

The other number you need is the target number of concurrent users that system should support while maintaining some quality of service (like response times within some defined limit).
Again, this number should be reached using a representative sampling based on different use cases
The client's input and their specs also should be taken, though they are only likely to give you a ceiling on this number.
Take that ceiling number and add a buffer like 20% to it - that is the number the system should handle. It's never good idea to aim for exactly the estimated number given by client - it's better to aim for something slightly higher.

This number is only a desirable upper limit, but the system could start failing much earlier.
What the engineers of the system are interested in is not just a yes/no answer to "does it work with 3000 concurrent users?".
What they are interested in is how do their systems perform both externally (ie, as seen by users via response times, availability, zero errors, etc) and internally (ie, resources like memory usage, IO speeds, etc) - at various load levels.
If they can find system faults at lower levels, then they can think it over, redesign the system, try out various scenarios and possibly make it endure much higher levels.

So you should start with a small number of concurrent users and then build it up in increments.
All the servers should have monitoring software installed, which can alert if any resource goes above limit.
At each increment, measure response times and the standard deviation (SD).
The SD is a measure of the reliability of those results - if the SD is small, then response times are fairly consistent and close to the mean value.
But if the SD is large, it's possible the system is already starting to buckle at that load level.
Developers have a habit of taking the most optimistic values from a set of results and ignoring other results as anomalies. The SD will help decide whether some results are indeed anomalies, or whether system is buckling.

The other thing to keep in mind, when having large numbers of users like 1500 concurrent users, is - don't test it from a single test machine running jmeter.
The problem with a single machine is that it's likely to clog the outgoing network bandwidth of your test machine, making the test unrealistic.
Instead, distribute the testing over a cluster of test machines, with each machine simulating a smaller number of users that its bandwidth can handle.
The jmeter wiki explains how to set up such a cluster of test machines and run a test plan from multiple machines.

Alon Girmonsky
Greenhorn

Joined: Jan 27, 2012
Posts: 1
It depends highly on how extensive is your script. The general recommendation is to keep the number of threads below 300 per engine. I can tell you that there are various factors that will affect JMeter and its ability to generate this kind of traffic: bandwidth, hits/s and MEM consumption.
For 3,000 users I would suggest 10 engines + running headless. The console should have a lot of memory as everything goes through the console.
Bear in mind that if your environment is not well configured, it will either provide you with false confidence or be very challenging as engines will dump their core and you will never know why.

Consider using http://blazemeter.com/ . It provides an out-of-the-box JMeter distributed environment. A single test can generate a load of more than 20,000 users.

Best,
Alon.
http://blazemeter.com
The JMeter Cloud
Sabine Chemaly
Greenhorn

Joined: Dec 08, 2011
Posts: 16
Hi,
Thank you for your replies.
Karthik you reply is REALLY helpful but I have some questions.
1- Regarding the SD, when it's a big value, I don't take the response time value into consideration? I only save the values where the SD values are small?
2- When distributing the testing, how do I specify the number of threads to be tested on each machine? and how do i read the values for each machine separately?
I just found links that show how to configure the distributed environment but not how to do the actual testing (if you can recommend any link, that would be perfect!)

Thank you in advance
Karthik Shiraly
Ranch Hand

Joined: Apr 04, 2009
Posts: 501
    
    5
Sabine Chemaly wrote:
1- Regarding the SD, when it's a big value, I don't take the response time value into consideration? I only save the values where the SD values are small?

Large SD means response times are not consistent. It means many of the measurements deviate significantly away from the mean line. You should not ignore those results, especially if multiple tests consistently return large SDs - it means the system is buckling under load.
But if SD is small and only some response times are high, then it means most measurements are close to mean line and the high measurements are probably anomalies.


2- When distributing the testing, how do I specify the number of threads to be tested on each machine? and how do i read the values for each machine separately?
I just found links that show how to configure the distributed environment but not how to do the actual testing (if you can recommend any link, that would be perfect!)

In distributed setting, each slave machine receives the same test plan from server. Each slave will run as many threads as thread group specifies. If you want 600 concurrent users and you have 2 slave machines, then set threadgroup to 300.
Why would you want to read the values for each machine separately? The idea is that the slaves report back all results to master. I have not tried out capturing individual results, but perhaps adding a "save to file" or "simple data writer" listener to test plan will capture that data...

I don't have any good link to share other than ones already on google. I learnt many good principles from the book "Pro Java EE 5 Performance Management And Optimization" (though it's neither specific to JMeter nor goes deep into JMeter). The rest is learnt from experience, mailing lists and discussion forums.
Sabine Chemaly
Greenhorn

Joined: Dec 08, 2011
Posts: 16
Hi again,
1- Since the values with the small SD are the accurate ones then why would I want to display the non accurate values (values with with high SD, where the system is not stable)
2- Beginning with which value is the SD considered to be a high value?
Thank you again
Karthik Shiraly
Ranch Hand

Joined: Apr 04, 2009
Posts: 501
    
    5
Sabine Chemaly wrote:1- Since the values with the small SD are the accurate ones then why would I want to display the non accurate values (values with with high SD, where the system is not stable)

I feel you have misunderstood the concept. SD is not about "accuracy" of the values. High SD does not mean the values are "wrong" in any sense.
What it means is that the response times are varying wildly at that load level, which is a sign that the system under load is showing signs of stress (this could be due to memory overuse, I/O bottlenecks, high CPU usage, network clogging, etc). It's a sign that some resources are in shortage. If the servers are being monitored, knowing when a system is starting to buckle is a very important threshold value for developers - it's a reference line that can then be improved upon by redesigning, and compared against.

2- Beginning with which value is the SD considered to be a high value?

There's no globally correct answer for that. It's a matter of judgement, and depends on what can be considered a "good" response time for a particular use case. It's pretty subjective, though you can use some rules of thumb. In fact, in trying to frame an answer to the previous point in a simple way, I stumbled upon this article which I think will be useful to you too. In fact, it's a series of articles all of which have to do with stress testing with lots of useful tips.
Sabine Chemaly
Greenhorn

Joined: Dec 08, 2011
Posts: 16
I understand the idea now.
Thank you for the clear explanation and for the article.
I will go a little bit out of subject now but I need a one last suggestion from you, what do you suggest as software monitoring tool to be installed on Windows server 2008 and which will give me graphs that I could use in my report?
Karthik Shiraly
Ranch Hand

Joined: Apr 04, 2009
Posts: 501
    
    5
You're welcome!

It's been a while since I worked on Windows production environments, so my information may be outdated, but Windows already had an inbuilt performance monitoring tool named Perfmon. In fact, it's still there on my Windows7 dev machine (type "perfmon" in Run dialog) - so I guess it's still around. You can add lots of system counters through it and log them. Perfmon tool can later import the logs and plot data.

I mostly work on Linux production environments where there are some reputed products for network wide monitoring - ganglia, nagios, cacti, collectd, munin,....
It looks like cacti can be installed on Windows too. I'm not sure about the others. If you have / will have multiple Windows servers, then these tools are capable of giving dashboard views of all machines on a single web page. Doing that with windows perfmon may prove cumbersome.
Sabine Chemaly
Greenhorn

Joined: Dec 08, 2011
Posts: 16
Hi again,
If I don't want to perform distributed testing.
If I have the below PC specifications, will they be enough?
PC specifications: HP DL360 (or DL380) Generation 6 or 7, 2 x XEON. 8-16GB ram
Thank you!
Karthik Shiraly
Ranch Hand

Joined: Apr 04, 2009
Posts: 501
    
    5
If you want to find out if one test machine is enough, then run perfmon on the test machine too and keep an eye on the network bytes/sec, memory and CPU usage as you increase number of users.
The limit on the network would be whatever is the slowest link in your network (usually the link from your router to ISP, or from your client's ISP to their server).
When one of them is close to its limit, then that's how many users can be safely generated by that test machine for your particular test plan.
Since everybody's test plans and environments are different, it's impossible to say if a machine is good enough just from its hardware configuration.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: JMeter Stress Testing Logic