planning to desing a stress test tool

[vinod kumar h v]

What are all the requirements for designing the tool?? i know one thing, that is i should simulate a huge number of users for generating requests... One thing is i need to designa tool which is simple and easy to use.... Consider i want to design the tool which works for only HTML paged web sites.... (i do know some parsing is required).. so please guide me on this... first all the requirements to start with..

[William Brogden]

First requirement - get really familiar with the java.net package, especially the URLConnection and HttpURLConnection classes.

The open-source HttpClient toolkit is well developed.

Bill

[vinod kumar h v]

Hi william,

yeah i have started with it. I am familiar with URLConnection and HttpUrlConeection class. Right now i have so many things goig in my mind... i do know that i need to simulate a huge number of requests to generate load on the server.. So one thing is, can i use a single port for sending huge number of requests and receive response for those requests ..... And whats the way to measure the bottleneck on the server... in what form i should get results...??? do i need parsing ???

Thank you in advance..

[Tim Holloway]

The server port remains constant, By default it's 8080.

The receiver port is dynamically and automatically assigned to the client on a per-request basis, because HTTP is not a continuous-connection protocol. You don't set it - the OS tcp/ip stack does.

If you you want to see how a Java stress tester looks, take a look at jmeter.

[William Brogden]

The program simulating multiple clients will have to use a separate Thread to execute each request (or user session if you are simulating more than simple requests).

Start by designing a Runnable class to execute one request and process the response. I would record the time delay between flushing the request and getting the full response. Be sure to check the response status code.

As the number of "simultaneous" client requests increases, you may become more limited by the client simulator CPU time than the server. So you may need multiple machines on the network running client simulators. When I did this I could run about 50 clients from a single machine, each running a sequence of requests defined by an XML script.

Bill

[vinod kumar h v]

Hi tim,
Hello william,

Tim , yeah i do know about jmeter, and i tried using it. but i didnt get the entire working process of jmeter. i searched the web and i got some posts suggesting how to use it. But all they have given is for testing the tool in a local server. I have already designed a stress testing tool . But i don know whether the results am getting are correct or not. i need to compare the results with some standard testing tool....Do i need to measure the response time while stress testing??? Am not aiming to design performance testing tool where i need to measure the response time. I guess what am sayin makes sense to you.

@both(William and Tim)

The reason i posted is to know whether the tool i have already designed is working correctly or not...

A brief description of my tool is like this...

1) My tool takes two commandline arguments, one the url of the website and second the number of users it wants to simulate.

2) Then the tool generates the number of requests specified, and then calls the spider module.

3) the creation of threads and the call to the spider module is done in a loop and the status is checked with "HTTP_OK". and am displaying all the URLs that each thread crawled that means each page is requested 50 times if the second argument is 50....


Now tell me how to measure the stress ?? This is the current work am done with.. what are the other thing that i need to do??

[Andrew Monkhouse]

I could not easily tell from your posts whether you are trying to run the tests on the same computer as the Tomcat server. I think you are running it on a different box, but I am not sure. For the benefit of anyone else reading this post who is not sure why this might matter - rule 1 of stress testing is not to run anything on the computer that is running the system under stress. In other words, if you were testing a tomcat application that has database connections and sends emails out, then you would have multiple servers in the mix. One that only runs Tomcat and your web application, one that runs the database (and possibly the email sink), and one (or more) that is running the testing software.

You mentioned that all you were able to find on JMeter was for testing on a local server - there is far more to JMeter than just that. JMeter has capabilities that were designed for stress testing. When doing a stress test you would normally set JMeter up to write the most basic of statistics out to a file, then process the statistics later - processing in real time is a bad idea. Normally you would also want to run JMeter in headless mode. You may also want to run multiple JMeter instances in server mode, all controlled by one master JMeter application.

vinod kumar h v wrote:
Now tell me how to measure the stress ?? This is the current work am done with.. what are the other thing that i need to do??

Once again, I think you might want to look at JMeter. Not necessarily looking at using it if you are already happy with what your program is doing, but more for looking at what statistics it gives you.

Most people care about transactions per second (TPS) or Throughput and elapsed time for transactions. You probably care about the number of successful transactions versus the number of error transactions.

Minimum, maximum, average, and standard deviation are all good statistics for those metrics.

JMeter also has plugins that allows you to monitor CPU and memory load. It is good to monitor these on both the computer under load and the computer running the test. I have been caught before in a situation where we ramped up the load to such a point that the computer running the test could no longer handle the TPS, so we were seeing artificially low numbers. Once we spread the load to 4 test clients, our throughput became reasonable.

Unlike a load test tool, I would expect any decent stress test tool to have a configurable "ramp" for the test. As an example, I might want to run 1000 loops, where each loop starts at 500 TPS and ramps up to 1500 TPS. Then I would consider graphing those results so that I can see where we consistently get to the edges of reasonable load.

Note that it might not be reasonable to try and fix performance issues at the failure point. What you will typically find is that an application will process TPS in an almost linear fashion up to a certain point, then start leveling off. Find that point and fix the bottleneck there, then go back and do your tests and fixes all over again. This way you will actually be fixing the same problems that will be affecting your users first. If you stress your application to breaking point, then try to fix the biggest bottleneck, you may be fixing a problem that only affects a small number of users.

[William Brogden]

Recording raw data and processing off-line is certainly the way to go.

In addition to normal statistics, I suggest that a histogram of response times can be very interesting as one or two really long times can throw off the averages.

Of course, just repeating the same request 50 times could be very unrealistic if the service is more complicated than just serving a static page.

Bill

[vinod kumar h v]

Thank you Andrew and William,

Andrew, as you guessed am measuring stress in terms of number of successful transactions versus number of error transactions. And was trying in real time. Doing the tests on a web site. Now am thinking of switching to offline . And the example you gave "ramp up from 500 TPS to 15000 TPS then plotting graph and finding the breaking point is certainly the best idea. Thank you once again . Now how to measure performance of the server??? and now i will switch to stress testing locally using tomcat server.

William, yeah my service is just serving the static pages.

[Andrew Monkhouse]

Before we can begin to answer that, we need to know what your definition of "measur[ing] the performance of the server" is.

Currently it is a bit like someone asking me to come and get their car going. I do not know if they mean that they need fuel, or if their battery is dead, or if their tires are flat, or if the engine needs to be rebuilt, or ...

Similarly, when you say you want to measure the performance of the server, I don't know if you are talking about CPU, or Memory, or Hard Drive, or Network, or ... Even for those statistics there are often other related questions. For example, if you said you were interested in checking the Hard Disk performance, then you should probably mention whether you are interested in writes, reads, or random-access. Or for memory, are you talking about physical or virtual, writes versus reads, page faults, etc.

[vinod kumar h v]

To Andrew

Am sure i didnt make clear what i wanted say to you.My fault

I want to measure the response time of the server at different loads like 500 TPS, 1000 TPS, But you already told me thats one way of measuring stress . Can i use the same to measure performance ??? (the throughput of the server in a given bandwidth, given processor and the RAM) Am just interested in measuring the performance in one way and increase its performance by adding hardware ..

[Andrew Monkhouse]

You can, however you really need to learn to qualify your answers when you go down that path, and learning statistical terms will help you in this regard.

Bill mentioned that you probably want to have a histogram of response times. I suggested that you would want to repeat the same tests many times. And we have talked about ramping up the tests to find the point where the system is no longer able to handle the load and starts getting stressed.

Based on all of this, it would be able to write up your data in a way that gives indications as to performance. For example "An otherwise unloaded i7 with 8Gb RAM, RAID 10 drives, and a 1 Gb Ethernet connection was able to reliable sustain an average of 1000 TPS and 0% errors 98% of the time with a standard deviation of 10 TPS. Average payload for these tests was 10 kB with a standard deviation of 1 kB." (This would be the BLUF to a much larger document describing your testing methodology, that would explain why you chose your parameters, and where the system started feeling stress, and what the stress point was at that time.