Can not get stable result when using multiple threads to calculate the average response time

[tuntony wei]

Hi, guys:
I met a problem when I use multple threads to calculate the avaerage reponse time. My problem is I can not get the stable result, I put my code here and could anybody give me some suggestion.

1. My goal is to test the performance of a compnenet in our product. This is a J2ee product. Based on design, it is able to handle concurrent user's request.
2. my idea is to make a simple HTML, a link on the HTML can invoke one performance test case. In the test case, I record start time before the component starting to do the business logic, then I record the end time as soon as the component finished the job. So my response time = end - start. Finally I put it into a result.csv file.
3. I use JMeter to generate 100 concurrent user request to the HTML

Here is my code:

1. HTML:
<a href="TestServlet?command=KPI001" target="_self">KPI001</a>

2. TestServlet.java

... ...
protected javax.naming.Context ic = null;
... ...
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
             // TODO Auto-generated method stub
             String commandToExecute = null;
             commandToExecute = request.getParameter("command"); // this one will also be used by Jmeter to generate a GET request
             String result = "success";
             try {
                    ic = new InitialContext();
                    if (commandToExecute.equals("KPI001")) {
                           doKPI001();
                           }
                    else if (commandToExecute.equalsIgnoreCase("KPI002")){
                           doKPI002();
                           }
                           ... ....
                     } catch (NamingException e) {
                           // TODO Auto-generated catch block
                           e.printStackTrace();
                          }

             ...  ...		
             }
           
private void doKPI001() {
           TargetPerformance perf = new TargetPerformance ();
           try {
	   perf.startTest(filename,"D:/ADP/VerifyTmplate.xml",casename,"false", "D:/ADP/log.html");
                } catch (InvalidMappingException e) {} 
                   catch (IOException e) {}		
	       }

3. TargetPerformance.java

... ...
public static PrintWriter fwriter;
... ...
public void startTest(String name, String verifyXml, String testcasename, String OnlyVerify, String log) throws InvalidMappingException, IOException{
         long startTime = 0;
         long endTime = 0;
         String testResult="";
         String user = Thread.currentThread().getName();
         ...
         doThinkTime(new Random().nextInt(19998) + 2); //2 - 20 seconds thinking time
         ... 
         startTime = System.currentTimeMillis();
         Future<AAValueOutputs> exec = transMachine.submitOperation("prams");  // this is the key business logic which I have to measure
         exec.get();                                                                                             // wait for finished state
         endTime = System.currentTimeMillis();
         ...
         result = vlidate(testcasename,verifyXml);
         printf("result.csv", user, startTime, endTime, testResult);                                   
}

public static void printf(String testcase, String pricessId, long start, long end, String result) throws IOException{
        SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd'-'HH:mm:ss");
        dateFormat.setTimeZone(TimeZone.getTimeZone("GMT"));
        if(fwriter ==null){
                    fwriter = new PrintWriter(new FileOutputStream(testcase+".csv")); 
         }		
         fwriter.println(dateFormat.format(new Date(System.currentTimeMillis()))+","+pricessId+","
		+String.valueOf(start)+","+String.valueOf(end)+","+String.valueOf(end-start)+","+result);
         fwriter.flush();
         }

My result:
if I run 100 user request through Jmeter 1 by 1, then my average response time is 2+ seconds, but if I run 100 concurrent user through Jmeter, then my average response time was around 4 seconds. This makes me worried and confused

Question:
1. Dirrectly feeling, should I worry about the start tiem and end time? When mutiple users come into the startTest method, is there any risk for them to be re-written?
2. In my code, I did not use any synchronized method/block because I think I do not need to. My understanding is that when concurrent users make requests, the TestServlet always did "TargetPerformance perf = new TargetPerformance ()", and then when the users did the "perf.startTest()", different user alwasy has its own copy of start time, end time and test result.
3. This is what I think. I think it should be this way but this part is also the part I worried most
Am I right? Could anybody give me some suggestion?
Thank you so so so much

[Himanshu Gupta]

if I run 100 user request through Jmeter 1 by 1, then my average response time is 2+ seconds, but if I run 100 concurrent user through Jmeter, then my average response time was around 4 seconds. This makes me worried and confused

When you run one by run do you send the other request just after the first one or you wait to get the results of the previous request?

When you are running 100 concurrent threads you are increasing the load on the server. The request flood comes and server starts handling them. When it is one by one the server gets more time to process the 100 request.

[tuntony wei]

Himanshu Gupta wrote:

if I run 100 user request through Jmeter 1 by 1, then my average response time is 2+ seconds, but if I run 100 concurrent user through Jmeter, then my average response time was around 4 seconds. This makes me worried and confused

When you run one by run do you send the other request just after the first one or you wait to get the results of the previous request?

When you are running 100 concurrent threads you are increasing the load on the server. The request flood comes and server starts handling them. When it is one by one the server gets more time to process the 100 request.

When I run it one by one, I run it through JMeter. Basically, Jmeter send the 1st requst to HTML and then wait for a HTTP code such as 200. And then, Jmeter will send the 2nd and so on... With this method, my response time was around 2 seconds

With concurrent threads, the response time I got was around 4 seconds. But my understanding is that under the concurrent test, the response time should be close to the 2 seconds. That's why I worried about if statime/endtime were messed up when multiple threads comming. Do you think my code, or my test method, is OK? Do you mean the longer response time supposes to happen when dealing with the multple threads?

Thank you so much

[steve souza]

Being as your start and stop times are local variables in TargetPerformance your code is thread safe. Why would you be surprised that 100 simultaneous users would be slower than 1 sequential one. I would expect that. You should get an idea about this being correct simply by timing with a wall clock the total time of your tests and dividing by 100 to get average time.

Having said that I would use JAMon (or other tool of your choice) to do the monitoring. It is much more flexible. It can easily monitor your business methods (MonitorFactory.start("doKPI001()") and do a lot more than that such as monitor your jdbc, page hits, ejb calls without changing your code at all. And for this reason it allows you to monitor in production as well as dev which is where your problems are likely to occur.

In fact I just made a release today that also lets your monitor your unix/linux scripts with jamon by calling the unix scripts I provide such as jstart.sh/jstop.sh and jadd.sh (adds an arbitrary value).

Steve - www.jamonapi.com

[tuntony wei]

steve souza wrote:Being as your start and stop times are local variables in TargetPerformance your code is thread safe. Why would you be surprised that 100 simultaneous users would be slower than 1 sequential one. I would expect that. You should get an idea about this being correct simply by timing with a wall clock the total time of your tests and dividing by 100 to get average time.

Having said that I would use JAMon (or other tool of your choice) to do the monitoring. It is much more flexible. It can easily monitor your business methods (MonitorFactory.start("doKPI001()") and do a lot more than that such as monitor your jdbc, page hits, ejb calls without changing your code at all. And for this reason it allows you to monitor in production as well as dev which is where your problems are likely to occur.

In fact I just made a release today that also lets your monitor your unix/linux scripts with jamon by calling the unix scripts I provide such as jstart.sh/jstop.sh and jadd.sh (adds an arbitrary value).

Steve - www.jamonapi.com

OK, great, now I am talking with our developers about my finding. And good to know the JAMon, we do need some good tool to monitor our KPI. I will try it. Thank you so much again