Win a copy of Re-engineering Legacy Software this week in the Refactoring forum
or Docker in Action in the Cloud/Virtualization forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Why web applications fail in Production ??

 
Ramm Abhiram G
Greenhorn
Posts: 7
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi ,

We have developed a web application and tested well.
We have also performed memory leak tests and observed that there are no
memory leaks in the application.
My application performs well for one or two days and fails suddenly in production environment.

Generally why web applications fails in Production environment ?
What could be the possible reasons and how to prevent them to happen?

Please help me out.

Regards,
Abhiram
 
Ilja Preuss
author
Sheriff
Posts: 14112
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
There is a myriad of possible reasons for an arbitrary web application to fail in an arbitrary way - from a bug in your code, a bug in the JVM, a neutrino hitting an electron and thereby flipping a bit, to the cleaning lady plugging in her vacuum cleaner into the same electric circuit as the server.

How exactly *does* your application "fail"?
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13056
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Perhaps the real-world load causes an out of memory problem or exceeds some other system limitation.

Did your testing include multiple "simultaneous" requests?

How did you decide that you had no memory leaks?

Surely there must be more information in the log files. Also, how close is the production environment to the test environment in terms of servlet container, Java version, operating system, database, etc....

I am researching for an article on web application testing, so this is an interesting problem.

Bill
 
Jimmy Clark
Ranch Hand
Posts: 2187
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Generally why web applications fails in Production environment ?
What could be the possible reasons and how to prevent them to happen?


In most cases, web applications fail when the power to the server machine goes out. This can be either beacause someone unplugged the machine or there was a nearby storm that cut power. These situations can be prevented by having a sufficient alternate power source to take over when the main power goes down. Also, it is helpful to have the plug to the server located in a very visible place so that it can be noticed quickly when it is unplugged.

Hope this helps!
 
Ramm Abhiram G
Greenhorn
Posts: 7
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi All,
Thanks for your replies.
Following are my observations.

1) Server power supply is proper.
2) There is no memory leak in the system, because i couldnt find any
OutOfMemoryError in the log files. In fact there is no error in the log
files.
3) The testing environment and Production environment are exactly same.
4) We have performed the load testing of our application to check if it
handle multiple concurrent user requests. It handles upto 500 users.
In real time the load may not be that much.

Any other reasons ?
Generally how to debug these kind if situations in production environments ?

Please help.

Regards,
Abhiram
 
Ulf Dittmer
Rancher
Pie
Posts: 42967
73
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
This is the question you need to answer:
How exactly *does* your application "fail"?

Without knowing that it's hard to tell what might be going on.
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13056
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
With my Tomcat server I routinely use the "management" app to monitor the system for things like free memory, number of Threads, longest running Thread, number of errors, etc. -

What facilities for monitoring do you have?

Bill
 
steve souza
Ranch Hand
Posts: 862
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
As many people have said there are many possible explanations why a server would 'go down'. If you went to a doctor and said, "I am sick". Would you expect them to solve your problem? No, you have to provide more information.

As ridiculous as the above sounds it is the same thing as saying, "my server is down why?". Look in this forum and you will see that we always answer this question in the same way. "Give us more details and use tools to monitor your application".
[ June 19, 2008: Message edited by: steve souza ]
 
Ramm Abhiram G
Greenhorn
Posts: 7
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi All,

Really i dont have any other information to share. I know the information which i am providing is vague. But I coudnt find out any error messages from the logger also.
Some times the application runs for 2 days, some times it runs for 3 days.
Then it is getting down suddenly without any errors.
That's why i am asking you guys how to debug these kind of situations ?

Regards,
Abhiram
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13056
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
There is one bit of evidence conspicuously lacking:
Then it is getting down suddenly without any errors.


What does that mean? Please pick one of the following:
1. The entire server dies including everybody elses applications and the operator has to restart the whole thing.
2. Requests to your app get an error response - what is the error? Even the error number would be a help.
3. Requests to your app get no response and just sit there for hours.

How are requests to this web app being made? There are so many possiblities - AJAX style web page, plain HTML, custom client, etc etc.

Is this a Tomcat server? If so, can you use the Management app?
Which operating system? Java version?

Bill
 
steve souza
Ranch Hand
Posts: 862
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
<<Really i dont have any other information to share.>>
Is this happening at a certain time of day? Does it happen after certain pages are invoked? Can the server deliver a simple html, or gif file when it is "down" or can it respond to nothing? Is the process running or is it down? If the process is up how much memory is it consuming?

No one could solve your problem the way it is currently framed. Software can go wrong in too many ways to give an answer when no facts are provided. So, no matter what the problem is if you want to solve it you need to make your system give you more information about its state at any given time. This is called monitoring or profiling. Here is what I suggest.

Use the link to jamon below to help you do the following. None of the suggestions below require code changes and you can have them running in a few minutes.:
  • Track jdbc calls and sql. This will show if any exceptions are thrown if any queries are slow etc. (assuming your program issues sql. you didn't mention that)
  • Use the jamon tomcat valve to track all page performance stats, exceptions, bytes sent, status codes and more.
  • If you use log4j (and you should use log4j or some other logging package), then use the log4j appender to track stats about what you are writing to the log. If you aren't putting enough info in the log to diagnose the problem then put it in.
  • Use any other tools that can help you understand what is happening in your program. Right now your application is a windowless building, and you are standing on the outside trying to look in. You need to give it windows!


  • You must always write your application in a way that you make what happens inside it transparent as possible (give it windows). Logging, web page monitoring, jdbc/sql monitoring, exception monitoring, resource monitoring are all a part of this.
    [ June 14, 2008: Message edited by: steve souza ]
     
    Rajah Nagur
    Ranch Hand
    Posts: 239
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    Originally posted by Ramm Abhiram G:

    3) The testing environment and Production environment are exactly same.


    Are you sure of this? To the exact OS patches, Server patches?
    The problem could be anywhere - OS, app/web server settings, bad code, app configuration etc

    Are you facing a JVM hang kind of situation? Then you should consider taking a thread dump to analyze.
    Need more info such as :
    1. What kind of app are you talking abt? Database intensive or process intensive?
    2. What is the configuration? OS, Server, JVM, Database etc
    3. What is the GC setting for you app?
    4. Please share your app's profiling info
     
    • Post Reply
    • Bookmark Topic Watch Topic
    • New Topic