Two Laptop Bag*
The moose likes Performance and the fly likes Why web applications fail in Production ?? Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Murach's Java Servlets and JSP this week in the Servlets forum!
JavaRanch » Java Forums » Java » Performance
Bookmark "Why web applications fail in Production ??" Watch "Why web applications fail in Production ??" New topic
Author

Why web applications fail in Production ??

Ramm Abhiram G
Greenhorn

Joined: Jun 12, 2008
Posts: 7
Hi ,

We have developed a web application and tested well.
We have also performed memory leak tests and observed that there are no
memory leaks in the application.
My application performs well for one or two days and fails suddenly in production environment.

Generally why web applications fails in Production environment ?
What could be the possible reasons and how to prevent them to happen?

Please help me out.

Regards,
Abhiram
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
There is a myriad of possible reasons for an arbitrary web application to fail in an arbitrary way - from a bug in your code, a bug in the JVM, a neutrino hitting an electron and thereby flipping a bit, to the cleaning lady plugging in her vacuum cleaner into the same electric circuit as the server.

How exactly *does* your application "fail"?


The soul is dyed the color of its thoughts. Think only on those things that are in line with your principles and can bear the light of day. The content of your character is your choice. Day by day, what you do is who you become. Your integrity is your destiny - it is the light that guides your way. - Heraclitus
William Brogden
Author and all-around good cowpoke
Rancher

Joined: Mar 22, 2000
Posts: 12759
    
    5
Perhaps the real-world load causes an out of memory problem or exceeds some other system limitation.

Did your testing include multiple "simultaneous" requests?

How did you decide that you had no memory leaks?

Surely there must be more information in the log files. Also, how close is the production environment to the test environment in terms of servlet container, Java version, operating system, database, etc....

I am researching for an article on web application testing, so this is an interesting problem.

Bill
Jimmy Clark
Ranch Hand

Joined: Apr 16, 2008
Posts: 2187
Generally why web applications fails in Production environment ?
What could be the possible reasons and how to prevent them to happen?


In most cases, web applications fail when the power to the server machine goes out. This can be either beacause someone unplugged the machine or there was a nearby storm that cut power. These situations can be prevented by having a sufficient alternate power source to take over when the main power goes down. Also, it is helpful to have the plug to the server located in a very visible place so that it can be noticed quickly when it is unplugged.

Hope this helps!
Ramm Abhiram G
Greenhorn

Joined: Jun 12, 2008
Posts: 7
Hi All,
Thanks for your replies.
Following are my observations.

1) Server power supply is proper.
2) There is no memory leak in the system, because i couldnt find any
OutOfMemoryError in the log files. In fact there is no error in the log
files.
3) The testing environment and Production environment are exactly same.
4) We have performed the load testing of our application to check if it
handle multiple concurrent user requests. It handles upto 500 users.
In real time the load may not be that much.

Any other reasons ?
Generally how to debug these kind if situations in production environments ?

Please help.

Regards,
Abhiram
Ulf Dittmer
Marshal

Joined: Mar 22, 2005
Posts: 41068
    
  43
This is the question you need to answer:
How exactly *does* your application "fail"?

Without knowing that it's hard to tell what might be going on.


Ping & DNS - my free Android networking tools app
William Brogden
Author and all-around good cowpoke
Rancher

Joined: Mar 22, 2000
Posts: 12759
    
    5
With my Tomcat server I routinely use the "management" app to monitor the system for things like free memory, number of Threads, longest running Thread, number of errors, etc. -

What facilities for monitoring do you have?

Bill
steve souza
Ranch Hand

Joined: Jun 26, 2002
Posts: 860
As many people have said there are many possible explanations why a server would 'go down'. If you went to a doctor and said, "I am sick". Would you expect them to solve your problem? No, you have to provide more information.

As ridiculous as the above sounds it is the same thing as saying, "my server is down why?". Look in this forum and you will see that we always answer this question in the same way. "Give us more details and use tools to monitor your application".
[ June 19, 2008: Message edited by: steve souza ]

http://www.jamonapi.com/ - a fast, free open source performance tuning api.
JavaRanch Performance FAQ
Ramm Abhiram G
Greenhorn

Joined: Jun 12, 2008
Posts: 7
Hi All,

Really i dont have any other information to share. I know the information which i am providing is vague. But I coudnt find out any error messages from the logger also.
Some times the application runs for 2 days, some times it runs for 3 days.
Then it is getting down suddenly without any errors.
That's why i am asking you guys how to debug these kind of situations ?

Regards,
Abhiram
William Brogden
Author and all-around good cowpoke
Rancher

Joined: Mar 22, 2000
Posts: 12759
    
    5
There is one bit of evidence conspicuously lacking:
Then it is getting down suddenly without any errors.


What does that mean? Please pick one of the following:
1. The entire server dies including everybody elses applications and the operator has to restart the whole thing.
2. Requests to your app get an error response - what is the error? Even the error number would be a help.
3. Requests to your app get no response and just sit there for hours.

How are requests to this web app being made? There are so many possiblities - AJAX style web page, plain HTML, custom client, etc etc.

Is this a Tomcat server? If so, can you use the Management app?
Which operating system? Java version?

Bill
steve souza
Ranch Hand

Joined: Jun 26, 2002
Posts: 860
<<Really i dont have any other information to share.>>
Is this happening at a certain time of day? Does it happen after certain pages are invoked? Can the server deliver a simple html, or gif file when it is "down" or can it respond to nothing? Is the process running or is it down? If the process is up how much memory is it consuming?

No one could solve your problem the way it is currently framed. Software can go wrong in too many ways to give an answer when no facts are provided. So, no matter what the problem is if you want to solve it you need to make your system give you more information about its state at any given time. This is called monitoring or profiling. Here is what I suggest.

Use the link to jamon below to help you do the following. None of the suggestions below require code changes and you can have them running in a few minutes.:
  • Track jdbc calls and sql. This will show if any exceptions are thrown if any queries are slow etc. (assuming your program issues sql. you didn't mention that)
  • Use the jamon tomcat valve to track all page performance stats, exceptions, bytes sent, status codes and more.
  • If you use log4j (and you should use log4j or some other logging package), then use the log4j appender to track stats about what you are writing to the log. If you aren't putting enough info in the log to diagnose the problem then put it in.
  • Use any other tools that can help you understand what is happening in your program. Right now your application is a windowless building, and you are standing on the outside trying to look in. You need to give it windows!


  • You must always write your application in a way that you make what happens inside it transparent as possible (give it windows). Logging, web page monitoring, jdbc/sql monitoring, exception monitoring, resource monitoring are all a part of this.
    [ June 14, 2008: Message edited by: steve souza ]
    Rajah Nagur
    Ranch Hand

    Joined: Nov 06, 2002
    Posts: 239
    Originally posted by Ramm Abhiram G:

    3) The testing environment and Production environment are exactly same.


    Are you sure of this? To the exact OS patches, Server patches?
    The problem could be anywhere - OS, app/web server settings, bad code, app configuration etc

    Are you facing a JVM hang kind of situation? Then you should consider taking a thread dump to analyze.
    Need more info such as :
    1. What kind of app are you talking abt? Database intensive or process intensive?
    2. What is the configuration? OS, Server, JVM, Database etc
    3. What is the GC setting for you app?
    4. Please share your app's profiling info


    You can't wake a person who is <b><i>pretending</i></b> to be asleep.<br />Like what <b>"it"</b> does not like - <i> Gurdjieff </i>
     
    wood burning stoves
     
    subject: Why web applications fail in Production ??
     
    Similar Threads
    Sun Java App server 8 in production environment
    Which Caching system is better, JCS / OSCACHE / SWARMCACHE ?
    Restart TOMCAT is required if server time is changed?
    forward to different context
    Regarding deployment on Websphere App server