aspose file tools*
The moose likes Tomcat and the fly likes Memory Leak in tomcat due threads are not kill while stopping tomcat Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of The Java EE 7 Tutorial Volume 1 or Volume 2 this week in the Java EE forum
or jQuery UI in Action in the JavaScript forum!
JavaRanch » Java Forums » Products » Tomcat
Bookmark "Memory Leak in tomcat due threads are not kill while stopping tomcat" Watch "Memory Leak in tomcat due threads are not kill while stopping tomcat" New topic
Author

Memory Leak in tomcat due threads are not kill while stopping tomcat

nitin yeola
Greenhorn

Joined: Mar 27, 2012
Posts: 18
Hello

In our production environment we are facing memory leak issue for tomcat.

When we shut down tomcat we getting following error in catalina log

created a ThreadLocal with key of type [null] (value [null]) and a value of type [org.apache.lucene.index.SegmentTermEnum] (value [org.apache.lucene.index.SegmentTermEnum@19a419a4]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.


this issue slowing our production environment.

Can you please help in this to resolve.
How to kill these threads before shutdown tomcat?
And
Also there are some threads in waiting states how to kill that threads?

Thank you in advanced.
Ulf Dittmer
Marshal

Joined: Mar 22, 2005
Posts: 41574
    
  54
How can it create a memory leak if Tomcat is being shut down? The only way it could create leak is if the web app was stopped and restarted (which is not recommended for production systems anyway).

I also don't understand how this could "slow down" the production environment, since it happens at shutdown time, not at runtime.


Ping & DNS - my free Android networking tools app
nitin yeola
Greenhorn

Joined: Mar 27, 2012
Posts: 18
Thanks Ulf

we are getting following log error when we shutdown tomcat for restart

Jul 18, 2013 6:21:21 AM org.quartz.core.QuartzScheduler shutdown
INFO: Scheduler Scheduler_$_NON_CLUSTERED shutdown complete.
Jul 18, 2013 6:21:26 AM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/app] registered the JDBC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Jul 18, 2013 6:21:26 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/app] appears to have started a thread named [Connection Keep-Alive nullBroker #1@xxx:6849] but has failed to stop it. This is very likely to create a memory leak.
Jul 18, 2013 6:21:26 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/app] appears to have started a thread named [LinkReader Broker #1@xxx:6849] but has failed to stop it. This is very likely to create a memory leak.

also we observed in threaddump these threads are in waiting state.

our production environment slowing down day by day we need to restart in every week.
Hope you got my point.
Ulf Dittmer
Marshal

Joined: Mar 22, 2005
Posts: 41574
    
  54
Again: those warnings have precisely zero impact. There is no memory leak until you shut down Tomcat. And once you have shut down Tomcat, there can't possibly be a memory leak.

It may well be that there is a memory leak somewhere in your code, or that it slows down for any number of possible reasons - but that has nothing to do with these messages, which you can safely ignore.
nitin yeola
Greenhorn

Joined: Mar 27, 2012
Posts: 18
But when I shut down tomcat and check the threaddump using jstack it is showing threads mentioned earlier in waiting state.

I think these thread should not appear in threaddump.

Also can you help me what are ways to kill threads in web-application which are causing memory leaks?

thanks in advance..
Tim Holloway
Saloon Keeper

Joined: Jun 25, 2001
Posts: 16019
    
  20

Tomcat does not have a history of memory or thread leakage. All known cases of such leakages have been resolved to webapp bugs, not Tomcat bugs.

Given the limited amount of data, I'd guess that one of the components of a webapp's text search subsystem (Lucene) isn't being terminated when Tomcat shuts down and invokes the servlet destroy methods for the deployed webapps.


Customer surveys are for companies who didn't pay proper attention to begin with.
nitin yeola
Greenhorn

Joined: Mar 27, 2012
Posts: 18
Thanks..

In most of the places its mentioned that tomcat 6.0.24 have memory leakage bugs.
So we came to know from this information it may happening because of tomcat(check it http://wiki.apache.org/tomcat/MemoryLeakProtection)

is there any way to stop these long lasting threads of webaaps as I am new in multithreaded environment please help me to solve this?

If required any details let me know.
Ulf Dittmer
Marshal

Joined: Mar 22, 2005
Posts: 41574
    
  54
nitin yeola wrote:But when I shut down tomcat and check the threaddump

I think somewhere here is the source of the misunderstanding. That thread may well be there during shutdown, but it's gone (rather obviously) after shutdown. So I'm unclear what exactly it is that you're worried about.

You may also wish to check the APIs that create those threads (whichever libraries those may be) if they have cleanup/destroy/shutdown methods you could or should be calling in a context listener. That may speed up the process of shutting down.
Ulf Dittmer
Marshal

Joined: Mar 22, 2005
Posts: 41574
    
  54
nitin yeola wrote:In most of the places its mentioned that tomcat 6.0.24 have memory leakage bugs.
So we came to know from this information it may happening because of tomcat(check it http://wiki.apache.org/tomcat/MemoryLeakProtection)

Those are not Tomcat memory leaks. As the page states specifically : ...against memory leaks when stopping or redeploying applications. If an app doesn't clear up after itself, that's not something that's under Tomcat's control (but it tries to do the cleanup in lieu of the app).
Tim Holloway
Saloon Keeper

Joined: Jun 25, 2001
Posts: 16019
    
  20

nitin yeola wrote:Thanks..

In most of the places its mentioned that tomcat 6.0.24 have memory leakage bugs.
So we came to know from this information it may happening because of tomcat(check it http://wiki.apache.org/tomcat/MemoryLeakProtection)

is there any way to stop these long lasting threads of webaaps as I am new in multithreaded environment please help me to solve this?

If required any details let me know.


That page isn't about leaks in Tomcat. It's a tool for finding leaks in webapps.

I suspect that the "In most of the places its mentioned that tomcat 6.0.24 have memory leakage bugs" actually is referencing the infamous "PermGen" space issue. Which isn't a Tomcat memory leak, it's a case where when a webapp is redeployed memory from the PermGen pool isn't freed due to classloader ambiguities. You don't have that. You'd know it if you did.
nitin yeola
Greenhorn

Joined: Mar 27, 2012
Posts: 18
Thanks Ulf & Tim

Can you please suggest some way to debug this and find out where is the issue?
Ulf Dittmer
Marshal

Joined: Mar 22, 2005
Posts: 41574
    
  54
I still don't understand why this matters. If the app slows down over a period of time, then that has nothing to do with these messages. Which problem, exactly, is it you're trying to solve?
nitin yeola
Greenhorn

Joined: Mar 27, 2012
Posts: 18
I want to know which threads are creating issue (i.e. slowing down) application.

Means how to find what are the problematic area of application causing issue?

Hope now you understand this.
nitin yeola
Greenhorn

Joined: Mar 27, 2012
Posts: 18
Hello,

In our application we sending out posted data to other application using broker url via jms this integration of code causing following memory leaks.

Jul 18, 2013 6:21:26 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/app] appears to have started a thread named [Connection Keep-Alive nullBroker #1@xxx:6849] but has failed to stop it. This is very likely to create a memory leak.
Jul 18, 2013 6:21:26 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/app] appears to have started a thread named [LinkReader Broker #1@xxx:6849] but has failed to stop it. This is very likely to create a memory leak.


For this we included jar and used methods from this library to send data.

These methods creating threads for each posting which are not stopping causing memory leak.

Can you suggest any way how to stop these threads in our environment because we are not able to find when they to have stop and which are they?
Ulf Dittmer
Marshal

Joined: Mar 22, 2005
Posts: 41574
    
  54
this integration of code causing following memory leaks.

OK, one last time: those messages do not indicate a memory leak. Whatever they appear to say is moot with the shutdown of the JVM.

These methods creating threads for each posting which are not stopping causing memory leak.

Now it gets interesting. How do you know this? Post the evidence you have that each access is causing a thread to be started but never to be finished.
William Brogden
Author and all-around good cowpoke
Rancher

Joined: Mar 22, 2000
Posts: 12769
    
    5
SEVERE: The web application [/app] appears to have started a thread named [Connection Keep-Alive nullBroker #1@xxx:6849] but has failed to stop it. This is very likely to create a memory leak.


Does your application really start new Threads? If so that is not a Tomcat problem but an application problem.

Threads take up memory, you can't just keep creating them endlessly if you don't clean them up by exiting the run() method.

I would use the Tomcat manager application to look at the state of the request Threads and memory.

Bill
nitin yeola
Greenhorn

Joined: Mar 27, 2012
Posts: 18
SEVERE: The web application [/app] appears to have started a thread named [Connection Keep-Alive nullBroker #1@xxx:6849] but has failed to stop it. This is very likely to create a memory leak.


This log that indicating failed to stop it(thread), should I ignore it?

You mean to say that these threads get stopped when tomcat shutdown and they will not present in memory, Right?

And

Evidence is : For each post one new thread(above) get started and not stopped until tomcat shutdown.
Consider in one day 100 posts are created , when we shutdown tomcat we get 100 warnings messages mentioned as above.

If we never shutdown the tomcat, application takes more cpu usage and memory usage which leads to slowing down the application.
Day by day memory usage gets increased never released back.

Note: We are using jforum(Version 2.1.8) by integrating external applications to pass the post info to other applications. Is there any way in jforum to stop external threads ?

Ulf Dittmer
Marshal

Joined: Mar 22, 2005
Posts: 41574
    
  54
nitin yeola wrote:This log that indicating failed to stop it(thread), should I ignore it?

Yes.

You mean to say that these threads get stopped when tomcat shutdown and they will not present in memory,

Yes.

Evidence is : For each post one new thread(above) get started and not stopped until tomcat shutdown. Consider in one day 100 posts are created , when we shutdown tomcat we get 100 warnings messages mentioned as above.

Does this mean that you have seen those 100 messages? Or merely that you think that's what might happen? If you see just one of these, just forget about them. If you see 100 after you have sent 100 messages, then you need to investigate further (in this case I'd suspect that the API that is sending these messages -whatever that means- is not being used correctly).

We are using jforum(Version 2.1.8) by integrating external applications to pass the post info to other applications. Is there any way in jforum to stop external threads ?

I know JForum very well, and it does not do anything like what you describe (like "send messages"). So wherever this originates, it must be from some addition to it that you made, so I couldn't give an opinion on that, except taht JForum by itself obviously would not know about any of your additions, unless you built in some management facilities for these additional features.
nitin yeola
Greenhorn

Joined: Mar 27, 2012
Posts: 18
Thanks for your valuable feedback.

Does this mean that you have seen those 100 messages? Or merely that you think that's what might happen? If you see just one of these, just forget about them. If you see 100 after you have sent 100 messages, then you need to investigate further (in this case I'd suspect that the API that is sending these messages -whatever that means- is not being used correctly).


Yes, we have seen 100 warning messages for 100 posts.
We will investigate further related to API also.

Is it possible to stop external API threads in jforum after use?
Ulf Dittmer
Marshal

Joined: Mar 22, 2005
Posts: 41574
    
  54
JForum knows nothing about whatever code you have added to it. You'll have to consult the documentation of whatever API you're using.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Memory Leak in tomcat due threads are not kill while stopping tomcat