Issue while using the mysql replication driver in tomcat

[Elan Ram]

Hi,

We are using the mysql replication connector (version 5.12) with Tomcat 5.5 & Tomcat 6 to connect the master, slave databases. But the servers are hanging periodically (almost once in a day) due to the following error (excerpt from thread dump):

    
Full thread dump OpenJDK 64-Bit Server VM (1.6.0-b09 mixed mode):

..............................
..............................

"http-8080-Processor74" daemon prio=10 tid=0x0000000013463800 nid=0x737e waiting for monitor entry [0x0000000046696000..0x0000000046697b10]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.commons.pool.impl.GenericObjectPool.getNumIdle(GenericObjectPool.java:911)
	- waiting to lock <0x00002aab05f3c5f8> (a org.apache.commons.dbcp.AbandonedObjectPool)
	at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:78)
	at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
	at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
	at sun.reflect.GeneratedMethodAccessor185.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:304)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
	at $Proxy0.getConnection(Unknown Source)
	at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:200)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:350)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:262)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:101)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at $Proxy9.findPersonnelIdByImei(Unknown Source)
	at com.ncs.mobileconnect.filter.MobileClientFilter.trackRequest(MobileClientFilter.java:153)
	at com.ncs.mobileconnect.filter.MobileClientFilter.doFilter(MobileClientFilter.java:117)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:183)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:138)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
	at org.apache.catalina.valves.FastCommonAccessLogValve.invoke(FastCommonAccessLogValve.java:482)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
	at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
	at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
	at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
	at java.lang.Thread.run(Thread.java:636)

..........................



"http-8080-Processor30" daemon prio=10 tid=0x00000000136cc000 nid=0x72db in Object.wait() [0x0000000041136000..0x0000000041136d10]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aaafed55d20> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:502)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:657)
	- locked <0x00002aaafed55d20> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:636)

"http-8080-Processor29" daemon prio=10 tid=0x00000000136cb400 nid=0x72da runnable [0x0000000041034000..0x0000000041035d90]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:146)
	at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:114)
	at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:161)
	at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:189)
	- locked <0x00002aaafeced6a0> (a com.mysql.jdbc.util.ReadAheadInputStream)
	at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2500)
	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2953)
	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2942)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3485)
	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1960)
	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2114)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2690)
	- locked <0x00002aaafece8880> (a java.lang.Object)
	at com.mysql.jdbc.ConnectionImpl.rollbackNoChecks(ConnectionImpl.java:4925)
	at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4810)
	- locked <0x00002aaafece8880> (a java.lang.Object)
	at com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4418)
	at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1555)
	- locked <0x00002aab00ad75d8> (a com.mysql.jdbc.JDBC4Connection)
	at com.mysql.jdbc.LoadBalancingConnectionProxy.closeAllConnections(LoadBalancingConnectionProxy.java:310)
	- locked <0x00002aab00ad7570> (a com.mysql.jdbc.LoadBalancingConnectionProxy)
	at com.mysql.jdbc.LoadBalancingConnectionProxy.invoke(LoadBalancingConnectionProxy.java:346)
	at $Proxy63.close(Unknown Source)
	at com.mysql.jdbc.ReplicationConnection.close(ReplicationConnection.java:131)
	- locked <0x00002aab00ad6d80> (a com.mysql.jdbc.ReplicationConnection)
	at org.apache.commons.dbcp.DelegatingConnection.close(DelegatingConnection.java:214)
	at org.apache.commons.dbcp.PoolableConnection.reallyClose(PoolableConnection.java:102)
	at org.apache.commons.dbcp.PoolableConnectionFactory.destroyObject(PoolableConnectionFactory.java:305)
	at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:865)
	- locked <0x00002aab05f3c5f8> (a org.apache.commons.dbcp.AbandonedObjectPool)
	at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:84)
	at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
	at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
	at sun.reflect.GeneratedMethodAccessor185.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:304)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
	at $Proxy0.getConnection(Unknown Source)
	at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:200)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:350)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:262)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:101)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at $Proxy9.findPersonnelIdByImei(Unknown Source)
	at com.ncs.mobileconnect.filter.MobileClientFilter.trackRequest(MobileClientFilter.java:153)
	at com.ncs.mobileconnect.filter.MobileClientFilter.doFilter(MobileClientFilter.java:117)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:183)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:138)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
	at org.apache.catalina.valves.FastCommonAccessLogValve.invoke(FastCommonAccessLogValve.java:482)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
	at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
	at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
	at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
	at java.lang.Thread.run(Thread.java:636)

"http-8080-Processor28" daemon prio=10 tid=0x000000001348f000 nid=0x72d9 in Object.wait() [0x0000000040f34000..0x0000000040f34a10]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aaafe954f98> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:502)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:657)
	- locked <0x00002aaafe954f98> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:636)

"http-8080-Processor27" daemon prio=10 tid=0x000000001348e400 nid=0x72d8 in Object.wait() [0x0000000040e33000..0x0000000040e33a90]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aab00156b88> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:502)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:657)
	- locked <0x00002aab00156b88> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:636)

"http-8080-Processor26" daemon prio=10 tid=0x000000001348dc00 nid=0x72d7 waiting for monitor entry [0x0000000040d31000..0x0000000040d32b10]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.commons.pool.impl.GenericObjectPool.getNumIdle(GenericObjectPool.java:911)
	- waiting to lock <0x00002aab05f3c5f8> (a org.apache.commons.dbcp.AbandonedObjectPool)
	at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:78)
	at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
	at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
	at sun.reflect.GeneratedMethodAccessor185.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:304)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
	at $Proxy0.getConnection(Unknown Source)
	at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:200)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:350)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:262)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:101)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at $Proxy9.findPersonnelIdByImei(Unknown Source)
	at com.ncs.mobileconnect.filter.MobileClientFilter.trackRequest(MobileClientFilter.java:153)
	at com.ncs.mobileconnect.filter.MobileClientFilter.doFilter(MobileClientFilter.java:117)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:183)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:138)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
	at org.apache.catalina.valves.FastCommonAccessLogValve.invoke(FastCommonAccessLogValve.java:482)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
	at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
	at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
	at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
	at java.lang.Thread.run(Thread.java:636)

"Timer-0" daemon prio=10 tid=0x0000000014739000 nid=0x7122 in Object.wait() [0x0000000043e6f000..0x0000000043e6fb90]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aab05e5eb88> (a java.util.TaskQueue)
	at java.lang.Object.wait(Object.java:502)
	at java.util.TimerThread.mainLoop(Timer.java:505)
	- locked <0x00002aab05e5eb88> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:484)

"http-8080-Monitor" prio=10 tid=0x0000000013675c00 nid=0x7118 in Object.wait() [0x0000000043d6e000..0x0000000043d6ec10]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aab05e13e38> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
	at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:560)
	- locked <0x00002aab05e13e38> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
	at java.lang.Thread.run(Thread.java:636)

"http-8080-Processor25" daemon prio=10 tid=0x00000000136f0400 nid=0x7117 in Object.wait() [0x0000000043c6d000..0x0000000043c6dc90]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aab05e72800> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:502)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:657)
	- locked <0x00002aab05e72800> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:636)

"http-8080-Processor24" daemon prio=10 tid=0x0000000014447c00 nid=0x7116 in Object.wait() [0x0000000043b6c000..0x0000000043b6cd10]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aab05ed3318> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:502)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:657)
	- locked <0x00002aab05ed3318> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:636)

"http-8080-Processor23" daemon prio=10 tid=0x0000000013c2c400 nid=0x7115 in Object.wait() [0x0000000043a6b000..0x0000000043a6bd90]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aab05e11d18> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:502)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:657)
	- locked <0x00002aab05e11d18> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:636)

"http-8080-Processor22" daemon prio=10 tid=0x0000000013c2b000 nid=0x7114 in Object.wait() [0x000000004396a000..0x000000004396aa10]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aab05e71d10> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:502)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:657)
	- locked <0x00002aab05e71d10> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:636)

"http-8080-Processor21" daemon prio=10 tid=0x000000001389f000 nid=0x7113 in Object.wait() [0x0000000043869000..0x0000000043869a90]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aab05ec8288> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:502)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:657)
	- locked <0x00002aab05ec8288> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:636)

"http-8080-Processor20" daemon prio=10 tid=0x000000001364c800 nid=0x7112 in Object.wait() [0x0000000043768000..0x0000000043768b10]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aab05ebf5e8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:502)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:657)
	- locked <0x00002aab05ebf5e8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:636)

"http-8080-Processor19" daemon prio=10 tid=0x0000000013dca800 nid=0x7111 in Object.wait() [0x0000000043667000..0x0000000043667b90]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aab05e72918> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:502)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:657)
	- locked <0x00002aab05e72918> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:636)

"http-8080-Processor18" daemon prio=10 tid=0x0000000013736000 nid=0x7110 in Object.wait() [0x0000000043566000..0x0000000043566c10]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aab05e71d48> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:502)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:657)
	- locked <0x00002aab05e71d48> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:636)

"http-8080-Processor17" daemon prio=10 tid=0x0000000013fbdc00 nid=0x710f in Object.wait() [0x0000000043465000..0x0000000043465c90]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aab05e72838> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:502)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:657)
	- locked <0x00002aab05e72838> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:636)

"http-8080-Processor16" daemon prio=10 tid=0x0000000014848800 nid=0x710e in Object.wait() [0x0000000043364000..0x0000000043364d10]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aab05e72870> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:502)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:657)
	- locked <0x00002aab05e72870> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:636)

"http-8080-Processor15" daemon prio=10 tid=0x0000000013728400 nid=0x710d in Object.wait() [0x0000000043263000..0x0000000043263d90]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aab05e728a8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:502)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:657)
	- locked <0x00002aab05e728a8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:636)

"http-8080-Processor14" daemon prio=10 tid=0x000000001395fc00 nid=0x710c in Object.wait() [0x0000000043162000..0x0000000043162a10]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aab05e728e0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:502)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:657)
	- locked <0x00002aab05e728e0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:636)

"http-8080-Processor13" daemon prio=10 tid=0x0000000013e79400 nid=0x710b waiting for monitor entry [0x0000000043060000..0x0000000043061a90]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.commons.pool.impl.GenericObjectPool.getNumIdle(GenericObjectPool.java:911)
	- waiting to lock <0x00002aab05f3c5f8> (a org.apache.commons.dbcp.AbandonedObjectPool)
	at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:78)
	at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
	at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
	at sun.reflect.GeneratedMethodAccessor185.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:304)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
	at $Proxy0.getConnection(Unknown Source)
	at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:200)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:350)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:262)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:101)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at $Proxy9.findPersonnelIdByImei(Unknown Source)
	at com.ncs.mobileconnect.filter.MobileClientFilter.trackRequest(MobileClientFilter.java:153)
	at com.ncs.mobileconnect.filter.MobileClientFilter.doFilter(MobileClientFilter.java:117)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:183)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:138)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
	at org.apache.catalina.valves.FastCommonAccessLogValve.invoke(FastCommonAccessLogValve.java:482)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
	at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
	at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
	at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
	at java.lang.Thread.run(Thread.java:636)

....................



"http-8080-Processor3" daemon prio=10 tid=0x0000000015e76400 nid=0x7101 in Object.wait() [0x0000000041715000..0x0000000041715b90]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aab05ea8dd8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:502)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:657)
	- locked <0x00002aab05ea8dd8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:636)

"http-8080-Processor2" daemon prio=10 tid=0x0000000013634400 nid=0x7100 runnable [0x0000000040c31000..0x0000000040c31c10]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:358)
	at java.net.ServerSocket.implAccept(ServerSocket.java:470)
	at java.net.ServerSocket.accept(ServerSocket.java:438)
	at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)
	at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:408)
	at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:71)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
	at java.lang.Thread.run(Thread.java:636)

"http-8080-Processor1" daemon prio=10 tid=0x0000000013559800 nid=0x70ff in Object.wait() [0x0000000040b30000..0x0000000040b30c90]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aab0639ec88> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:502)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:657)
	- locked <0x00002aab0639ec88> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:636)

"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10 tid=0x000000001374fc00 nid=0x70fe waiting on condition [0x0000000040a2f000..0x0000000040a2fd10]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1548)
	at java.lang.Thread.run(Thread.java:636)

"Low Memory Detector" daemon prio=10 tid=0x0000000013239000 nid=0x70f9 runnable [0x0000000000000000..0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x0000000013237400 nid=0x70f8 waiting on condition [0x0000000000000000..0x000000004072b440]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x0000000013233c00 nid=0x70f7 waiting on condition [0x0000000000000000..0x000000004062a4c0]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x0000000013232000 nid=0x70f6 runnable [0x0000000000000000..0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x0000000013230c00 nid=0x70f5 waiting on condition [0x0000000000000000..0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x0000000013202c00 nid=0x70f4 in Object.wait() [0x0000000041f1c000..0x0000000041f1cd10]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aab05ec7df0> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133)
	- locked <0x00002aab05ec7df0> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)

"Reference Handler" daemon prio=10 tid=0x0000000013201800 nid=0x70f3 in Object.wait() [0x0000000041e1b000..0x0000000041e1bd90]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00002aab05e2a380> (a java.lang.ref.Reference$Lock)
	at java.lang.Object.wait(Object.java:502)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
	- locked <0x00002aab05e2a380> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x00000000130b5400 nid=0x70e6 runnable [0x000000004052a000..0x000000004052aef0]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:358)
	at java.net.ServerSocket.implAccept(ServerSocket.java:470)
	at java.net.ServerSocket.accept(ServerSocket.java:438)
	at org.apache.catalina.core.StandardServer.await(StandardServer.java:379)
	at org.apache.catalina.startup.Catalina.await(Catalina.java:616)
	at org.apache.catalina.startup.Catalina.start(Catalina.java:576)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:295)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:433)

"VM Thread" prio=10 tid=0x00000000131fc400 nid=0x70f2 runnable 

"Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00000000130bb400 nid=0x70e7 runnable 

"Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00000000130bc800 nid=0x70e8 runnable 

"Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00000000130bdc00 nid=0x70e9 runnable 

"Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00000000130bf000 nid=0x70ea runnable 

"Gang worker#4 (Parallel GC Threads)" prio=10 tid=0x00000000130c0400 nid=0x70eb runnable 

"Gang worker#5 (Parallel GC Threads)" prio=10 tid=0x00000000130c1800 nid=0x70ec runnable 

"Gang worker#6 (Parallel GC Threads)" prio=10 tid=0x00000000130c2c00 nid=0x70ed runnable 

"Gang worker#7 (Parallel GC Threads)" prio=10 tid=0x00000000130c4000 nid=0x70ee runnable 

"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x000000001319c800 nid=0x70f1 runnable 
"Gang worker#0 (Parallel CMS Threads)" prio=10 tid=0x0000000013199c00 nid=0x70ef runnable 

"Gang worker#1 (Parallel CMS Threads)" prio=10 tid=0x000000001319b000 nid=0x70f0 runnable 

"VM Periodic Task Thread" prio=10 tid=0x000000001323b000 nid=0x70fa waiting on condition 

JNI global references: 1169

Heap
 par new generation   total 1314176K, used 546653K [0x00002aaaae5f0000, 0x00002aab05df0000, 0x00002aab05df0000)
  eden space 1194752K,  41% used [0x00002aaaae5f0000, 0x00002aaacc75d130, 0x00002aaaf74b0000)
  from space 119424K,  44% used [0x00002aaafe950000, 0x00002aab01dba5c8, 0x00002aab05df0000)
  to   space 119424K,   0% used [0x00002aaaf74b0000, 0x00002aaaf74b0000, 0x00002aaafe950000)
 concurrent mark-sweep generation total 2760704K, used 42286K [0x00002aab05df0000, 0x00002aabae5f0000, 0x00002aabae5f0000)
 concurrent-mark-sweep perm gen total 68096K, used 67855K [0x00002aabae5f0000, 0x00002aabb2870000, 0x00002aabce5f0000)

Any help/pointer is highly appreciated.