aspose file tools*
The moose likes Websphere and the fly likes MDB Listeners down Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Products » Websphere
Bookmark "MDB Listeners down" Watch "MDB Listeners down" New topic
Author

MDB Listeners down

Priya Parthiban
Greenhorn

Joined: Apr 16, 2013
Posts: 2
Hello All,

We have two MDB listeners on our APP server. These are used by two applications to contact our applciation and get data.
These listeners are getting down frequently in recent times. This is causing severe business loss as ours is a mission critical application.
These listeners when get message to our application, we are running couple of DB queries to get data.
Some large transaction is holding the database tables as we see some unusual lock escalation happening on DB tables at the time listeners were down.

Please see below logs and please help me in finding root cause for the issue. Is it a DB related issue? or JDBC connections related issue??? Please help me out!!

Prod APP server logs:

[4/11/13 15:17:00:114 EDT] 00000042 FreePool E J2CA0045E: Connection not available while invoking method createOrWaitForConnection for resource jdbc/ABCD_EFGH.
[4/11/13 15:17:00:118 EDT] 00000042 JDBCException W org.hibernate.util.JDBCExceptionReporter logExceptions SQL Error: 0, SQLState: null
[4/11/13 15:17:00:119 EDT] 00000042 JDBCException E org.hibernate.util.JDBCExceptionReporter logExceptions Connection not available, Timed out waiting for 10005
[4/11/13 15:17:00:126 EDT] 00000042 SibMessage W [:] CWSJY0003W: JMSCC0109: A message driven bean threw a runtime exception '
Message : java.lang.StringIndexOutOfBoundsException: String index out of range: -814
Class : class java.lang.StringIndexOutOfBoundsException
Stack : java.lang.String.substring(String.java:1937)

[4/11/13 15:17:00:132 EDT] 00000042 FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /global/site/vendor/WAS/WebSphere7/AppServer/profiles/base/logs/ffdc/ABCD_EFGH.txt com.ibm.ejs.jms.listener.ServerSession.connectionConsumerOnMessage 450
[4/11/13 15:17:01:094 EDT] 00000043 FreePool E J2CA0045E: Connection not available while invoking method createOrWaitForConnection for resource jdbc/ABCD_EFGH.
[4/11/13 15:17:01:103 EDT] 00000042 FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /global/site/vendor/WAS/WebSphere7/AppServer/profiles/base/logs/ffdc/ABCD_EFGH1.txt com.ibm.ejs.container.MDBWrapper.onMessage 83
[4/11/13 15:17:01:106 EDT] 00000043 JDBCException W org.hibernate.util.JDBCExceptionReporter logExceptions SQL Error: 0, SQLState: null
[4/11/13 15:17:01:107 EDT] 00000043 JDBCException E org.hibernate.util.JDBCExceptionReporter logExceptions Connection not available, Timed out waiting for 10001
[4/11/13 15:17:01:111 EDT] 00000043 SibMessage W [:] CWSJY0003W: JMSCC0109: A message driven bean threw a runtime exception '
Message : java.lang.StringIndexOutOfBoundsException: String index out of range: -814
Class : class java.lang.StringIndexOutOfBoundsException
Stack : java.lang.String.substring(String.java:1937)

[4/11/13 15:17:01:111 EDT] 00000042 LocalExceptio E CNTR0020E: EJB threw an unexpected (non-declared) exception during invocation of method "onMessage" on bean "BeanId(ABCDEFGH#ABCDMDB.jar#ABCDEFHBean, null)". Exception data: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor2357.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.ibm.ejs.jms.listener.ServerSessionDispatcher.dispatch(ServerSessionDispatcher.java:44)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:100)
at com.ibm.ejs.container.MDBWrapper.onMessage(MDBWrapper.java:136)
at com.ibm.ejs.jms.listener.ServerSession.run(ServerSession.java:567)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1604)
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -814
at java.lang.String.substring(String.java:1937)

[4/11/13 15:17:01:674 EDT] 00000042 FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /global/site/vendor/WAS/WebSphere7/AppServer/profiles/base/logs/ffdc/ABCD.txt com.ibm.ejs.container.LocalExceptionMappingStrategy.setUncheckedException 178
[4/11/13 15:17:01:689 EDT] 00000042 LocalTranCoor E WLTC0017E: Resources rolled back due to setRollbackOnly() being called.
[4/11/13 15:17:01:689 EDT] 00000043 LocalTranCoor E WLTC0017E: Resources rolled back due to setRollbackOnly() being called.
[4/11/13 15:17:02:256 EDT] 00000042 FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /global/site/vendor/WAS/WebSphere7/AppServer/profiles/base/logs/ffdc/ABCD1.txt com.ibm.ejs.csi.TranStrategy.rollback 375
[4/11/13 15:17:02:295 EDT] 00000042 FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /global/site/vendor/WAS/WebSphere7/AppServer/profiles/base/logs/ffdc/ABCD2.txt com.ibm.ejs.jms.listener.ServerSession.run 283
[4/11/13 15:17:02:434 EDT] 00000042 ServerSession E com.ibm.ejs.jms.listener.ServerSession run WMSG0036E: Maximum message delivery retry count of 0 reached for MDB ABCDBean, JMSDestination jms/ABCDReq, MDBListener stopped
[4/11/13 15:17:02:434 EDT] 00000043 ServerSession E com.ibm.ejs.jms.listener.ServerSession run WMSG0036E: Maximum message delivery retry count of 0 reached for MDB DEFHBean, JMSDestination jms/DEFHReq, MDBListener stopped


[4/11/13 15:17:07:485 EDT] 00000079 MDBListenerIm I WMSG0043I: MDB Listener stopped for JMSDestination jms/ABCDReq
[4/11/13 15:17:07:519 EDT] 0000007b MDBListenerIm I WMSG0043I: MDB Listener stopped for JMSDestination jms/DEFHReq
[4/11/13 15:17:12:464 EDT] 00000073 FreePool E J2CA0045E: Connection not available while invoking method createOrWaitForConnection for resource jdbc/ABCD_EFGH.
[4/11/13 15:17:12:467 EDT] 00000073 JDBCException W org.hibernate.util.JDBCExceptionReporter logExceptions SQL Error: 0, SQLState: null
[4/11/13 15:17:12:468 EDT] 00000073 JDBCException E org.hibernate.util.JDBCExceptionReporter logExceptions Connection not available, Timed out waiting for 10001
[4/11/13 15:17:12:471 EDT] 00000073 servlet E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0068E: Uncaught exception created in one of the service methods of the servlet ABCDValidationService in application ABCDEFGH. Exception created : org.hibernate.exception.GenericJDBCException: Cannot open connection
at org.hibernate.exception.SQLStateConverter.handledNonSpecificException(SQLStateConverter.java:103)
at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:91)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:43)
at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:29)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:426)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:119)
at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:57)
at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1326)
at sun.reflect.GeneratedMethodAccessor300.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.hibernate.context.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:301)
at $Proxy32.beginTransaction(Unknown Source)
at com.util.InsertABCDTables.createRequestId(InsertABCDTables.java:113)
at com.handler.MainValidatonHandler.validateContract(MainValidatonHandler.java:85)
at com.service.ABCDValidationService.doPost(ABCDValidationService.java:69)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1657)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:939)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:502)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:179)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:864)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1583)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:452)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:511)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:305)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:83)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java:557)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java:607)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java:984)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java:1069)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1604)
Caused by: com.ibm.websphere.ce.cm.ConnectionWaitTimeoutException: Connection not available, Timed out waiting for 10001
at com.ibm.ws.rsadapter.AdapterUtil.toSQLException(AdapterUtil.java:1524)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:685)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:635)
at org.hibernate.connection.DatasourceConnectionProvider.getConnection(DatasourceConnectionProvider.java:69)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
... 31 more
Caused by: com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: Connection not available, Timed out waiting for 10001
at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1666)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2496)
at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:1063)
at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:700)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:668)
... 34 more

[4/11/13 15:17:28:349 EDT] 0000000c ThreadMonitor W WSVR0605W: Thread "WebContainer : 5" (00000047) has been active for 635968 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.FilterInputStream.read(FilterInputStream.java:116)
at com.wily.introscope.agent.probe.net.ManagedSocketInputStream.read(ManagedSocketInputStream.java:214)
at com.ibm.db2.jcc.t4.ab.b(ab.java:195)
at com.ibm.db2.jcc.t4.ab.c(ab.java:243)
at com.ibm.db2.jcc.t4.ab.c(ab.java:354)
at com.ibm.db2.jcc.t4.ab.v(ab.java:1139)
at com.ibm.db2.jcc.t4.db.a(db.java:37)
at com.ibm.db2.jcc.t4.t.a(t.java:32)
at com.ibm.db2.jcc.t4.sb.h(sb.java:141)
at com.ibm.db2.jcc.b.fm.bb(fm.java:1923)
at com.ibm.db2.jcc.b.gm.dc(gm.java:2551)
at com.ibm.db2.jcc.b.gm.e(gm.java:3252)
at com.ibm.db2.jcc.b.gm.Rb(gm.java:612)
at com.ibm.db2.jcc.b.gm.executeUpdate(gm.java:595)
at com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.pmiExecuteUpdate(WSJdbcPreparedStatement.java:1175)
at com.ibm.ws.rsadapter.jdbc.WSJdbcPreparedStatement.executeUpdate(WSJdbcPreparedStatement.java:816)
at org.hibernate.jdbc.NonBatchingBatcher.addToBatch(NonBatchingBatcher.java:23)
at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2247)
at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2660)
at org.hibernate.action.EntityInsertAction.execute(EntityInsertAction.java:56)
at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:250)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:234)
at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:141)
at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298)
at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)
at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1000)
at org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:338)
at org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:106)
at com.util.InsertABCDTables.createRequestId(InsertABCDTables.java:118)
at com.handler.MainValidatonHandler.validateContract(MainValidatonHandler.java:85)
at com.validation.service.ABCDValidationService.doPost(ABCDValidationService.java:69)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:738)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:831)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1657)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:939)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:502)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:179)
at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:91)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:864)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1583)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:452)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:511)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:305)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:276)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java:557)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java:607)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java:984)
at com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java:1069)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1604)
Anton Novopashin
Greenhorn

Joined: Feb 25, 2008
Posts: 19
Hi, Indu
There are 2 issue in your description.

First of all check datasource connection pool, it is not enough connection:
Connection not available while invoking method createOrWaitForConnection for resource jdbc/ABCD_EFGH.
[4/11/13 15:17:12:467 EDT] 00000073 JDBCException W org.hibernate.util.JDBCExceptionReporter logExceptions SQL Error: 0, SQLState: null
[4/11/13 15:17:12:468 EDT] 00000073 JDBCException E org.hibernate.util.JDBCExceptionReporter logExceptions Connection not available, Timed out waiting for 10001


All your threads waited free connection form pool. Also i woukd recommend to you to read Hibernate + WebSphere

Second problem is application design. As far as i understand you get data to MDB and try to process it immideatly. This is not good for your application.
You can you use delayed processing with claim check pattern to overcome your issues.
Just process data in EJB with new transaction, do not join MDB.




OCJP (1Z0-851).
Integration Developer (000-607).
WAS Admin (000-377).
Priya Parthiban
Greenhorn

Joined: Apr 16, 2013
Posts: 2
Indu Priyanka wrote:Hello All,

We have two MDB listeners on our APP server. These are used by two applications to contact our applciation and get data.
These listeners are getting down frequently in recent times. This is causing severe business loss as ours is a mission critical application.
These listeners when get message to our application, we are running couple of DB queries to get data.
Some large transaction is holding the database tables as we see some unusual lock escalation happening on DB tables at the time listeners were down.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: MDB Listeners down
 
Similar Threads
Hibernate configuration issue while getting the Oracle Database 11G database connection
Websphere Application Server 7 ... EJB 2.0 Error :: PMGR1012E
Websphere App Server 7 and EJB 2.0 problem
Websphere exception while accessing Weblogic JMS queue over Generic JMS inbound export
Login page is not opening