RJVM has already been shutdown error while looking upto Foreign Server

[Priyam Srivastava]

Hi,

I am trying to set up 2 new servers in a clustered environment. In this I have configured a Foreign Server which is looking up to queues in different server.

The structure is like this: 1 Foreign Server with 2 Queues and 1 Connection factory with exactly same name as they present on the host server.
While starting the server, I am getting an exception:

<Nov 5, 2012 7:39:58 PM EET> <Warning> <EJB> <BEA-010061> <The Message-Driven EJB: ClonedQueueAnonymousWaiter is unable to connect to the JMS destination: MDB_QUEUE_RESPONSE. The Error was:
javax.naming.CommunicationException [Root exception is java.rmi.UnmarshalException: RJVM has already been shutdown; nested exception is:
java.io.IOException: RJVM has already been shutdown]
at weblogic.jndi.internal.ExceptionTranslator.toNamingException(ExceptionTranslator.java:74)
at weblogic.jndi.internal.WLContextImpl.translateException(WLContextImpl.java:470)
at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:426)
at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:411)
at javax.naming.InitialContext.lookup(InitialContext.java:392)
at weblogic.deployment.jms.ForeignOpaqueReference.getReferent(ForeignOpaqueReference.java:221)
at weblogic.jndi.internal.WLNamingManager.getObjectInstance(WLNamingManager.java:96)
at weblogic.jndi.internal.ServerNamingNode.resolveObject(ServerNamingNode.java:377)
at weblogic.jndi.internal.BasicNamingNode.resolveObject(BasicNamingNode.java:856)
at weblogic.jndi.internal.BasicNamingNode.lookup(BasicNamingNode.java:209)
at weblogic.jndi.internal.WLEventContextImpl.lookup(WLEventContextImpl.java:254)
at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:411)
at javax.naming.InitialContext.lookup(InitialContext.java:392)
at weblogic.jms.common.CDS$2.run(CDS.java:486)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.jms.common.CrossDomainSecurityManager.runAs(CrossDomainSecurityManager.java:131)
at weblogic.jms.common.CDS.lookupDestination(CDS.java:480)
at weblogic.jms.common.CDS.lookupDDAndCalloutListener(CDS.java:345)
at weblogic.jms.common.CDS.access$100(CDS.java:41)
at weblogic.jms.common.CDS$DDListenerRegistrationTimerListener.timerExpired(CDS.java:193)
at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)
Caused by: java.rmi.UnmarshalException: RJVM has already been shutdown; nested exception is:
java.io.IOException: RJVM has already been shutdown
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:360)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:259)
at weblogic.jndi.internal.ServerNamingNode_1034_WLStub.lookup(Unknown Source)
at weblogic.jndi.internal.WLContextImpl.lookup(WLContextImpl.java:423)
... 21 more
Caused by: java.io.IOException: RJVM has already been shutdown
at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:234)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:348)
... 24 more
Caused by: java.io.IOException: RJVM has already been shutdown
at weblogic.rjvm.RJVMImpl.addPendingResponse(RJVMImpl.java:513)
at weblogic.rjvm.RJVMImpl.send(RJVMImpl.java:900)
at weblogic.rjvm.MsgAbbrevOutputStream.flushAndSend(MsgAbbrevOutputStream.java:394)
at weblogic.rjvm.MsgAbbrevOutputStream.sendRecv(MsgAbbrevOutputStream.java:404)
at weblogic.rjvm.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:109)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:345)
... 24 more

Weblogic is installed in Solaris machine and welogic's version is 10.3.4.0 with Java version 1.6.0 update 33. I have checked the firewall, /etc/hosts but there are no issues with that.

The JNDI Connection URL is configure like: t3://xx.xxx.x.xx,xx.xxx.x.xx:7001/
I have tried giving like this also: t3://xx.xxx.x.xx:7001,xx.xxx.x.xx:7001/ or t3://hostname1,hostname2:7001/ or t3://hostname1:7001,hostname2:7001/
but nothing worked.

The foreign servers are accessible from my Application Server when I do the telnet on the port. There is no cross domain security required. Funny thing is I have a same set-up in two other application server (in cluster again) and there on configuring the Foreign server to same IPs, I am not facing any issues at all.

Can anyone please help me with this. Searched all over the place but have no clue regarding this. Many forums mentioned about firewall and etc/hosts but that is also not case it seems.

Regards,
Priyam>