JNDI Lookup & Caching issue on multiple BEA instances

[Joe Harry]

Guys,

I have a scenario where in upon application start up (of my client) I do a look up on different BEA instances (physical servers). I have a Value Object that stores the cached remote reference. Thus I have a class that holds the List of these two Value Objects. Now my clients make a call to a method in the remote facade using the cached remote reference in the Value Object. Following that I undeploy the EJB's in one of the BEA server and try to call a method with the cached remote reference. As per the EJB specification, expected is a NoSuchEJBException, but I'm getting some strange exception as below:

16:02:49,119  INFO ServiceInstance:20 - ipAddress = t3://localhost:7001 jndiName= CustomerFacadeRemote/remote#com.user.facade.CustomerFacadeRemote 
userName = weblogic password = weblogic
16:02:49,119  INFO ServiceInstance:20 - ipAddress = t3://ANOTHERCOMPUTER:7003 jndiName= CustomerFacadeRemote/remote#com.user.facade.CustomerFacadeRemote 
userName = weblogic password = weblogic
16:02:50,213  INFO ServiceInstance:22 - run: JNDI lookup OKAY!
16:02:50,213  INFO ServiceInstance:22 - run: JNDI lookup OKAY!
16:02:50,229  INFO ServiceInstance:240 - setServiceInstanceAsActive(): Setting the lookedup service reference
16:02:50,229  INFO ServiceInstance:240 - setServiceInstanceAsActive(): Setting the lookedup service reference
16:02:50,276  INFO ServiceInstance:245 - setServiceInstanceAsActive(): The ServiceInstance is com.dispatcher.ServiceInstance@9abc69[
  rwl=java.util.concurrent.locks.ReentrantReadWriteLock@5c98f3[Write locks = 1, Read locks = 0]
  id=1
  ipAddress=t3://localhost:7001
  jndiName=CustomerFacadeRemote/remote#com.user.facade.CustomerFacadeRemote
  state=ACTIVE
  timeToRefresh=300000
  timeOfLastUnsuccessfulRefresh=<null>
  userName=weblogic
  password=weblogic
  remoteReference=weblogic.rmi.internal.BasicRemoteRef - hostID: '-7327730829344636601S:127.0.0.1:[7001,7001,-1,-1,-1,-1,-1]:APP_Domain:AdminServer', oid: '459', channel: 'null'
]
16:02:50,276  INFO ServiceInstance:245 - setServiceInstanceAsActive(): The ServiceInstance is com.dispatcher.ServiceInstance@c792d4[
  rwl=java.util.concurrent.locks.ReentrantReadWriteLock@1d840cd[Write locks = 1, Read locks = 0]
  id=2
  ipAddress=t3://ANOTHERCOMPUTER:7003
  jndiName=CustomerFacadeRemote/remote#com.user.facade.CustomerFacadeRemote
  state=ACTIVE
  timeToRefresh=300000
  timeOfLastUnsuccessfulRefresh=<null>
  userName=weblogic
  password=weblogic
  remoteReference=weblogic.rmi.internal.BasicRemoteRef - hostID: '-2438836063980619818S:12.34.78.1451:[7003,7003,-1,-1,-1,-1,-1]:APP:AnotherAppServer', oid: '445', channel: 'null'
]
16:02:55,838  INFO ServiceDispatcher:88 - START: getActiveServiceInstance():
16:02:55,838  INFO ServiceInstance:196 - getActiveRemoteReference(): Service Reference Found: 1
16:02:55,838  INFO ServiceDispatcher:109 - END: getActiveServiceInstance():
16:02:55,838  INFO CustomerServiceClient:680 - The service instance obtained in the client is com.dispatcher.ServiceInstance@9abc69[
  rwl=java.util.concurrent.locks.ReentrantReadWriteLock@5c98f3[Write locks = 0, Read locks = 0]
  id=1
  ipAddress=t3://localhost:7001
  jndiName=CustomerFacadeRemote/remote#com.user.facade.CustomerFacadeRemote
  state=ACTIVE
  timeToRefresh=300000
  timeOfLastUnsuccessfulRefresh=<null>
  userName=weblogic
  password=weblogic
  remoteReference=weblogic.rmi.internal.BasicRemoteRef - hostID: '-7327730829344636601S:127.0.0.1:[7001,7001,-1,-1,-1,-1,-1]:APP_Domain:AdminServer', oid: '459', channel: 'null'
]
16:02:55,838  INFO CustomerServiceClient:684 - The remote reference obtained from the service instance is 
weblogic.rmi.internal.BasicRemoteRef - hostID: '-7327730829344636601S:127.0.0.1:[7001,7001,-1,-1,-1,-1,-1]:APP_Domain:AdminServer', oid: '459', channel: 'null'
Enter the text "go" (Without double quotes) as soon as you are done with your activity: go
16:03:27,962  INFO CustomerServiceClient:706 - Using the cached remote reference weblogic.rmi.internal.BasicRemoteRef - hostID: '-7327730829344636601S:127.0.0.1:[7001,7001,-1,-1,-1,-1,-1]:APP_Domain:AdminServer', oid: '459', channel: 'null'
16:03:28,087 ERROR CustomerServiceClient:721 - EJB Exception occurred and the cause is 
javax.ejb.EJBException: [WorkManager:002916]Request belonging to WorkManager weblogic.kernel.Default, application development-ear#20557 is 
rejected as the WorkManager is shutdown; nested exception is: weblogic.work.WorkRejectedException: [WorkManager:002916]Request 
belonging to WorkManager weblogic.kernel.Default, application development-ear#20557 is rejected as the WorkManager is shutdown
weblogic.work.WorkRejectedException: [WorkManager:002916]Request belonging to WorkManager weblogic.kernel.Default, application development-ear#20557 is 
rejected as the WorkManager is shutdown
	at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:221)
	at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:224)
	at com.user.facade.impl.CustomerServiceSessionBean_hqoh1z_CustomerFacadeRemoteImpl_1001_WLStub.getCustomerByCustomerIDServiceDispatcherTest(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at weblogic.ejb.container.internal.RemoteBusinessIntfProxy.invoke(RemoteBusinessIntfProxy.java:63)
	at $Proxy4.getCustomerByCustomerIDServiceDispatcherTest(Unknown Source)
	at com.user.client.service.impl.CustomerServiceClient.getCustomerByCustomerIDServiceDispatcherTest(CustomerServiceClient.java:707)
	at com.ejbDispatcher.EJBServicesDispatcherTest.testServiceDispatcherBEAUndeployAndRunning(EJBServicesDispatcherTest.java:181)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:59)
	at org.junit.internal.runners.MethodRoadie.runTestMethod(MethodRoadie.java:98)
	at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:79)
	at org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:87)
	at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:77)
	at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:42)
	at org.junit.internal.runners.JUnit4ClassRunner.invokeTestMethod(JUnit4ClassRunner.java:88)
	at org.junit.internal.runners.JUnit4ClassRunner.runMethods(JUnit4ClassRunner.java:51)
	at org.junit.internal.runners.JUnit4ClassRunner$1.run(JUnit4ClassRunner.java:44)
	at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:27)
	at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:37)
	at org.junit.internal.runners.JUnit4ClassRunner.run(JUnit4ClassRunner.java:42)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:46)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
javax.ejb.EJBException: [WorkManager:002916]Request belonging to WorkManager weblogic.kernel.Default, application development-ear#20557 is rejected as the WorkManager 
is shutdown; nested exception is: weblogic.work.WorkRejectedException: [WorkManager:002916]Request belonging to WorkManager weblogic.kernel.Default, 
application development-ear#20557 is rejected as the WorkManager is shutdown
	at weblogic.ejb.container.internal.RemoteBusinessIntfProxy.invoke(RemoteBusinessIntfProxy.java:80)
	at $Proxy4.getCustomerByCustomerIDServiceDispatcherTest(Unknown Source)
	at com.user.client.service.impl.CustomerServiceClient.getCustomerByCustomerIDServiceDispatcherTest(CustomerServiceClient.java:707)
	at com.ejbDispatcher.EJBServicesDispatcherTest.testServiceDispatcherBEAUndeployAndRunning(EJBServicesDispatcherTest.java:181)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:59)
	at org.junit.internal.runners.MethodRoadie.runTestMethod(MethodRoadie.java:98)
	at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:79)
	at org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:87)
	at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:77)
	at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:42)
	at org.junit.internal.runners.JUnit4ClassRunner.invokeTestMethod(JUnit4ClassRunner.java:88)
	at org.junit.internal.runners.JUnit4ClassRunner.runMethods(JUnit4ClassRunner.java:51)
	at org.junit.internal.runners.JUnit4ClassRunner$1.run(JUnit4ClassRunner.java:44)
	at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:27)
	at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:37)
	at org.junit.internal.runners.JUnit4ClassRunner.run(JUnit4ClassRunner.java:42)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:46)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
16:03:28,150 ERROR ExceptionMessageFactory:209 - Allgemeiner technischer Fehler : createCustomer(CustomerObject custObject), EJB Exception Occured
javax.ejb.EJBException: [WorkManager:002916]Request belonging to WorkManager weblogic.kernel.Default, application development-ear#20557 is rejected 
as the WorkManager is shutdown; nested exception is: weblogic.work.WorkRejectedException: [WorkManager:002916]Request belonging to 
WorkManager weblogic.kernel.Default, application development-ear#20557 is rejected as the WorkManager is shutdown
weblogic.work.WorkRejectedException: [WorkManager:002916]Request belonging to WorkManager weblogic.kernel.Default, application development-ear#20557 is
rejected as the WorkManager is shutdown
	at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:221)
	at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:224)
	at com.user.facade.impl.CustomerServiceSessionBean_hqoh1z_CustomerFacadeRemoteImpl_1001_WLStub.getCustomerByCustomerIDServiceDispatcherTest(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at weblogic.ejb.container.internal.RemoteBusinessIntfProxy.invoke(RemoteBusinessIntfProxy.java:63)
	at $Proxy4.getCustomerByCustomerIDServiceDispatcherTest(Unknown Source)
	at com.user.client.service.impl.CustomerServiceClient.getCustomerByCustomerIDServiceDispatcherTest(CustomerServiceClient.java:707)
	at com.ejbDispatcher.EJBServicesDispatcherTest.testServiceDispatcherBEAUndeployAndRunning(EJBServicesDispatcherTest.java:181)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:59)
	at org.junit.internal.runners.MethodRoadie.runTestMethod(MethodRoadie.java:98)
	at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:79)
	at org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:87)
	at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:77)
	at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:42)
	at org.junit.internal.runners.JUnit4ClassRunner.invokeTestMethod(JUnit4ClassRunner.java:88)
	at org.junit.internal.runners.JUnit4ClassRunner.runMethods(JUnit4ClassRunner.java:51)
	at org.junit.internal.runners.JUnit4ClassRunner$1.run(JUnit4ClassRunner.java:44)
	at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:27)
	at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:37)
	at org.junit.internal.runners.JUnit4ClassRunner.run(JUnit4ClassRunner.java:42)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:46)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
javax.ejb.EJBException: [WorkManager:002916]Request belonging to WorkManager weblogic.kernel.Default, application development-ear#20557 is rejected as 
the WorkManager is shutdown; nested exception is: weblogic.work.WorkRejectedException: [WorkManager:002916]Request belonging to 
WorkManager weblogic.kernel.Default, application development-ear#20557 is rejected as the WorkManager is shutdown
	at weblogic.ejb.container.internal.RemoteBusinessIntfProxy.invoke(RemoteBusinessIntfProxy.java:80)
	at $Proxy4.getCustomerByCustomerIDServiceDispatcherTest(Unknown Source)
	at com.user.client.service.impl.CustomerServiceClient.getCustomerByCustomerIDServiceDispatcherTest(CustomerServiceClient.java:707)
	at com.ejbDispatcher.EJBServicesDispatcherTest.testServiceDispatcherBEAUndeployAndRunning(EJBServicesDispatcherTest.java:181)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:59)
	at org.junit.internal.runners.MethodRoadie.runTestMethod(MethodRoadie.java:98)
	at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:79)
	at org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:87)
	at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:77)
	at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:42)
	at org.junit.internal.runners.JUnit4ClassRunner.invokeTestMethod(JUnit4ClassRunner.java:88)
	at org.junit.internal.runners.JUnit4ClassRunner.runMethods(JUnit4ClassRunner.java:51)
	at org.junit.internal.runners.JUnit4ClassRunner$1.run(JUnit4ClassRunner.java:44)
	at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:27)
	at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:37)
	at org.junit.internal.runners.JUnit4ClassRunner.run(JUnit4ClassRunner.java:42)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:46)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
16:03:28,165  INFO CustomerServiceClient:737 - About to throw a GlobalRuntimeException.

I perform the JNDI lookup on the different servers during initialization of my Service Dispatcher (happens only once): The client will call new ServiceDispatcher
constructor with the necessary parameters as shown below in the code.

	public ServiceDispatcher(List<String> ipAddresses, String jndiName, String userName, String password) {
		init(ipAddresses, jndiName, userName, password);
	}	
	
	private void init(List<String> ipAddresses, String jndiName, String userName, String password) {
		this.jndiName = jndiName;
		this.ipAddresses = ipAddresses;
		this.userName = userName;
		this.password = password;
		
		serviceInstanceList = new ArrayList<ServiceInstance>();
		LOG.debug("Starting to initialize the service instances list");
		int id = 1;
		for (String ipAddress : ipAddresses) {
			ServiceInstance instance = new ServiceInstance(ipAddress, jndiName, userName, password, maxTimeForRefresh, id);			
			id++;
			/* Do an eagar JNDI lookup on the service instances. */
			instance.doEagarJNDILookUp();
			serviceInstanceList.add(instance); // The serviceInstanceList is a member variable in this Service Dispatcher class
		}
		LOG.debug("Done initializing the service instances list");
	}

The ServiceInstance is just a Value Object and the JNDI LookUp method is as below:

	protected void doEagarJNDILookUp() {
		LOG.info("doEagarJNDILookUp(): Performing JNDI LookUp during initialization with the following parameters: ");
		LOG.info("ipAddress = " + this.ipAddress + ", jndiName= " + this.jndiName + ", userName = " + this.userName + ", password = " + this.password);
		Context context = null;
		try {
			context = getContext(this.ipAddress, this.userName, this.password);
			Object lookedUpObject = context.lookup(this.jndiName);
			
			this.setServiceInstanceAsActive(lookedUpObject);
			LOG.info("doJNDILookUp(): JNDI LookUp during initialization was successful! ");
		} 
		catch (Throwable e) {
			LOG.error("doEagarJNDILookUp(): JNDI Lookup Fail", e);
			this.setServiceInstanceAsDown();				
		}
		finally {
			if(context != null) {
				try {
					context.close();
				} 
				catch (Throwable e) {
					LOG.error("doEagarJNDILookUp(): Context cannot be closed", e);
					this.setServiceInstanceAsDown();
				}				
			}						
		}
	}

Baffled with the exception that Weblogic throws. Below is how my client call looks like:

   
	public CustomerVO getCustomerByIDServiceDispatcherTest(String user)
			throws GlobalRuntimeException {
    	boolean isServiceCallExecuted = false;
    	CustomerVO cust = null;
    	
    	while(!isServiceCallExecuted) {
    		ServiceDispatcher dispatcher = this.getServiceDispatcher(); // calls the constructor of ServiceDispatcher with the necessary parameters...
    		ServiceInstance serviceInstance = dispatcher.getActiveServiceInstance(); //getCustomerFacadeFromDispatcher(dispatcher);
    		LOG.info("The service instance obtained in the client is " + serviceInstance);
    		Object remoteServiceReference = serviceInstance.getRemoteReference();
        	CustomerFacadeRemote customerFacadeRemote = (CustomerFacadeRemote) remoteServiceReference;
        	try {
        		LOG.info("The remote reference obtained from the service instance is " + customerFacadeRemote);
        		cust = customerFacadeRemote.getCustomerByCustomerIDServiceDispatcherTest(serviceId, customerId);
        		//isServiceCallExecuted = true;        		
        	}
            catch (EJBException ejb) {
                throw new GlobalRuntimeException(errMessage, ejb);
            }
        	
        	/*
        	 * Wait for the user input (do any one of the following in the meantime)
        	 * (1) Restart local weblogic
        	 * (2) Undeploy EJB's from local weblogic
        	 * (3) Shutdown local weblogic
        	 */
        	waitForUserInput(); //this is where I undeploy the EJB's in my local BEA server and proceed furthur with the call as below...
        	
        	/* Call the service once again. */
        	try {
        		LOG.info("Using the cached remote reference " + customerFacadeRemote);
        		cust = customerFacadeRemote.getCustomerByCustomerIDServiceDispatcherTest(serviceId, customerId); // EXCEPTION HAPPENS HERE!!!!!!!!!!!!
        		isServiceCallExecuted = true;        		
        	}
        	catch (NoSuchEJBException ex) {
        		LOG.info("NoSuchEJBException occurred " + ex.getCause());
        		isServiceCallExecuted = false;
        		/*
        		 * Here the reference from the cache is invalid. So set this
        		 * reference as invalid and make the call again. 
        		 */
        		dispatcher.setServiceInstanceAsInvalid(serviceInstance);
        	}
        	catch (EJBException ejb) {
        		/* Log the exception. */
        		LOG.error("EJB Exception occurred and the cause is ", ejb);
        		final Throwable cause = ExceptionUtils.getRootCause(ejb);
        		if (cause instanceof java.rmi.ConnectException || cause instanceof java.net.ConnectException) {
        			LOG.info("One of the cause is either a java.rmi.ConnectException or java.net.ConnectException.");
        			isServiceCallExecuted = false;
        			
            		/*
            		 * Here the reference from the cache is invalid. So set this
            		 * reference as invalid and make the call again. 
            		 */
            		dispatcher.setServiceInstanceAsInvalid(serviceInstance);        			
        		}
        		else {
                     throw new GlobalRuntimeException(errMessage, ejb);        			
        		}        		
        	}
    	}
    	return cust;		
	}

[Deepak Bala]

So you want to confirm if the spec is followed correctly ? Sounds like a case to raise a ticket with BEA ?

[Joe Harry]

Is this an issue with BEA? Or is it a design issue with what I have done.

[Deepak Bala]

A close look at the exception suggests that the work manager behind the EAR application by itself has shutdown ?

Have you tried undeploying the EJBs alone ?

[Joe Harry]

What I did was just to stop the EAR application and then tried to access the remote facade with the cached reference. Later I even un deployed the EJB's but even after that I keep getting this same exception message.