JBoss Cache, replication problem on server restart

[seba ccaa]

Hi everyone,
I configured a Jboss cluster, with Jboss portal and a replicated cache. After some months I noticed an error that occurs during the restart of one of the nodes in the cluster, it seem that the node is not able to receive the cache informations, this is the error:

11:05:17,511 INFO  [InterceptorChainFactory] interceptor chain is:
class org.jboss.cache.interceptors.CallInterceptor
class org.jboss.cache.interceptors.PessimisticLockInterceptor
class org.jboss.cache.interceptors.UnlockInterceptor
class org.jboss.cache.interceptors.InvalidationInterceptor
class org.jboss.cache.interceptors.TxInterceptor
class org.jboss.cache.interceptors.CacheMgmtInterceptor
11:05:17,646 INFO  [STDOUT]
-------------------------------------------------------
GMS: address is 192.168.100.108:37760
-------------------------------------------------------
11:05:17,842 INFO  [TreeCache] viewAccepted(): [192.168.100.110:46818|30] [192.168.100.110:46818, 192.168.100.109:57600, 192.168.100.106:33415, 192.168.100.105:60165, 192.168.100.108:37760]
11:05:17,881 INFO  [TreeCache] TreeCache local address is 192.168.100.108:37760
11:08:59,574 WARN  [STATE_TRANSFER] discovered that the state provider (192.168.100.110:46818) crashed; will return null state to application
11:08:59,576 WARN  [STATE_TRANSFER] digest received from 192.168.100.108:37760 is null, skipping setting digest !
11:08:59,576 WARN  [STATE_TRANSFER] state received from 192.168.100.108:37760 is null, will return null state to application
11:08:59,579 INFO  [TreeCache] viewAccepted(): [192.168.100.109:57600|31] [192.168.100.109:57600, 192.168.100.106:33415, 192.168.100.105:60165, 192.168.100.108:37760]
11:08:59,683 WARN  [ServiceController] Problem starting service portal:service=CMSTreeCache
org.jboss.cache.CacheException: Initial state transfer failed: Channel.getState() returned false
        at org.jboss.cache.TreeCache.fetchStateOnStartup(TreeCache.java:3349)
        at org.jboss.cache.TreeCache.startService(TreeCache.java:1543)
        at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
        at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
        at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
        at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
        at $Proxy0.start(Unknown Source)
        at org.jboss.system.ServiceController.start(ServiceController.java:417)
        at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
        at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
        at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
        at $Proxy8.deploy(Unknown Source)
        at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:421)
        at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:634)
        at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:263)
        at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:336)
        at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
        at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
        at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
        at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
        at $Proxy0.start(Unknown Source)
        at org.jboss.system.ServiceController.start(ServiceController.java:417)
        at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
        at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
        at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
        at $Proxy4.start(Unknown Source)
        at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302)
        at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
        at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
        at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
        at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:766)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
        at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
        at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
        at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
        at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
        at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
        at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
        at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
        at $Proxy5.deploy(Unknown Source)
        at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:482)
        at org.jboss.system.server.ServerImpl.start(ServerImpl.java:362)
        at org.jboss.Main.boot(Main.java:200)
        at org.jboss.Main$1.run(Main.java:490)
        at java.lang.Thread.run(Thread.java:595)
11:08:59,726 INFO  [STDOUT]
-------------------------------------------------------
GMS: address is 192.168.100.108:37307
-------------------------------------------------------
11:09:04,731 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:09:11,738 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:09:18,745 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:09:25,752 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:09:32,758 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:09:39,763 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:09:46,768 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:09:53,775 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:10:00,782 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:10:07,789 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:10:14,796 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:10:21,802 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:10:28,810 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:10:35,818 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:10:42,824 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:10:49,830 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:10:56,838 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:11:03,846 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:11:10,853 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:11:17,859 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:11:24,866 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:11:31,873 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:11:38,881 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:11:45,889 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:11:52,895 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:11:59,903 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:12:06,910 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:12:13,918 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:12:20,926 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:12:27,932 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:12:34,938 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:12:41,945 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:12:48,953 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:12:55,959 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:13:02,967 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:13:09,974 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:13:16,981 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:13:23,988 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:13:30,994 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:13:38,001 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:13:45,008 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:13:52,016 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:13:59,021 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:14:06,028 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:14:13,036 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:14:20,042 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:14:27,050 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:14:34,055 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:14:41,060 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:14:48,067 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:14:55,073 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:15:02,080 WARN  [GMS] join(192.168.100.108:37307) sent to 192.168.100.110:49286 timed out, retrying
11:15:04,562 INFO  [TreeCache] viewAccepted(): [192.168.100.110:49286|30] [192.168.100.110:49286, 192.168.100.189:40258, 192.168.100.186:36131, 192.168.100.185:38182, 192.168.100.108:37307]
11:15:04,567 INFO  [TreeCache] TreeCache local address is 192.168.100.108:37307
11:15:04,606 INFO  [TreeCache] received the state (size=65536 bytes)
11:15:04,758 INFO  [TreeCache] state was retrieved successfully (in 191 milliseconds)
11:15:04,761 INFO  [PageViewCache] Starting ADR cache service
11:15:04,773 INFO  [PageViewCache] ADR cache service started
11:15:04,817 INFO  [STDOUT]
-------------------------------------------------------
GMS: address is 192.168.100.108:53489
-------------------------------------------------------
11:15:05,656 INFO  [TreeCache] viewAccepted(): [192.168.100.110:51372|30] [192.168.100.110:51372, 192.168.100.109:56764, 192.168.100.106:50895, 192.168.100.105:37731, 192.168.100.108:53489]
11:15:05,659 INFO  [TreeCache] TreeCache local address is 192.168.100.108:53489
11:15:07,827 INFO  [TreeCache] received the state (size=4194304 bytes)
11:15:08,534 INFO  [TreeCache] state was retrieved successfully (in 2875 milliseconds)

At the and of the restart there is the info about an incomplete Deployment :

11:18:43,795 ERROR [URLDeploymentScanner] Incomplete Deployment listing:

--- MBeans waiting for other MBeans ---
ObjectName: portal:service=CMSTreeCache
  State: FAILED
  Reason: org.jboss.cache.CacheException: Initial state transfer failed: Channel.getState() returned false
  I Depend On:
    jboss:service=Naming
    jboss:service=TransactionManager
  Depends On Me:
    portal:service=CMSTreeCacheService

ObjectName: portal:service=CMSTreeCache
  State: FAILED
  Reason: org.jboss.cache.CacheException: Initial state transfer failed: Channel.getState() returned false
  I Depend On:
    jboss:service=Naming
    jboss:service=TransactionManager
  Depends On Me:
    portal:service=CMSTreeCacheService


11:18:43,882 INFO  [Http11BaseProtocol] Starting Coyote HTTP/1.1 on http-192.168.100.108-8080
11:18:44,028 INFO  [ChannelSocket] JK: ajp13 listening on /192.168.100.108:8009
11:18:44,047 INFO  [JkMain] Jk running ID=0 time=1/55  config=null
11:18:44,082 INFO  [Server] JBoss (MX MicroKernel) [4.0.5.GA (build: CVSTag=Branch_4_0 date=200610162339)] Started in 13m:38s:963ms

is this a bug o a configuratin problem?


thanks in advance
Sebax