Nodeagent is getting error when I start

[Pavandevops Kumar]

Hi,

When I try to start the nodeagent as windows service I am receiving the below logs in startServer.log.
WAS for nodeagent is getting started but not invoking the JVM in it.
When I try to start node agent using cmd startNode.bat then JVM's are gettign started.
Below are the logs that were generated.

************ Start Display Current Environment ************
Host Operating System is Windows Server 2019, version 10.0
Java version = 1.8.0_321, Java Runtime Version = 8.0.7.5 - pwa6480sr7fp5-20220208_01(SR7 FP5), Java Compiler = j9jit29, Java VM name = IBM J9 VM
was.install.root = D:\IBM\WebSphere\AppServer
user.install.root = D:\IBM\WebSphere\AppServer\profiles\Custom02
Java Home = D:\IBM\WebSphere\AppServer\java\8.0\jre
ws.ext.dirs = D:\\IBM\\WebSphere\\AppServer\\java\\8.0\\lib;D:\\IBM\\WebSphere\\AppServer\\classes;D:\\IBM\\WebSphere\\AppServer\\lib;D:\\IBM\\WebSphere\\AppServer\\installedChannels;D:\\IBM\\WebSphere\\AppServer\\lib\\ext;D:\\IBM\\WebSphere\\AppServer\\web\\help;D:\\IBM\\WebSphere\\AppServer\\deploytool\\itp\\plugins\\com.ibm.etools.ejbdeploy\\runtime
Classpath = D:\IBM\WebSphere\AppServer\profiles\Custom02\properties;D:\IBM\WebSphere\AppServer\properties;D:\IBM\WebSphere\AppServer\lib\startup.jar;D:\IBM\WebSphere\AppServer\lib\bootstrap.jar;D:\IBM\WebSphere\AppServer\java\8.0\lib\tools.jar;D:\IBM\WebSphere\AppServer/lib/lmproxy.jar;D:\IBM\WebSphere\AppServer/lib/urlprotocols.jar
Java Library path = D:\IBM\WebSphere\AppServer\java\8.0\jre\bin\compressedrefs;D:\IBM\WebSphere\AppServer\java\8.0\jre\bin;C:\Windows\system32;C:\Windows;D:\IBM\WebSphere\AppServer\lib\native\win\x86_64;D:\IBM\WebSphere\AppServer\bin;D:\IBM\WebSphere\AppServer\java\8.0\bin;D:\IBM\WebSphere\AppServer\java\8.0\jre\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\SafeNet\Authentication\SAC\x64;C:\Program Files\SafeNet\Authentication\SAC\x32;C:\Users\bps-systitus\AppData\Local\Microsoft\WindowsApps;;.
Orb Version = IBM Java ORB build orb80-20220208.00
Max file descriptor count = --
Current trace specification = *=info:com.ibm.*=all
************* End Display Current Environment *************
[4/28/22 12:07:31:917 PDT] 00000001 ManagerAdmin  I   TRAS0017I: The startup trace state is *=info:com.ibm.*=all.
[4/28/22 12:07:31:932 PDT] 00000001 WsServerContr 1   Executing executeUtilityOnWindows with args: D:\IBM\WebSphere\AppServer\profiles\Custom02\config ctgCell01 HDH-MAX-APP02-Node01 nodeagent -nodeagent
[4/28/22 12:07:31:932 PDT] 00000001 WsServerContr 1   Getting WindowsService obj with: nodeagent and D:\IBM\WebSphere\AppServer\profiles\Custom02\config ctgCell01 HDH-MAX-APP02-Node01 nodeagent -nodeagent
[4/28/22 12:07:31:932 PDT] 00000001 WindowsServic >  WindowsService for server nodeagent Entry
                                <null>
[4/28/22 12:07:31:932 PDT] 00000001 WindowsServic >  getWindowsServiceNameForServer nodeagent Entry
                                <null>
[4/28/22 12:07:31:932 PDT] 00000001 WindowsServic >  getWASServiceCmdPath  Entry
                                <null>
[4/28/22 12:07:31:932 PDT] 00000001 WindowsServic 1   In executeCommand, args are: D:\IBM\WebSphere\AppServer\bin\WASService.exe -getServiceName nodeagent -profilePath D:\IBM\WebSphere\AppServer\profiles\Custom02
[4/28/22 12:07:31:932 PDT] 00000001 WindowsServic 1   Runtime.exec() returned process java.lang.ProcessImpl@89017ce5
[4/28/22 12:07:31:979 PDT] 00000016 WindowsServic 3   inputRec == IBMWAS90Service - nodeagent
[4/28/22 12:07:31:979 PDT] 00000001 WindowsServic 1   retCode from childProcess.waitFor() is 0
[4/28/22 12:07:31:979 PDT] 00000001 WindowsServic I   ADMU7714I: WASService command for server nodeagent completed successfully.
[4/28/22 12:07:31:979 PDT] 00000001 WindowsServic 3   exitCode = 0
[4/28/22 12:07:32:995 PDT] 00000001 WindowsServic <  getWindowsServiceNameForServer Exit
[4/28/22 12:07:32:995 PDT] 00000001 WindowsServic 1   Trying to new up WindowsService using serverName=nodeagent, winServiceName=nodeagent
[4/28/22 12:07:32:995 PDT] 00000001 WindowsServic >  getServiceLogName for nodeagent Entry
[4/28/22 12:07:32:995 PDT] 00000001 WindowsServic >  getWASServiceCmdPath  Entry
                                <null>
[4/28/22 12:07:32:995 PDT] 00000001 WindowsServic 1   In executeCommand, args are: D:\IBM\WebSphere\AppServer\bin\WASService.exe -getServiceLogName nodeagent -profilePath D:\IBM\WebSphere\AppServer\profiles\Custom02
[4/28/22 12:07:32:995 PDT] 00000001 WindowsServic 1   Runtime.exec() returned process java.lang.ProcessImpl@52acbbfd
[4/28/22 12:07:33:042 PDT] 00000018 WindowsServic 3   inputRec == D:\IBM\WebSphere\AppServer\profiles\Custom02\logs\nodeagent\startNode.log
[4/28/22 12:07:33:042 PDT] 00000001 WindowsServic 1   retCode from childProcess.waitFor() is 0
[4/28/22 12:07:33:042 PDT] 00000001 WindowsServic I   ADMU7714I: WASService command for server nodeagent completed successfully.
[4/28/22 12:07:33:042 PDT] 00000001 WindowsServic 3   exitCode = 0
[4/28/22 12:07:34:057 PDT] 00000001 WindowsServic <  getServiceLogName Exit
[4/28/22 12:07:34:057 PDT] 00000001 WindowsServic <  WindowsService Exit
[4/28/22 12:07:34:057 PDT] 00000001 WindowsServic >  getServiceStatus Entry
[4/28/22 12:07:34:057 PDT] 00000001 WindowsServic >  getWASServiceCmdPath  Entry
                                <null>
[4/28/22 12:07:34:057 PDT] 00000001 WindowsServic 1   In executeCommand, args are: D:\IBM\WebSphere\AppServer\bin\WASService.exe -status nodeagent
[4/28/22 12:07:34:057 PDT] 00000001 WindowsServic 1   Runtime.exec() returned process java.lang.ProcessImpl@185bfdb9
[4/28/22 12:07:34:104 PDT] 0000001a WindowsServic 3   inputRec == The service is stopped.
[4/28/22 12:07:34:104 PDT] 00000001 WindowsServic 1   retCode from childProcess.waitFor() is 1
[4/28/22 12:07:34:104 PDT] 00000001 WindowsServic I   ADMU7714I: WASService command for server nodeagent completed successfully.
[4/28/22 12:07:34:104 PDT] 00000001 WindowsServic 3   exitCode = 1
[4/28/22 12:07:34:104 PDT] 00000001 WindowsServic <  getServiceStatus Exit
[4/28/22 12:07:34:104 PDT] 00000001 WindowsServic >  getServiceStatus Entry
[4/28/22 12:07:34:104 PDT] 00000001 WindowsServic >  getWASServiceCmdPath  Entry
                                <null>
[4/28/22 12:07:34:104 PDT] 00000001 WindowsServic 1   In executeCommand, args are: D:\IBM\WebSphere\AppServer\bin\WASService.exe -status nodeagent
[4/28/22 12:07:34:104 PDT] 00000001 WindowsServic 1   Runtime.exec() returned process java.lang.ProcessImpl@6255b8dd
[4/28/22 12:07:34:151 PDT] 0000001c WindowsServic 3   inputRec == The service is stopped.
[4/28/22 12:07:34:151 PDT] 00000001 WindowsServic 1   retCode from childProcess.waitFor() is 1
[4/28/22 12:07:34:151 PDT] 00000001 WindowsServic I   ADMU7714I: WASService command for server nodeagent completed successfully.
[4/28/22 12:07:34:151 PDT] 00000001 WindowsServic 3   exitCode = 1
[4/28/22 12:07:34:151 PDT] 00000001 WindowsServic <  getServiceStatus Exit
[4/28/22 12:07:34:151 PDT] 00000001 WsServerContr 1   Window service status (1=stopped, 2=started, 3=not-registred) is 1
[4/28/22 12:07:34:151 PDT] 00000001 WindowsServic >  doWASServiceAction action start Entry
                                <null>
[4/28/22 12:07:34:151 PDT] 00000001 AdminTool     A   ADMU7701I: Because nodeagent is registered to run as a Windows Service, the request to start this server will be completed by starting the associated Windows Service.
[4/28/22 12:07:34:151 PDT] 00000001 WindowsServic >  getWASServiceCmdPath  Entry
                                <null>
[4/28/22 12:07:34:151 PDT] 00000001 WindowsServic 1   In executeCommand, args are: D:\IBM\WebSphere\AppServer\bin\WASService.exe -start nodeagent -nodeagent
[4/28/22 12:07:34:151 PDT] 00000001 WindowsServic 1   Runtime.exec() returned process java.lang.ProcessImpl@6609c2bb
[4/28/22 12:07:34:151 PDT] 00000020 WindowsServic 3   waitForFileExist(D:\IBM\WebSphere\AppServer\profiles\Custom02\logs\nodeagent\startNode.log)
[4/28/22 12:07:34:651 PDT] 00000020 EncodingUtils >  getProperEncoding Entry
[4/28/22 12:07:34:651 PDT] 00000020 EncodingUtils 3   System.getProperty("file.encoding") is: Cp1252
[4/28/22 12:07:34:651 PDT] 00000020 EncodingUtils >  isPlatformWindows() Entry
[4/28/22 12:07:34:651 PDT] 00000020 WindowsServic 3   *** returned readerencoding = Cp850
[4/28/22 12:07:36:917 PDT] 0000001e WindowsServic 3   inputRec ==
[4/28/22 12:07:36:917 PDT] 00000001 WindowsServic 1   retCode from childProcess.waitFor() is -3
[4/28/22 12:07:36:917 PDT] 00000001 WindowsServic E   ADMU7713E: WASService command for server nodeagent failed. Unexpected exit code -3 from running D:\IBM\WebSphere\AppServer\bin\WASService.exe -start nodeagent -nodeagent .
[4/28/22 12:07:36:917 PDT] 00000001 WindowsServic 3   -- interrupt the thread that is collecting stdout
[4/28/22 12:07:37:917 PDT] 00000001 WindowsServic 3   exitCode = -3
[4/28/22 12:07:37:917 PDT] 00000020 WindowsServic 3   readAndDisplayServiceLogFile(), caught InterruptedException
[4/28/22 12:07:37:917 PDT] 00000001 AdminTool     A   ADMU7704E: Failed while trying to start the Windows Service associated with server: nodeagent;
probable error executing WASService.exe: ""


Please assist me with the error.

Regards,
Pavan.

[Stephan van Hulst]

What happens when you run the following command directly?
D:\IBM\WebSphere\AppServer\bin\WASService.exe -start nodeagent -nodeagent

[Pavandevops Kumar]

-When I use the above command Node agent is getting started & jvm are running, but the windows service is not running.
-When I stop the nodeagent with the above command & start the windows service, nodeagent is getting started but not the jvm in the nodeagent.

I assume 2 services are running under same path. Please suggest any process.

Thanks,
Pavan.

[Pavandevops Kumar]

The below is the systemout log of the jvm running on node

[4/28/22 11:54:26:164 PDT] 0000004f ViewReceiver  I   DCSV1033I: DCS Stack DefaultCoreGroup at Member ctgCell01\MAX-Node01\HDH-MAX-VM1: Confirmed all new view members in view identifier (17:0.ctgCell01\MAX-Node01\HDH-MAX-VM1). View channel type is View|Ptp.
[4/28/22 11:55:19:263 PDT] 0000004f RLSHAGroupCal W   CWRLS0030W: Waiting for HAManager to activate recovery processing for local WebSphere server. For more information about how to troubleshoot startup, see https://www.ibm.com/support/knowledgecenter/en/SSAW57_9.0.0/com.ibm.websphere.nd.multiplatform.doc/ae/ttrb_server_startup.html
[4/28/22 11:56:19:270 PDT] 0000004e RLSHAGroupCal W   CWRLS0030W: Waiting for HAManager to activate recovery processing for local WebSphere server. For more information about how to troubleshoot startup, see https://www.ibm.com/support/knowledgecenter/en/SSAW57_9.0.0/com.ibm.websphere.nd.multiplatform.doc/ae/ttrb_server_startup.html
[4/28/22 11:57:14:804 PDT] 0000004f CoreGroupMemb I   DCSV0008I: DCS Stack DefaultCoreGroup at Member ctgCell01\MAX-Node01\HDH-MAX-VM1: General information message. Internal details: Failed to Form Inital View. Number of tries= 13 [ctgCell01\MAX-Node01\HDH-MAX-VM1 ctgCell01\MAX-Node01\nodeagent ctgCell01\ctgCellManager01\dmgr]
[4/28/22 11:57:19:286 PDT] 0000004e RLSHAGroupCal W   CWRLS0030W: Waiting for HAManager to activate recovery processing for local WebSphere server. For more information about how to troubleshoot startup, see https://www.ibm.com/support/knowledgecenter/en/SSAW57_9.0.0/com.ibm.websphere.nd.multiplatform.doc/ae/ttrb_server_startup.html
[4/28/22 11:58:19:287 PDT] 00000050 RLSHAGroupCal W   CWRLS0030W: Waiting for HAManager to activate recovery processing for local WebSphere server. For more information about how to troubleshoot startup, see https://www.ibm.com/support/knowledgecenter/en/SSAW57_9.0.0/com.ibm.websphere.nd.multiplatform.doc/ae/ttrb_server_startup.html
[4/28/22 11:59:19:301 PDT] 00000050 RLSHAGroupCal W   CWRLS0030W: Waiting for HAManager to activate recovery processing for local WebSphere server. For more information about how to troubleshoot startup, see https://www.ibm.com/support/knowledgecenter/en/SSAW57_9.0.0/com.ibm.websphere.nd.multiplatform.doc/ae/ttrb_server_startup.html
[4/28/22 12:00:19:317 PDT] 0000004e RLSHAGroupCal W   CWRLS0030W: Waiting for HAManager to activate recovery processing for local WebSphere server. For more information about how to troubleshoot startup, see https://www.ibm.com/support/knowledgecenter/en/SSAW57_9.0.0/com.ibm.websphere.nd.multiplatform.doc/ae/ttrb_server_startup.html
[4/28/22 12:01:19:331 PDT] 0000004b RLSHAGroupCal W   CWRLS0030W: Waiting for HAManager to activate recovery processing for local WebSphere server. For more information about how to troubleshoot startup, see https://www.ibm.com/support/knowledgecenter/en/SSAW57_9.0.0/com.ibm.websphere.nd.multiplatform.doc/ae/ttrb_server_startup.html
[4/28/22 12:02:19:343 PDT] 00000050 RLSHAGroupCal W   CWRLS0030W: Waiting for HAManager to activate recovery processing for local WebSphere server. For more information about how to troubleshoot startup, see https://www.ibm.com/support/knowledgecenter/en/SSAW57_9.0.0/com.ibm.websphere.nd.multiplatform.doc/ae/ttrb_server_startup.html
[4/28/22 12:03:19:352 PDT] 0000004e RLSHAGroupCal W   CWRLS0030W: Waiting for HAManager to activate recovery processing for local WebSphere server. For more information about how to troubleshoot startup, see https://www.ibm.com/support/knowledgecenter/en/SSAW57_9.0.0/com.ibm.websphere.nd.multiplatform.doc/ae/ttrb_server_startup.html
[4/28/22 12:04:19:355 PDT] 0000004b RLSHAGroupCal W   CWRLS0030W: Waiting for HAManager to activate recovery processing for local WebSphere server. For more information about how to troubleshoot startup, see https://www.ibm.com/support/knowledgecenter/en/SSAW57_9.0.0/com.ibm.websphere.nd.multiplatform.doc/ae/ttrb_server_startup.html
[4/28/22 12:04:26:168 PDT] 0000003e RoleMergeLead I   DCSV8054I: DCS Stack DefaultCoreGroup at Member ctgCell01\MAX-Node01\HDH-MAX-VM1: View change in process.
[4/28/22 12:04:26:168 PDT] 0000003f VSyncAlgo2    I   DCSV2004I: DCS Stack DefaultCoreGroup at Member ctgCell01\MAX-Node01\HDH-MAX-VM1: View synchronization completed successfully. The View Identifier is (17:0.ctgCell01\MAX-Node01\HDH-MAX-VM1). The internal details are The leader equals ctgCell01\MAX-Node01\HDH-MAX-VM1, ccv equals ((17:0.ctgCell01\MAX-Node01\HDH-MAX-VM1).1.0).
[4/28/22 12:04:26:168 PDT] 0000003e RoleMergeLead I   DCSV8030I: DCS Stack DefaultCoreGroup at Member ctgCell01\MAX-Node01\HDH-MAX-VM1: Failed  to join or establish a view with member [ctgCell01\MAX-Node01\nodeagent]. The reason is Sender's reason: Merge Set did not contain all of target view.
[4/28/22 12:04:26:168 PDT] 0000003e CoreGroupMemb I   DCSV8050I: DCS Stack DefaultCoreGroup at Member ctgCell01\MAX-Node01\HDH-MAX-VM1: New view installed, identifier (18:0.ctgCell01\MAX-Node01\HDH-MAX-VM1), view size is 1 (AV=1, CD=3, CN=3, DF=10)
[4/28/22 12:04:26:183 PDT] 0000004b ViewReceiver  I   DCSV1033I: DCS Stack DefaultCoreGroup at Member ctgCell01\MAX-Node01\HDH-MAX-VM1: Confirmed all new view members in view identifier (18:0.ctgCell01\MAX-Node01\HDH-MAX-VM1). View channel type is View|Ptp.
[4/28/22 12:05:19:370 PDT] 0000004f RLSHAGroupCal W   CWRLS0030W: Waiting for HAManager to activate recovery processing for local WebSphere server. For more information about how to troubleshoot startup, see https://www.ibm.com/support/knowledgecenter/en/SSAW57_9.0.0/com.ibm.websphere.nd.multiplatform.doc/ae/ttrb_server_startup.html
[4/28/22 12:05:26:509 PDT] 00000049 DiscoveryRcv  W   DCSV1115W: DCS Stack DefaultCoreGroup at Member ctgCell01\MAX-Node01\HDH-MAX-VM1: Member ctgCell01\MAX-Node01\nodeagent connection  was closed. Member will  be removed from view. DCS connection status is Discovery|Ptp, receiver closed. Connection closed by 10.113.40.100:9354||0.
[4/28/22 12:06:19:379 PDT] 0000004f RLSHAGroupCal W   CWRLS0030W: Waiting for HAManager to activate recovery processing for local WebSphere server. For more information about how to troubleshoot startup, see https://www.ibm.com/support/knowledgecenter/en/SSAW57_9.0.0/com.ibm.websphere.nd.multiplatform.doc/ae/ttrb_server_startup.html
[4/28/22 12:07:19:385 PDT] 00000050 RLSHAGroupCal W   CWRLS0030W: Waiting for HAManager to activate recovery processing for local WebSphere server. For more information about how to troubleshoot startup, see https://www.ibm.com/support/knowledgecenter/en/SSAW57_9.0.0/com.ibm.websphere.nd.multiplatform.doc/ae/ttrb_server_startup.html
[4/28/22 12:08:19:391 PDT] 0000004f RLSHAGroupCal W   CWRLS0030W: Waiting for HAManager to activate recovery processing for local WebSphere server. For more information about how to troubleshoot startup, see https://www.ibm.com/support/knowledgecenter/en/SSAW57_9.0.0/com.ibm.websphere.nd.multiplatform.doc/ae/ttrb_server_startup.html