Question

Unable to restart Pre-prod server due to HazelCast error

getting below error while starting up our pre-prod server which is running on pega 7.1.9. Attached are server and pegarules log

2019-11-28 11:42:12,116 [ lx64546.sbcore.net] [ STANDARD] [ ] ( hazelcast.instance.Node) FATAL - [127.0.0.1]:5702 [dad666dc600d6f25c06ea7b7f3d4f674] [3.4.1] Could not join cluster in 300000 ms. Shutting down now!

2019-11-28 11:42:12,152 [ lx64546.sbcore.net] [ STANDARD] [ ] ( internal.access.DatabaseImpl) ERROR - Encountered problem when performing a Commit

java.lang.IllegalStateException: Node failed to start!

at com.hazelcast.instance.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:125)

at com.hazelcast.instance.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:153)

at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:136)

at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:112)

at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:58)

at com.pega.pegarules.cluster.internal.PRClusterHazelcastImpl.initialize(PRClusterHazelcastImpl.java:448)

at com.pega.pegarules.cluster.internal.PRClusterImpl.getInstance(PRClusterImpl.java:195)

at com.pega.pegarules.cluster.internal.PRClusterProviderImpl.getInstance(PRClusterProviderImpl.java:96)

at com.pega.pegarules.cluster.internal.PRClusterProviderImpl.setSystemPulseTransport(PRClusterProviderImpl.java:149)

at com.pega.pegarules.cluster.internal.PRClusterProviderImpl.useClusterSystemPulse(PRClusterProviderImpl.java:128)

at com.pega.pegarules.data.internal.access.Committer.performOps(Committer.java:529)

at com.pega.pegarules.data.internal.access.DatabaseImpl.attemptToProcessUpdates(DatabaseImpl.java:2669)

at com.pega.pegarules.data.internal.access.DatabaseImpl.processUpdates(DatabaseImpl.java:2322)

at com.pega.pegarules.data.internal.access.Saver.save(Saver.java:651)

at com.pega.pegarules.data.internal.access.DatabaseImpl.save(DatabaseImpl.java:5108)

at com.pega.pegarules.data.internal.access.DatabaseImpl.save(DatabaseImpl.java:5094)

at com.pega.pegarules.data.internal.access.DatabaseImpl.save(DatabaseImpl.java:5086)

at com.pega.pegarules.session.internal.mgmt.PRNodeImpl.initializeSystem(PRNodeImpl.java:864)

at com.pega.pegarules.session.internal.mgmt.PREnvironment.getThreadAndInitialize(PREnvironment.java:375)

at com.pega.pegarules.session.internal.PRSessionProviderImpl.getThreadAndInitialize(PRSessionProviderImpl.java:1998)

at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineStartup.initEngine(EngineStartup.java:664)

at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._initEngine_privact(EngineImpl.java:165)

at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.doStartup(EngineImpl.java:138)

at com.pega.pegarules.session.internal.engineinterface.etier.ejb.EngineBean.doStartup(EngineBean.java:121)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)

at java.lang.reflect.Method.invoke(Method.java:620)

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:370)

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:411)

at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:223)

at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingException(AppServerBridgeToPega.java:245)

at com.pega.pegarules.internal.etier.ejb.EngineBeanBoot.doStartup(EngineBeanBoot.java:130)

at com.pega.pegarules.internal.etier.interfaces.EJSLocalStatelessEngineBMT_f2439d86.doStartup(Unknown Source)

at com.pega.pegarules.web.servlet.WebAppLifeCycleListener._contextInitialized_privact(WebAppLifeCycleListener.java:311)

at com.pega.pegarules.web.servlet.WebAppLifeCycleListener.contextInitialized(WebAppLifeCycleListener.java:218)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)

at java.lang.reflect.Method.invoke(Method.java:620)

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:370)

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:411)

at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:223)

at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(AppServerBridgeToPega.java:272)

at com.pega.pegarules.internal.web.servlet.WebAppLifeCycleListenerBoot.contextInitialized(WebAppLifeCycleListenerBoot.java:83)

at com.ibm.ws.webcontainer.webapp.WebApp.notifyServletContextCreated(WebApp.java:1709)

at com.ibm.ws.webcontainer.webapp.WebAppImpl.initialize(WebAppImpl.java:414)

at com.ibm.ws.webcontainer.webapp.WebGroupImpl.addWebApplication(WebGroupImpl.java:88)

at com.ibm.ws.webcontainer.VirtualHostImpl.addWebApplication(VirtualHostImpl.java:170)

at com.ibm.ws.webcontainer.WSWebContainer.addWebApp(WSWebContainer.java:904)

at com.ibm.ws.webcontainer.WSWebContainer.addWebApplication(WSWebContainer.java:789)

at com.ibm.ws.webcontainer.component.WebContainerImpl.install(WebContainerImpl.java:427)

at com.ibm.ws.webcontainer.component.WebContainerImpl.start(WebContainerImpl.java:719)

at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1178)

at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1383)

at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:639)

at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:972)

at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:777)

at com.ibm.ws.runtime.component.ApplicationMgrImpl$5.run(ApplicationMgrImpl.java:2197)

at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5486)

at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5612)

at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255)

at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2202)

at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:435)

at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)

at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:378)

at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:126)

at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:984)

at com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:502)

Correct Answer
December 3, 2019 - 12:43pm

Hi,

This issue was resolved after stopping all the running server which was pointing to same database schema and then restarting them one by one.

Comments

Keep up to date on this post and subscribe to comments

Pega
November 29, 2019 - 11:20am

Hi Anand,

You are getting this error because all the nodes are unable to communicate with each other.Are they in different subnet?How many nodes are present ?Please check the connectivity by using ping command.

Please share these details I will suggest you next action items based on these details.

Thanks,
Abhinav

December 3, 2019 - 12:43pm

Hi,

This issue was resolved after stopping all the running server which was pointing to same database schema and then restarting them one by one.