Question

Hung threads on one jvm while starting websphere application server with pega7.1.9 installed on it

Pega7.1.9 installed on webpshere 8.5.5.7 cluster with two jvms, MSSQL is the DB.

While starting the application server only one jvm reporting these hung threads and taking very unusual time to comeup when compared to the other jvm.

Can someone please tell me why?

[11/10/16 16:36:22:654 CST] 000000e7 ThreadMonitor W   WSVR0605W: Thread "server.startup : 2" (00000086) has been active for 707114 milliseconds and may be hung.  There is/are 1 thread(s) in total in the server that may be hung.
        at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:985)
        at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:776)
        at com.ibm.ws.runtime.component.ApplicationMgrImpl$5.run(ApplicationMgrImpl.java:2195)
        at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5477)
        at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5603)
        at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255)
        at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2200)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:435)
        at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:378)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:126)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:984)
        at com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:502)
        at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1881)
[11/10/16 16:36:22:702 CST] 000000e7 ThreadMonitor W   WSVR0605W: Thread "server.startup : 1" (00000085) has been active for 707118 milliseconds and may be hung.  There is/are 2 thread(s) in total in the server that may be hung.
        at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:985)
        at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:776)
        at com.ibm.ws.runtime.component.ApplicationMgrImpl$5.run(ApplicationMgrImpl.java:2195)
        at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5477)
        at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5603)
        at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255)
        at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2200)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:435)
        at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:378)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:126)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:984)
        at com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:502)
        at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1881)
[11/10/16 16:36:22:757 CST] 000000e7 ThreadMonitor W   WSVR0605W: Thread "brksvx247d.ad.navistar.com" (00000084) has been active for 707113 milliseconds and may be hung.  There is/are 3 thread(s) in total in the server that may be hung.
        at org.eclipse.jdt.internal.compiler.flow.UnconditionalFlowInfo.copy(UnconditionalFlowInfo.java:650)
        at org.eclipse.jdt.internal.compiler.ast.MessageSend.analyseCode(MessageSend.java:188)
        at org.eclipse.jdt.internal.compiler.ast.UnaryExpression.analyseCode(UnaryExpression.java:41)
        at org.eclipse.jdt.internal.compiler.ast.IfStatement.analyseCode(IfStatement.java:62)
        at org.eclipse.jdt.internal.compiler.ast.Block.analyseCode(Block.java:48)
        at org.eclipse.jdt.internal.compiler.ast.IfStatement.analyseCode(IfStatement.java:105)
        at org.eclipse.jdt.internal.compiler.ast.Block.analyseCode(Block.java:48)
        at org.eclipse.jdt.internal.compiler.ast.IfStatement.analyseCode(IfStatement.java:133)
        at org.eclipse.jdt.internal.compiler.ast.MethodDeclaration.analyseCode(MethodDeclaration.java:124)
        at org.eclipse.jdt.internal.compiler.ast.TypeDeclaration.internalAnalyseCode(TypeDeclaration.java:737)
        at org.eclipse.jdt.internal.compiler.ast.TypeDeclaration.analyseCode(TypeDeclaration.java:261)
        at org.eclipse.jdt.internal.compiler.ast.CompilationUnitDeclaration.analyseCode(CompilationUnitDeclaration.java:118)
        at org.eclipse.jdt.internal.compiler.Compiler.process(Compiler.java:776)
        at org.eclipse.jdt.internal.compiler.Compiler.compile(Compiler.java:470)
        at org.eclipse.jdt.internal.compiler.batch.Main.performCompilation(Main.java:4129)
        at org.eclipse.jdt.internal.compiler.tool.EclipseCompilerImpl.call(EclipseCompilerImpl.java:76)
        at org.eclipse.jdt.internal.compiler.tool.EclipseCompiler$1.call(EclipseCompiler.java:185)
        at com.pega.jsr199.Jsr199Bridge.compile_impl(Jsr199Bridge.java:277)
        at com.pega.jsr199.Jsr199Bridge.compile(Jsr199Bridge.java:209)
        at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
        at java.lang.reflect.Method.invoke(Method.java:611)
        at com.pega.pegarules.generation.internal.compiler.PRCompilerPoolImpl$Eclipse34BridgeClassInfo.invokeCompiler(PRCompilerPoolImpl.java:182)
        at com.pega.pegarules.generation.internal.compiler.PRCompilerEclipse._compile_privact(PRCompilerEclipse.java:581)
        at com.pega.pegarules.generation.internal.compiler.PRCompilerEclipse.compile(PRCompilerEclipse.java:503)
        at com.pega.pegarules.generation.internal.library.LibraryFunctionUtilityImpl.compileLibraryAndFunctions(LibraryFunctionUtilityImpl.java:2711)
        at com.pega.pegarules.generation.internal.library.LibraryFunctionUtilityImpl.generateLibrary(LibraryFunctionUtilityImpl.java:1105)
        at com.pega.pegarules.generation.internal.library.LibraryFunctionUtilityImpl.extractAll(LibraryFunctionUtilityImpl.java:452)
        at com.pega.pegarules.session.internal.mgmt.PRNodeImpl.handleExtraction(PRNodeImpl.java:1603)
        at com.pega.pegarules.session.internal.mgmt.PREnvironment.finishInit(PREnvironment.java:509)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
        at java.lang.reflect.Method.invoke(Method.java:611)
        at com.pega.pegarules.session.internal.PRSessionProviderImpl.performTargetActionWithLock(PRSessionProviderImpl.java:1270)
        at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:1008)
        at com.pega.pegarules.session.internal.PRSessionProviderImpl.doWithRequestorLocked(PRSessionProviderImpl.java:841)
        at com.pega.pegarules.session.internal.mgmt.PREnvironment.getThreadAndInitialize(PREnvironment.java:382)
        at com.pega.pegarules.session.internal.PRSessionProviderImpl.getThreadAndInitialize(PRSessionProviderImpl.java:1998)
        at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineStartup.initEngine(EngineStartup.java:664)
        at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl._initEngine_privact(EngineImpl.java:165)
        at com.pega.pegarules.session.internal.engineinterface.etier.impl.EngineImpl.doStartup(EngineImpl.java:138)
        at com.pega.pegarules.session.internal.engineinterface.etier.ejb.EngineBean.doStartup(EngineBean.java:121)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
        at java.lang.reflect.Method.invoke(Method.java:611)
        at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:370)
        at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:411)
        at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:223)
        at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingException(AppServerBridgeToPega.java:245)
        at com.pega.pegarules.internal.etier.ejb.EngineBeanBoot.doStartup(EngineBeanBoot.java:130)
        at com.pega.pegarules.internal.etier.interfaces.EJSLocalStatelessEngineBMT_f2439d86.doStartup(Unknown Source)
        at com.pega.pegarules.web.servlet.WebAppLifeCycleListener._contextInitialized_privact(WebAppLifeCycleListener.java:311)
        at com.pega.pegarules.web.servlet.WebAppLifeCycleListener.contextInitialized(WebAppLifeCycleListener.java:218)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
        at java.lang.reflect.Method.invoke(Method.java:611)
        at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:370)
        at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:411)
        at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:223)
        at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(AppServerBridgeToPega.java:272)
        at com.pega.pegarules.internal.web.servlet.WebAppLifeCycleListenerBoot.contextInitialized(WebAppLifeCycleListenerBoot.java:83)
        at com.ibm.ws.webcontainer.webapp.WebApp.notifyServletContextCreated(WebApp.java:1707)
        at com.ibm.ws.webcontainer.webapp.WebAppImpl.initialize(WebAppImpl.java:411)
        at com.ibm.ws.webcontainer.webapp.WebGroupImpl.addWebApplication(WebGroupImpl.java:88)
        at com.ibm.ws.webcontainer.VirtualHostImpl.addWebApplication(VirtualHostImpl.java:170)
        at com.ibm.ws.webcontainer.WSWebContainer.addWebApp(WSWebContainer.java:904)
        at com.ibm.ws.webcontainer.WSWebContainer.addWebApplication(WSWebContainer.java:789)
        at com.ibm.ws.webcontainer.component.WebContainerImpl.install(WebContainerImpl.java:427)
        at com.ibm.ws.webcontainer.component.WebContainerImpl.start(WebContainerImpl.java:719)
        at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:1177)
        at com.ibm.ws.runtime.component.DeployedApplicationImpl.fireDeployedObjectStart(DeployedApplicationImpl.java:1382)
        at com.ibm.ws.runtime.component.DeployedModuleImpl.start(DeployedModuleImpl.java:639)
        at com.ibm.ws.runtime.component.DeployedApplicationImpl.start(DeployedApplicationImpl.java:971)
        at com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:776)
        at com.ibm.ws.runtime.component.ApplicationMgrImpl$5.run(ApplicationMgrImpl.java:2195)
        at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5477)
        at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5603)
        at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255)
        at com.ibm.ws.runtime.component.ApplicationMgrImpl.start(ApplicationMgrImpl.java:2200)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:435)
        at com.ibm.ws.runtime.component.CompositionUnitImpl.start(CompositionUnitImpl.java:123)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.start(CompositionUnitMgrImpl.java:378)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl.access$500(CompositionUnitMgrImpl.java:126)
        at com.ibm.ws.runtime.component.CompositionUnitMgrImpl$CUInitializer.run(CompositionUnitMgrImpl.java:984)
        at com.ibm.wsspi.runtime.component.WsComponentImpl$_AsynchInitializer.run(WsComponentImpl.java:502)
        at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1881)

Thanks!

***Moderator Edit: Vidyaranjan| Updated Categories***

**Moderation Team has archived post**

This post has been archived for educational purposes. Contents and links will no longer be updated. If you have the same/similar question, please write a new post.

Comments

Keep up to date on this post and subscribe to comments

Pega
November 10, 2016 - 10:58pm

Please compare connection pool settings, heap space settings b/n both the jvms. It may give you a clue.

Refer http://www-01.ibm.com/support/docview.wss?uid=swg21448485

November 11, 2016 - 3:06pm

HI Gangababu,

I did check the heap size it is same on both jvm's, and the data source is created at cluster scope so whatever the conncetion pool properties is same for both.

Please let me know if you think of anything else.

 

Thank You!

December 14, 2016 - 9:05am

Verify the connection pool settings in websphere...