Question

DEV Slowness due to Pega Thread Dump

We are facing DEV slowness due to Java thread dump , unable to find the exact issue for thread dump. Most of the time i see below exception in logs. (Attached the pega rules.log)

( web.impl.WebStandardImpl) WARN - I/O Exception occurred while writing or flushing the stream

org.apache.catalina.connector.ClientAbortException: java.io.IOException: An established connection was aborted by the software in your host machine

at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:356) ~[catalina.jar:8.5.30]

at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:815) ~[catalina.jar:8.5.30]

at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:310) ~[catalina.jar:8.5.30]

at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:284) ~[catalina.jar:8.5.30]

at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118) ~[catalina.jar:8.5.30]

at com.pega.jcraft.jzlib.DeflaterOutputStream.flush(DeflaterOutputStream.java:159) ~[prjzlib-1.1.0.jar:?]

at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297) ~[?:1.8.0_202]

at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141) ~[?:1.8.0_202]

at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) ~[?:1.8.0_202]

at com.pega.pegarules.web.impl.DirectStreamWriter.flush(DirectStreamWriter.java:204) ~[prwebj2ee.jar:?]

at com.pega.pegarules.web.impl.DirectStreamWriter.flush(DirectStreamWriter.java:197) ~[prwebj2ee.jar:?]

at com.pega.pegarules.web.impl.WebStandardImpl.sendOutput(WebStandardImpl.java:999) ~[prwebj2ee.jar:?]

at com.pega.pegarules.web.impl.WebStandardImpl.doPost(WebStandardImpl.java:428) ~[prwebj2ee.jar:?]

at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source) ~[?:?]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_202]

at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_202]

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethod(PRBootstrap.java:370) ~[prbootstrap-7.3.1-218.jar:7.3.1-218]

at com.pega.pegarules.internal.bootstrap.PRBootstrap.invokeMethodPropagatingThrowable(PRBootstrap.java:411) ~[prbootstrap-7.3.1-218.jar:7.3.1-218]

at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethodPropagatingThrowable(AppServerBridgeToPega.java:224) ~[prbootstrap-api-7.3.1-218.jar:7.3.1-218]

at com.pega.pegarules.boot.internal.extbridge.AppServerBridgeToPega.invokeMethod(AppServerBridgeToPega.java:273) ~[prbootstrap-api-7.3.1-218.jar:7.3.1-218]

at com.pega.pegarules.internal.web.servlet.WebStandardBoot.doPost(WebStandardBoot.java:129) ~[prbootstrap-api-7.3.1-218.jar:7.3.1-218]

at com.pega.pegarules.internal.web.servlet.WebStandardBoot.doGet(WebStandardBoot.java:100) ~[prbootstrap-api-7.3.1-218.jar:7.3.1-218]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:635) ~[servlet-api.jar:?]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) ~[servlet-api.jar:?]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[catalina.jar:8.5.30]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[catalina.jar:8.5.30]

at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) ~[tomcat-websocket.jar:8.5.30]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[catalina.jar:8.5.30]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[catalina.jar:8.5.30]

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) ~[catalina.jar:8.5.30]

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) ~[catalina.jar:8.5.30]

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496) ~[catalina.jar:8.5.30]

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) ~[catalina.jar:8.5.30]

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) ~[catalina.jar:8.5.30]

at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) ~[catalina.jar:8.5.30]

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) ~[catalina.jar:8.5.30]

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) ~[catalina.jar:8.5.30]

at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803) ~[tomcat-coyote.jar:8.5.30]

at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) ~[tomcat-coyote.jar:8.5.30]

at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790) ~[tomcat-coyote.jar:8.5.30]

at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459) ~[tomcat-coyote.jar:8.5.30]

at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-coyote.jar:8.5.30]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_202]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_202]

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-util.jar:8.5.30]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]

Caused by: java.io.IOException: An established connection was aborted by the software in your host machine

at sun.nio.ch.SocketDispatcher.write0(Native Method) ~[?:1.8.0_202]

at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:51) ~[?:1.8.0_202]

at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:1.8.0_202]

at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[?:1.8.0_202]

at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[?:1.8.0_202]

at org.apache.tomcat.util.net.SecureNioChannel.flush(SecureNioChannel.java:144) ~[tomcat-coyote.jar:8.5.30]

at org.apache.tomcat.util.net.SecureNioChannel.write(SecureNioChannel.java:659) ~[tomcat-coyote.jar:8.5.30]

at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101) ~[tomcat-coyote.jar:8.5.30]

at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:157) ~[tomcat-coyote.jar:8.5.30]

at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1267) ~[tomcat-coyote.jar:8.5.30]

at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:670) ~[tomcat-coyote.jar:8.5.30]

at org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:450) ~[tomcat-coyote.jar:8.5.30]

at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:388) ~[tomcat-coyote.jar:8.5.30]

at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.doWrite(Http11OutputBuffer.java:623) ~[tomcat-coyote.jar:8.5.30]

at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:121) ~[tomcat-coyote.jar:8.5.30]

at org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:225) ~[tomcat-coyote.jar:8.5.30]

at org.apache.coyote.Response.doWrite(Response.java:541) ~[tomcat-coyote.jar:8.5.30]

at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:351) ~[catalina.jar:8.5.30]

... 45 more

2019-05-13 10:25:54,266 [.100.156-443-exec-11] [ ] [ ] [ ] ( web.impl.WebStandardImpl) WARN - Request Information is --

RequestURL = https://MyApp-api.u5.MyOrg.org/prweb/JvPY5BF13j1lYt-jhEhjGQ%5B%5B*/!@61b0a80bb0bc177e29478d1dbfa47cfe!TABTHREAD1

QueryString = pyActivity=%40baseclass.doUIAction&action=openRuleSpecific&openHandle=RULE-ADMIN-PRODUCT%20MyApp_R1_3%2001.01.01%20%2320190503T065925.210%20GMT&Format=harness&contentID=3abe2adb-1d8e-085c-78dd-322a04f9fb6a&dynamicContainerID=de12ecd7-b2b3-4849-bff1-ee2797014513&tabIndex=3&prevContentID=c4641d34-6e86-2d6e-ee49-7e59315da0bd&prevRecordkey=Product&portalThreadName=STANDARD&portalName=Developer&pzHarnessID=HIDA9B83A23E6E10D2DAE834C47137F1AEE&ClientInt=Start

Requestor Id = HEHNUOLAHKYHWTT5KUNOVP6IL70FG8H5K

Remote Address = 127.0.0.1

***Edited by Moderator: Pallavi to update platform capability tags & mask IP address***

Comments

Keep up to date on this post and subscribe to comments

May 13, 2019 - 8:18am

looks like you are exporting MyApp_R1_3 file, can you please let us know the size of file being exported?

 

2019-05-13 09:54:02,043 [7.100.156-443-exec-3] [TABTHREAD2] [                    ] [   MyApp_Dev:01.01.01] (         internal.util.MoveLog) INFO  MyApp-api.u5.MyOrg.org|127.0.0.1 TRIKUL - Operation Status: Constructing export file -- please wait...
2019-05-13 09:54:02,046 [7.100.156-443-exec-3] [TABTHREAD2] [                    ] [   MyApp_Dev:01.01.01] (         internal.util.MoveLog) INFO  MyApp-api.u5.MyOrg.org|127.0.0.1 TRIKUL - Operation Status: Constructing export file -- please wait....
2019-05-13 09:54:02,047 [7.100.156-443-exec-3] [TABTHREAD2] [                    ] [   MyApp_Dev:01.01.01] (         internal.util.MoveLog) INFO  MyApp-api.u5.MyOrg.org|127.0.0.1 TRIKUL - Operation Status: Constructing export file -- please wait.....
2019-05-13 09:54:02,047 [7.100.156-443-exec-3] [TABTHREAD2] [                    ] [   MyApp_Dev:01.01.01] (         internal.util.MoveLog) INFO  MyApp-api.u5.MyOrg.org|127.0.0.1 TRIKUL - Operation Status: Constructing export file -- please wait.....

***Edited by moderator: Lochan to mask IP address***

May 14, 2019 - 5:21am
Response to ARVINDMALAV

I don't think so it's an issue with exporting product RAP, However, we exported the product RAP file which has around 50 rules.

May 15, 2019 - 9:40am

Most of the DB alerts (Pega0005) are coming from D_PZWARNINGSINTRODUCED.  I see an article for the same issue which is suggesting to make the local change but still does it need for 7.3 version as well.

https://community.pega.com/support/support-articles/pega0005-alert-database-operations-ootb-tables

 

pyDefinitionKey=RULE-DECLARE-PAGES D_PZWARNINGSINTRODUCED #20140730T063130.936 GMT;pyClassName=Data-Rule-Summary;pyReportName=pzWarningsIntroduced;pyRunType=FullResolve;ExcludeRuleset=;pyPageName=pyDataPage;pyRuleSet=Pega-Desktop;pyReportClass=Data-Rule-Summary;pyPageStructure=list;CopyAlways=true;operatorID=;

May 16, 2019 - 2:41am
Response to Brahmesh@

Any input on this ?

May 20, 2019 - 3:56pm

Hi Brahmesh,

The guardrail warnings report is still a part of Designer Studio in 7.3.  Can you include the ALERT's log file that corresponds to the Pega-Rules log file so the requestor details can be matched to the thread dumps showing calls to the DB?