Question

Unable to start Stream Node

Hi,

We are using Pega 8.1.5 version which is configured on websphere 8.5.5 and Oracle 12.2.

Recent after the restart in our Production environment, we can’t get a STREAM connected due to below error. Any suggestion?

Node ID

0f5a02a02ad496b149108a84f3f4872e

Error message

StreamServiceException: Unable to start kafka broker, broker hangs in state: RecoveringFromUncleanShutdown
[2019-09-11 12:05:12,819] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2019-09-11 12:05:13,435] INFO starting (kafka.server.KafkaServer)
[2019-09-11 12:05:13,436] INFO Connecting to zookeeper on 127.0.0.1:2181 (kafka.server.KafkaServer)
[2019-09-11 12:05:13,464] INFO [ZooKeeperClient] Initializing a new session to 127.0.0.1:2181. (kafka.zookeeper.ZooKeeperClient)
[2019-09-11 12:05:13,481] INFO Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT (org.apache.zookeeper.ZooKeeper)
[2019-09-11 12:05:13,481] INFO Client environment:host.name=10.8.30.195 (org.apache.zookeeper.ZooKeeper)
[2019-09-11 12:05:13,481] INFO Client environment:java.version=1.8.0_191 (org.apache.zookeeper.ZooKeeper)
[2019-09-11 12:05:13,481] INFO Client environment:java.vendor=IBM Corporation (org.apache.zookeeper.ZooKeeper)
[2019-09-11 12:05:13,481] INFO Client environment:java.home=/sb/sys/was/L1/appserver/java/jre (org.apache.zookeeper.ZooKeeper)
[2019-09-11 12:05:13,481] INFO Client environment:java.class.path=/sb/env/app/S10/temp/kafka-1.1.0.3/libs/aopalliance-repackaged-2.5.0-b32.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/log4j-1.2.17.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/kafka-tools-1.1.0.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/jetty-http-9.2.24.v20180105.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/hk2-api-2.5.0-b32.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/javax.inject-1.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/jackson-module-jaxb-annotations-2.9.4.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/guava-20.0.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/argparse4j-0.7.0.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/jersey-guava-2.25.1.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/jetty-servlet-9.2.24.v20180105.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/connect-file-1.1.0.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/rocksdbjni-5.7.3.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/jetty-security-9.2.24.v20180105.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/javax.ws.rs-api-2.0.1.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/jetty-servlets-9.2.24.v20180105.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/maven-artifact-3.5.2.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/jersey-container-servlet-core-2.25.1.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/snappy-java-1.1.7.1.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/kafka-streams-examples-1.1.0.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/connect-transforms-1.1.0.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/osgi-resource-locator-1.0.1.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/slf4j-api-1.7.25.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/hk2-locator-2.5.0-b32.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/commons-lang3-3.5.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/jersey-client-2.25.1.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/hk2-utils-2.5.0-b32.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/plexus-utils-3.1.0.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/jackson-core-2.9.4.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/javax.annotation-api-1.2.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/jackson-databind-2.9.4.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/kafka_2.11-1.1.0.3.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/jetty-util-9.2.24.v20180105.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/jackson-annotations-2.9.4.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/zkclient-0.10.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/jersey-server-2.25.1.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/kafka-clients-1.1.0.3.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/zookeeper-3.4.10.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/scala-reflect-2.11.12.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/connect-api-1.1.0.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/jersey-media-jaxb-2.25.1.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/connect-json-1.1.0.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/jackson-jaxrs-json-provider-2.9.4.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/lz4-java-1.4.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/jetty-continuation-9.2.24.v20180105.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/javassist-3.20.0-GA.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/slf4j-log4j12-1.7.25.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/kafka-log4j-appender-1.1.0.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/jetty-io-9.2.24.v20180105.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/validation-api-1.1.0.Final.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/jopt-simple-5.0.4.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/jackson-jaxrs-base-2.9.4.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/javassist-3.21.0-GA.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/javax.servlet-api-3.1.0.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/kafka-streams-test-utils-1.1.0.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/jersey-common-2.25.1.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/jetty-client-9.2.24.v20180105.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/jetty-server-9.2.24.v20180105.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/kafka-streams-1.1.0.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/javax.inject-2.5.0-b32.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/reflections-0.9.11.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/scala-library-2.11.12.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/connect-runtime-1.1.0.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/metrics-core-2.2.0.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/scala-logging_2.11-3.7.2.jar:/sb/env/app/S10/temp/kafka-1.1.0.3/libs/jersey-container-servlet-2.25.1.jar (org.apache.zookeeper.ZooKeeper)
[2019-09-11 12:05:13,482] INFO Client environment:java.library.path=/sb/sys/was/L1/appserver/java/jre/lib/amd64/compressedrefs:/sb/sys/was/L1/appserver/java/jre/lib/amd64:/sb/sys/was/L1/appserver/lib/native/linux/x86_64/:/sb/sys/was/L1/appserver/bin:/sb/sys/was/L1/appserver/nulldllsdir:/usr/lib64:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2019-09-11 12:05:13,482] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2019-09-11 12:05:13,482] INFO Client environment:java.compiler= (org.apache.zookeeper.ZooKeeper)
[2019-09-11 12:05:13,482] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2019-09-11 12:05:13,482] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2019-09-11 12:05:13,482] INFO Client environment:os.version=3.10.0-957.21.3.el7.x86_64 (org.apache.zookeeper.ZooKeeper)
[2019-09-11 12:05:13,482] INFO Client environment:user.name=p986wasuser (org.apache.zookeeper.ZooKeeper)
[2019-09-11 12:05:13,482] INFO Client environment:user.home=/export/home/p986wasuser (org.apache.zookeeper.ZooKeeper)
[2019-09-11 12:05:13,482] INFO Client environment:user.dir=/sb/env/app/S10/temp/kafka-1.1.0.3/bin (org.apache.zookeeper.ZooKeeper)
[2019-09-11 12:05:13,485] INFO Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=30000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@60389438 (org.apache.zookeeper.ZooKeeper)
[2019-09-11 12:05:13,514] INFO Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2019-09-11 12:05:13,516] INFO [ZooKeeperClient] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2019-09-11 12:05:13,519] INFO Socket connection established to 127.0.0.1/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2019-09-11 12:05:13,667] INFO Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0xffffffffb7a90001, negotiated timeout = 30000 (org.apache.zookeeper.ClientCnxn)
[2019-09-11 12:05:13,674] INFO [ZooKeeperClient] Connected. (kafka.zookeeper.ZooKeeperClient)
[2019-09-11 12:05:14,287] INFO Cluster ID = 6LpSc_cWTYig405gRlSBIA (kafka.server.KafkaServer)
[2019-09-11 12:05:14,374] INFO KafkaConfig values:

Correct Answer
September 18, 2019 - 8:51am

When you shutdown your server, did you check if the kafka java process was shutdown as well? If yes, will the error happen again during startup? Sometimes, the kafka process did not get shutdown for various reasons.

Comments

Keep up to date on this post and subscribe to comments

September 18, 2019 - 8:51am

When you shutdown your server, did you check if the kafka java process was shutdown as well? If yes, will the error happen again during startup? Sometimes, the kafka process did not get shutdown for various reasons.