Question

Hazelcast Issues Pega 8

We are observing a lot of issues related to hazelcast in our environment.

Sometime hazelcast engine goes down still the prsysstatus node tables gets updated. PegaCluster log stops getting updated.

2019-08-17 04:25:01,920 [331.cached.thread-46] [ ] [ ] ( hazelcast.instance.Node) INFO - [xxxxx]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] Shutting down node engine...

2019-08-17 04:25:05,320 [331.cached.thread-46] [ ] [ ] (zelcast.instance.NodeExtension) INFO - [10.73.198.36]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] Destroying node NodeExtension.

2019-08-17 04:25:05,320 [331.cached.thread-46] [ ] [ ] ( hazelcast.instance.Node) INFO - [xxxxx]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] Hazelcast Shutdown is completed in 106401 ms.

2019-08-17 04:25:05,320 [331.cached.thread-46] [ ] [ ] (azelcast.core.LifecycleService) INFO - [xxxxx]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] [xxxxx]:5706 is SHUTDOWN

2019-08-17 04:25:05,321 [331.cached.thread-46] [ ] [ ] (azelcast.core.LifecycleService) INFO - [xxxxx]:5706 [a948064bf1085f5cb78ac1448bc1f331] [3.10] [xxxxx]:5706 is MERGE_FAILED

We get Pega0090 alerts.

A lot of times there are communication failures between members.

Reason: Exception in Connection[id=602, /yyyyy:5707->/yyyyy:55601, endpoint=[yyyyyy]:5706, alive=true, type=MEMBER], thread=hz._hzInstance_1_a948064bf1085f5cb78ac1448bc1f331.IO.thread-in-1][STACK][java.io.IOException: Connection reset by peer<CR> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)<CR> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)<CR> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)<CR> at sun.nio.ch.IOUtil.read(IOUtil.java:197)<CR> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)<CR> at com.hazelcast.nio.ssl.SSLChannel.read(SSLChannel.java:189)<CR> at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:129)<CR> at com.hazelcast.internal.networking.nio.NioThread.handleSelectionKey(NioThread.java:383)<CR> at com.hazelcast.internal.networking.nio.NioThread.handleSelectionKeys(NioThread.java:368)<CR> at com.hazelcast.internal.networking.nio.NioThread.selectLoopWithFix(NioThread.java:289)<CR> at com.hazelcast.internal.networking.nio.NioThread.run(NioThread.java:224)

Sub clusters getting formed

2019-08-17 04:23:54,941 [448bc1f331.migration] [ ] [ ] (artition.impl.MigrationManager) INFO - [yyyyyy]:5711 [a948064bf1085f5cb78ac1448bc1f331] [3.10] Partition balance is ok, no need to re-partition cluster data...

2019-08-17 04:23:57,935 [331.cached.thread-17] [ ] [ ] (nternal.cluster.ClusterService) INFO - [yyyyyy]:5711 [a948064bf1085f5cb78ac1448bc1f331] [3.10]

Members {size:1, ver:201} [

Member [yyyyy]:5711 - 54c532a9-36c3-4968-91c7-ae591b16648c this

]

2019-08-17 04:23:57,935 [331.cached.thread-17] [ ] [ ] (cluster.impl.MembershipManager) INFO - [xxxxxx]:5711 [a948064bf1085f5cb78ac1448bc1f331] [3.10] Mastership is claimed with: MembersView{version=201, members=[MemberInfo{address=[xxxxxxx]:5711, uuid=54c532a9-36c3-4968-91c7-ae591b16648c, liteMember=false, memberListJoinVersion=200}]}

Anyone has a cluster size of around 60?

***Edited by Moderator Marissa to update platform capability tags; update SR Details****

Group Tags

Comments

Keep up to date on this post and subscribe to comments

August 19, 2019 - 5:26pm

Hello Faisal,

It is better if you can create custom hazelcast groups... another reg: port communication, seems b/w servers ports are not opened properly.

Thanks

 

August 22, 2019 - 10:50am

Thanks for your reply Sreecharan. What benefits will we get from custom hazelcast group?