Question

Kafka Config

We’re upgrading from 7.4 with no kafka to 8.3

which implements kafka. There appears to be no reference to configuring kafka in the upgrade guide

other than saying

• Kafka stream nodes — Port 9092

• Stream REST service — Port 7003

I have two servers ( nodes ) and each node has two tomcat JVMs accessing the same QA database.

I just want to know

  1. Each JVM instance on one node needs a different Stream Rest Service Port ?
    instance 1 uses port 7003 and instance two uses port 7004 ?

  2. can they communicate with the same kafka instance on the node listening on port 9092 or do they
    each need a separate kafka process.

  3. If they need a separate Kafka process; how do I create and configure it ; The upgrade actually configured and started it automatically.

***Edited by Moderator Marissa to update SR Details***

Group Tags

Comments

Keep up to date on this post and subscribe to comments

December 4, 2019 - 8:44pm

Hi,

So it works like this.

If you have one machine and multiple nodes (tomcat) running you would need to set different ports.

You can check below post :

https://community.pega.com/knowledgebase/articles/advanced-configurations-stream-service

It has ​Changing the broker, keeper, and JMX port configuration details. This should help you sort things out

December 4, 2019 - 9:04pm

1. If both the JVM instances are hosted in the same machine and you have defined both the nodes as stream nodes, then you will have to change the port details (Settings to do these changes has been mentioned by Shubham in the above comment).

2. For each node there will be a separate kafka process and the kafka data will be stored in the application server.

3. The server will be taking care of creating the kafka process.

December 11, 2019 - 6:08am

Need a clarification :

I have two servers

Each server is running two JVM instances;  making a four node cluster

Does the cluster need to have a Stream Node on each server ; or is one stream node in the cluster sufficient ?

If a stream service is running on both servers do they sync with each other ?

If the cluster only has one stream service process then all nodes need to have the same pyPort value ?

If the cluster only has one stream service process; than that system must be started first and shutdown last ?