Question

Unable to connect to Kafka client using Data-Admin-Kafka instanc

Heloo,

Anyone configured Kafka client in pega 7.3.1.We are getting below error when we tried to connect to Kafka client using kafka Data-Admin-kafka instance.

Error connecting to Kafka. Failed to construct kafka consumer. java.io.IOException: Configuration

Steps followed:

1.Created a Data-Admin-kafka instance

2.Provided host name, port, keystore, truststore, pwd

3.Use authentication Kerberos.

4.JAAS config and krb5.config files passed in the java argument

Error Message

Connection failed

Error connecting to Kafka. Failed to construct kafka consumer. java.io.IOException: Configuration Error:
Line 1: expected [{], found [logging]. Configuration Error:
Line 1: expected [{], found [logging]

thanks in advance.

***Edited by Moderator Marissa to update platform capability tags****

Correct Answer
February 21, 2019 - 9:22am

Hi Saravan,

By passing the JVM args, its picking the JAAS file. But still getting the error "Error connecting to Kafka. Failed to construct kafka consumer. javax.security.auth.login.LoginException: Unable to obtain password from user.Unable to obtain password from user"

Please help me on this..

 

Comments

Keep up to date on this post and subscribe to comments

February 20, 2019 - 5:16am

Hi ,

While creating the Kafka conf instance, we are getting the "time out expires while connecting to kafka metadata" error. We are using kerberos authentication in our organization. we have placed the Kerberos file and keytab file in server and also added those file paths,principal names in standalone.xml file. 

Please let us any other things to be done for the kafka connection.

Regards,

Roja

February 20, 2019 - 8:30am
Response to RojaLakshmiM

Hi RojaLakshmiM,

Can you provide the following details to help you out?

  • Application Server
  • changes made in JASS.conf file

February 20, 2019 - 9:38am
Response to MSaravanan76

hi  Saravanan,

   We are using Pega 7.4 version.

    We have made the below changes in the JAAS file:

KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
doNotPrompt=true
useTicketCache=true
principal="bdplantview" ##<--Changed  to our Principal
useKeyTab=true
serviceName="kafka"
keyTab="/home/araghava/kafka/bdplantview.headless.keytab" ##<- Updated this with the location of your keytab file
client=true;
};

February 21, 2019 - 1:30am
Response to RojaLakshmiM

Hi Roja,

Everything looks good except this principal="bdplantview".

In our case, I gave it like this "kafkauser@ABC.XYZ.COM".

In this, ABC.XYZ.COM is our default realm which I got it from the KRB5.conf file.

Please check your krb5.conf file and look for an entry like default_realm.

Ex: If the realm is set as "default_realm = ABC.XYZ.COM" then you need to set your principal name as principal="bdplantview@ABC.XYZ.COM"

Note: 
If you make any changes in JASS file then server requires a restart to make it effective.

February 21, 2019 - 2:45am
Response to MSaravanan76

Hi Saravan,

Thanks for your reponse.

We had given the Principal name as "ABC@HDP_XYZ.COM" where "HDP_DEV.XYZ.COM" is the default_realm in KRB5.conf file, still we are not able to connect to the Kafka, getting the Time Out expires error.  Also,Is the JAAS file configuration necessary to set up a connection? Though the JAAS file is placed in the location, on the kAFkA config rule form we see "No JAAS configuration ruleset". Please let me know any other changes to be done from the app server side for establishing the connection.

 

 

 

 

February 21, 2019 - 3:06am
Response to RojaLakshmiM

yes, JASS configuration is required. It seems your JASS config file path is not added to JVM arguments.

You need to ask your Application server admin to add the following into the JVM args.

-Djava.security.auth.login.config="/<FOLDER NAME>/jass.conf"

Also, you need to add your krb5.conf file path into JVM args as well.

-Djava.security.krb5.conf=/<FOLDER NAME>/krb5.conf

Once done, you need to restart the server. After the restart, you can verify that from SMA --> System Management

Hope, it helps.

 

February 21, 2019 - 4:24am
Response to MSaravanan76

After adding JAAS Config file and Krb5 file to JVM arguments , connection is successful.

February 21, 2019 - 6:24am
Response to Sivajyothi

Hi Sivajyothi,

Please tell me where exactly the JVM args should be added. f

February 21, 2019 - 6:04am
Response to MSaravanan76

Please tell me exactly where these JVM args are added. In standalone.sh or start up scripts of The JVM.

February 21, 2019 - 9:22am
Response to RojaLakshmiM

Hi Saravan,

By passing the JVM args, its picking the JAAS file. But still getting the error "Error connecting to Kafka. Failed to construct kafka consumer. javax.security.auth.login.LoginException: Unable to obtain password from user.Unable to obtain password from user"

Please help me on this..

 

February 22, 2019 - 2:11am
Response to RojaLakshmiM

Hi Roja,

It seems the keytab file is not having the correct file permission. It should have 664. Please set and try again.

April 28, 2019 - 8:32am
Response to MSaravanan76

Hi Sarvanan,

We have successfully established the connection in our Dev env. But we are getting error while trying to connect to QA Kafka cluster from our QA env. Please find the attached screenshot and suggest what changes are required to establish the connection.

All the configuration like placing keytab,kerberos,Jaas file and changes to standalone file are done in QA server, still facing the issue.

Regards,

Roja

May 8, 2019 - 4:41am
Response to RojaLakshmiM

Hi Roja,

Make sure that KAFKA user is added to QA Kerberos database.

Saravanan M

February 22, 2019 - 2:39am
Response to RojaLakshmiM

Hi Saravan,

Thanks for the help. We have successfully established the connection by following the steps that you have provided.

Regards,

Roja

 

July 10, 2019 - 4:19am

Hi RajyaLakshmi,

Can you please explain me the detailed steps to implement Kafka Connectivity in Pega .

Thankyou