Connecting 3rd party software to Kafka

[paulocabrita-ionos]

Hi.

I'm trying to connect an open source project which has a Kafka Client configuration to a Kafka deployed using the stackablectl. But so far, no success. When I access the log from the simple-kafka pod, I have several lines like the following:

• kafka [2023-06-13 15:20:41,553] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1001] Failed authentication with /10.214.120.253 (channelId=10.214.120.218:9093-10.214.120.253:37898-18911) (SSL handshake failed) (org.apache.kafka.common.network.Selector) │

Here is what I did:

• stackablectl release install -i zookeeper -i kafka -i commons -i secret 23.4
• the yaml for the zookeeper is very simple:

---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperCluster
metadata:
  name: simple-zk
spec:
  image:
    productVersion: 3.8.0
    stackableVersion: 23.4.0
  servers:
    roleGroups:
      default:
        replicas: 1

• also a simple one for kafka:

---
apiVersion: kafka.stackable.tech/v1alpha1
kind: KafkaCluster
metadata:
  name: simple-kafka
spec:
  image:
    productVersion: 3.3.1
    stackableVersion: 23.4.0
  clusterConfig:
    zookeeperConfigMapName: simple-zk
  brokers:
    roleGroups:
      default:
        replicas: 1

• Everything is up and running fine;
• I configure the 3rd party with commonClientConfig."bootstrap\.servers"=simple-kafka:9093 and I have the failed authentication;

I know that I have a missing configuration for my client but I don't know what to add.

Some questions:

• what is the default authentication mechanism for my simple-kafka? Plaintext? SSL? TLS?
• how can I connect my 3dr party to the Stackable Kafka?

I think my issue is related to this one: https://github.com/orgs/stackabletech/discussions/24.

Thanks.

[Jimvin]

The default security protocol is SSL, so TLS is enabled without authentication. A bare minimal configuration needs to set security.protocol and configure the certificate trust store.

security.protocol=SSL
ssl.truststore.location=/stackable/tls_internal/truststore.p12
ssl.truststore.type=PKCS12
ssl.truststore.password=changeit

The path shown above is where the trust store is located by default in the Kafka containers. The following example show how you can get the client config onto a running container and run the kafka-topics.sh command.

kubectl exec -i -t simple-kafka-broker-brokers-0 -c kafka --  echo -e \
"security.protocol=SSL\nssl.truststore.location=/stackable/tls_internal/truststore.p12\nssl.truststore.type=PKCS12\nssl.truststore.password=changeit" > /stackable/config.properties

kubectl exec -i -t simple-kafka-broker-brokers-0 -c kafka -- bin/kafka-topics.sh \
  --bootstrap-server simple-kafka-broker-brokers.default.svc.cluster.local:9093 \
  --list --command-config /stackable/config.properties

[Jimvin]

You also asked about external access to Kafka. The longer term solution we're working on is the Stackable Listener Operator, but for now the brokers advertise the nodeport address. External access should work connecting to the nodeport, if you run into any difficulties report back and we can take a further look at this.

For reference here's a link to a relevant issue open on the Listener Operator - stackabletech/listener-operator#20

[paulocabrita-ionos]

First of all, thanks for your answer.

I'm still struggling to configure the client.

So if I understood correctly, I have to copy the trust store ot the client, right?

When I do the deployment of the Stackable Kafka, can I change the client authentication? For instance I want to use:

• SASL_TLS;
• user/password

Thanks