What is the purpose of starting dynamic listeners

[adiletkabylbekov]

I'm trying to understand kafka-proxy usage.
My brokers architecture is

172.28.132.4:9093 - kafka1
172.28.132.5:9093 - kafka2
172.28.132.6:9093 - kafka3

3 broker servers with TLS enabled

my kafka-proxy config is

      --bootstrap-server-mapping=172.28.132.4:9093,127.0.0.1:9091
      --bootstrap-server-mapping=172.28.132.5:9093,127.0.0.1:9092
      --bootstrap-server-mapping=172.28.132.6:9093,127.0.0.1:9093
      --tls-enable
      --tls-ca-chain-cert-file=ca.crt
      --tls-client-cert-file=cert.pem
      --tls-client-key-file=cert.key.pem
      --tls-insecure-skip-verify

In startup logs i identified of starting additional ports and addresses (Dynamic listeners ):

time="2022-12-26T11:45:30+03:00" level=info msg="Starting kafka-proxy version 0.3.3"
time="2022-12-26T11:45:30+03:00" level=info msg="Bootstrap server 172.28.132.4:9093 advertised as 127.0.0.1:9091"
time="2022-12-26T11:45:30+03:00" level=info msg="Bootstrap server 172.28.132.5:9093 advertised as 127.0.0.1:9092"
time="2022-12-26T11:45:30+03:00" level=info msg="Bootstrap server 172.28.132.6:9093 advertised as 127.0.0.1:9093"
time="2022-12-26T11:45:30+03:00" level=info msg="Listening on 127.0.0.1:9091 (127.0.0.1:9091) for remote 172.28.132.4:9093"
time="2022-12-26T11:45:30+03:00" level=info msg="Listening on 127.0.0.1:9092 (127.0.0.1:9092) for remote 172.28.132.5:9093"
time="2022-12-26T11:45:30+03:00" level=info msg="Listening on 127.0.0.1:9093 (127.0.0.1:9093) for remote 172.28.132.6:9093"
time="2022-12-26T11:45:30+03:00" level=info msg="Ready for new connections"
time="2022-12-26T11:45:30+03:00" level=info msg="New connection for 172.28.132.6:9093"
time="2022-12-26T11:45:30+03:00" level=info msg="Starting dynamic listener for broker kafka2.i:9093"
time="2022-12-26T11:45:30+03:00" level=info msg="Listening on 0.0.0.0:0 ([::]:38469) for remote kafka2.i:9093"
time="2022-12-26T11:45:30+03:00" level=info msg="Dynamic listener 0.0.0.0:38469 for broker kafka2.i:9093 advertised as 0.0.0.0:38469"
time="2022-12-26T11:45:30+03:00" level=info msg="Starting dynamic listener for broker kafka1.i:9093"
time="2022-12-26T11:45:30+03:00" level=info msg="Listening on 0.0.0.0:0 ([::]:33523) for remote kafka1.i:9093"
time="2022-12-26T11:45:30+03:00" level=info msg="Dynamic listener 0.0.0.0:33523 for broker kafka1.i:9093 advertised as 0.0.0.0:33523"
time="2022-12-26T11:45:30+03:00" level=info msg="Starting dynamic listener for broker kafka3.i:9093"
time="2022-12-26T11:45:30+03:00" level=info msg="Listening on 0.0.0.0:0 ([::]:38675) for remote kafka3.i:9093"
time="2022-12-26T11:45:30+03:00" level=info msg="Dynamic listener 0.0.0.0:38675 for broker kafka3.i:9093 advertised as 0.0.0.0:38675"
time="2022-12-26T11:45:30+03:00" level=info msg="New connection for 172.28.132.6:9093"
time="2022-12-26T11:46:00+03:00" level=info msg="New connection for kafka3.i:9093"
time="2022-12-26T11:53:12+03:00" level=info msg="New connection for kafka1.i:9093"
time="2022-12-26T11:56:00+03:00" level=info msg="Server 172.28.132.6:9093 closed connection"
time="2022-12-26T11:56:25+03:00" level=info msg="New connection for kafka1.i:9093"
time="2022-12-26T12:03:12+03:00" level=info msg="Server kafka1.i:9093 closed connection"
time="2022-12-26T12:06:25+03:00" level=info msg="Server 172.28.132.6:9093 closed connection"
time="2022-12-26T12:35:05+03:00" level=info msg="New connection for kafka1.i:9093"
time="2022-12-26T12:45:05+03:00" level=info msg="Server kafka1.i:9093 closed connection"
time="2022-12-26T12:51:37+03:00" level=info msg="New connection for kafka1.i:9093"
time="2022-12-26T13:02:00+03:00" level=info msg="Server kafka1.i:9093 closed connection"
time="2022-12-26T15:55:38+03:00" level=info msg="New connection for kafka1.i:9093"
time="2022-12-26T16:05:39+03:00" level=info msg="Server kafka1.i:9093 closed connection"

I've used endpoints "bootstrap.servers": "127.0.0.1:9091,127.0.0.1:9092,127.0.0.1:9093" on my clients, without providing ssl certs, and when producers is pushing messages i've got errors

Could not publish message (error: KafkaError{code=UNKNOWN,val=-1,str="Unknown broker error"})

There are no any errors in brokers logs, and i'm suggesting that my kafka-proxy or client configurations is not correct.

So my questions are:
Which endpoint i need to use in my clients?
What is the reason of starting Dynamic listener?
Is kafka-proxy terminating ssl on their own side and for karka-proxy clients connection is plaintext?

[ronaldpetty]

@adiletkabylbekov I am new to the proxy. Just curious, can you share one of your broker configs? I am guessing your listeners and advertised listeners differ (e.g. some reverse DNS lookup, dual stack, etc.). Not saying thats wrong, just thinking it might be the trigger.