rdkafka2 plugin not works with SSL authentication method....error="Waiting for delivery timed out after 30 seconds"

[kunalv89]

Hi Team, Requesting your help to setup secure communication with SSL from Log aggregator (td-agent) to kafka broker, but getting error each time.

my td-agent configuration as below.
<match unity> @type kafka2 brokers 192.168.10.1:9092,192.168.10.2:9092 use_event_time true security_protocol ssl ssl_truststore_location "/etc/td-agent/keys/kafka.client.keystore.jks" ssl_truststore_password "*****" ssl_keystore_location "/etc/td-agent/keys/kafka.client.keystore.jks" ssl_keystore_password "*****" key_password "*****" max_send_retries 3 <format> @type json </format> <buffer> @type file path /var/log/td-agent/buffer/td flush_interval 3s </buffer> topic_key topicone default_topic topicone </match>

Application version as below :
kafka version 2.2.2
fluentd-1.11.5
ruby="2.4.10"

Below are the sample error log.
path "/var/log/td-agent/buffer/td"
flush_interval 3s

is not used.
2022-03-10 18:46:52 +0530 [info]: #0 starting fluentd worker pid=18172 ppid=18167 worker=0
2022-03-10 18:46:52 +0530 [info]: #0 initialized kafka producer: fluentd
2022-03-10 18:46:52 +0530 [info]: #0 listening port port=8888 bind="0.0.0.0"
2022-03-10 18:46:52 +0530 [info]: #0 fluentd worker is now running worker=0
2022-03-10 18:46:53 +0530 [warn]: #0 Send exception occurred: Could not connect to any of the seed brokers:

• kafka://192.168.10.2:9092: Connection error EOFError: EOFError
• kafka://192.168.10.2:9092: Connection error EOFError: EOFError

[fujimotos]

This seems to be an issue in fluent-plugin-kafka. I found a similar
bug report there:

fluent/fluent-plugin-kafka#289

A few users commented that substituting topic_key with default_topic test
fixed their EOFError, so might yours as well. For details, see:

fluent/fluent-plugin-kafka#289 (comment)

[kunalv89]

@fujimotos , The error logs are similar but, they didn't use SSL authentication within config. i'm still getting the same error after remove topic_key.

2022-03-11 10:35:10 +0530 [warn]: #0 failed to flush the buffer. retry_time=4 next_retry_seconds=2022-03-11 10:35:19 +0530 chunk="5d9a1abb773def243b28c2b9e44f030d" error_class=Kafka::ConnectionError error="Could not connect to any of the seed brokers:\n- kafka://192.168.10.1:9092: Connection error EOFError: EOFError\n- kafka://192.168.10.2:9092: Connection error EOFError: EOFError\n- kafka://192.168.10.3:9092: Connection error EOFError: EOFError"