kafka-proxy-test

kafka-proxy test project.

Requirements

For setting up the local Kafka environment (localhost):

• Docker
• Docker Compose

For testing:

• JDK 1.8
• Java client embedded in Kafka

For bootstraping a local Kafka VM (kafka.example.org):

• Virtual Box
• Vagrant
• Ansible

Tests

curl -Ls https://mirrors.up.pt/pub/apache/kafka/2.5.0/kafka_2.12-2.5.0.tgz | tar xz

No auth test

Comment in docker-compose.yml:

      # - --auth-local-enable
      # - --auth-local-command=/auth-ldap
      # - --auth-local-param=--start-tls=false
      # - --auth-local-param=--url=ldap://openldap:389
      # - --auth-local-param=--bind-dn=cn=admin,dc=example,dc=org
      # - --auth-local-param=--bind-passwd=admin
      # - --auth-local-param=--user-search-base=ou=people,dc=example,dc=org
      # - --auth-local-param=--user-filter=(&(objectClass=inetOrgPerson)(uid=%u)(memberOf=cn=kafka-users,ou=groups,dc=example,dc=org))

Then:

docker-compose up -d

To list the topics:

kafka_2.12-2.5.0/bin/kafka-topics.sh --list --bootstrap-server localhost:32400,localhost:32401,localhost:32402

To create the test topic:

kafka_2.12-2.5.0/bin/kafka-topics.sh --create --bootstrap-server localhost:32400,localhost:32401,localhost:32402 --replication-factor 3 --partitions 1 --topic test

To produce a single message:

echo "Hello, World!" | kafka_2.12-2.5.0/bin/kafka-console-producer.sh --bootstrap-server localhost:32400,localhost:32401,localhost:32402 --topic test

To consume the message:

kafka_2.12-2.5.0/bin/kafka-console-consumer.sh --bootstrap-server localhost:32400,localhost:32401,localhost:32402 --topic test --from-beginning --max-messages 1

LDAP auth test

Uncomment in docker-compose.yml:

      - --auth-local-enable
      - --auth-local-command=/auth-ldap
      - --auth-local-param=--start-tls=false
      - --auth-local-param=--url=ldap://openldap:389
      - --auth-local-param=--bind-dn=cn=admin,dc=example,dc=org
      - --auth-local-param=--bind-passwd=admin
      - --auth-local-param=--user-search-base=ou=people,dc=example,dc=org
      - --auth-local-param=--user-filter=(&(objectClass=inetOrgPerson)(uid=%u)(memberOf=cn=kafka-users,ou=groups,dc=example,dc=org))

Then:

docker-compose up -d

Set the KAFKA_OPTS environment variable:

export KAFKA_OPTS="-Djava.security.auth.login.config=$(pwd)/jaas.conf"

To list the topics:

kafka_2.12-2.5.0/bin/kafka-topics.sh --list --bootstrap-server localhost:32400,localhost:32401,localhost:32402 --command-config client-sasl.properties

To create the test topic:

kafka_2.12-2.5.0/bin/kafka-topics.sh --create --bootstrap-server localhost:32400,localhost:32401,localhost:32402 --replication-factor 3 --partitions 1 --topic test --command-config client-sasl.properties

To produce a single message:

echo "Hello, World!" | kafka_2.12-2.5.0/bin/kafka-console-producer.sh --broker-list localhost:32400,localhost:32401,localhost:32402 --topic test --producer.config client-sasl.properties

To consume the message:

kafka_2.12-2.5.0/bin/kafka-console-consumer.sh --bootstrap-server localhost:32400,localhost:32401,localhost:32402 --topic test --from-beginning --max-messages 1 --consumer.config client-sasl.properties

ℹ️ Alternatively, you can run ./test.sh.

LDAP auth/TLS termination test

vagrant up

Set the KAFKA_OPTS environment variable:

export KAFKA_OPTS="-Djava.security.auth.login.config=$(pwd)/jaas.conf"

To list the topics:

kafka_2.12-2.5.0/bin/kafka-topics.sh --list --bootstrap-server kafka.example.org:32400,kafka.example.org:32401,kafka.example.org:32402 --command-config client-sasl-ssl.properties

To create the test topic:

kafka_2.12-2.5.0/bin/kafka-topics.sh --create --bootstrap-server kafka.example.org:32400,kafka.example.org:32401,kafka.example.org:32402 --replication-factor 3 --partitions 1 --topic test --command-config client-sasl-ssl.properties

To produce a single message:

echo "Hello, World!" | kafka_2.12-2.5.0/bin/kafka-console-producer.sh --broker-list kafka.example.org:32400,kafka.example.org:32401,kafka.example.org:32402 --topic test --producer.config client-sasl-ssl.properties

To consume the message:

kafka_2.12-2.5.0/bin/kafka-console-consumer.sh --bootstrap-server kafka.example.org:32400,kafka.example.org:32401,kafka.example.org:32402 --topic test --from-beginning --max-messages 1 --consumer.config client-sasl-ssl.properties

ℹ️ You can use 192.168.33.10 instead of kafka.example.org.