Skip to content

Commit

Permalink
Merge pull request #366 from odidev/kafka_update
Browse files Browse the repository at this point in the history
Replaced kafka binary with the latest one
  • Loading branch information
pareenaverma authored Jun 28, 2023
2 parents 01a40d4 + 9f0da49 commit 7f2a986
Show file tree
Hide file tree
Showing 2 changed files with 20 additions and 20 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -385,51 +385,51 @@ Using a text editor, save the code below in a file called `kafka_cluster.yaml`.
apt update
apt install -y default-jdk
mkdir kafka_node
cd kafka_node/ && wget https://dlcdn.apache.org/kafka/3.2.3/kafka_2.13-3.2.3.tgz && tar -xzf kafka_2.13-3.2.3.tgz && cd kafka_2.13-3.2.3
cd kafka_node/ && wget https://dlcdn.apache.org/kafka/3.5.0/kafka_2.13-3.5.0.tgz && tar -xzf kafka_2.13-3.5.0.tgz && cd kafka_2.13-3.5.0

- name: On all kafka instances create log directory
shell: mkdir /tmp/kafka-logs

- name: On kafka1 instance update broker id
when: "'kafka1' in group_names"
lineinfile:
path: /home/ubuntu/kafka_node/kafka_2.13-3.2.3/config/server.properties
path: /home/ubuntu/kafka_node/kafka_2.13-3.5.0/config/server.properties
regexp: '^(.*)broker.id(.*)$'
line: 'broker.id=1'
backrefs: yes

- name: On kafka2 instance update broker id
when: "'kafka2' in group_names"
lineinfile:
path: /home/ubuntu/kafka_node/kafka_2.13-3.2.3/config/server.properties
path: /home/ubuntu/kafka_node/kafka_2.13-3.5.0/config/server.properties
regexp: '^(.*)broker.id(.*)$'
line: 'broker.id=2'
backrefs: yes

- name: On kafka3 instance update broker id
when: "'kafka3' in group_names"
lineinfile:
path: /home/ubuntu/kafka_node/kafka_2.13-3.2.3/config/server.properties
path: /home/ubuntu/kafka_node/kafka_2.13-3.5.0/config/server.properties
regexp: '^(.*)broker.id(.*)$'
line: 'broker.id=3'
backrefs: yes

- name: On all kafka instance uncomment listeners
lineinfile:
path: /home/ubuntu/kafka_node/kafka_2.13-3.2.3/config/server.properties
path: /home/ubuntu/kafka_node/kafka_2.13-3.5.0/config/server.properties
regexp: '^(.*)#listeners=PLAINTEXT://:9092(.*)$'
line: 'listeners=PLAINTEXT://:9092'
backrefs: yes
- name: On all kafka instance update zookeeper.connect
lineinfile:
path: /home/ubuntu/kafka_node/kafka_2.13-3.2.3/config/server.properties
path: /home/ubuntu/kafka_node/kafka_2.13-3.5.0/config/server.properties
regexp: '^(.*)zookeeper.connect=localhost:2181(.*)$'
line: 'zookeeper.connect={{zk_1_ip}}:2181,{{zk_2_ip}}:2181,{{zk_3_ip}}:2181'
backrefs: yes
- name: Start kafka_server
command: /home/ubuntu/kafka_node/kafka_2.13-3.2.3/bin/kafka-server-start.sh /home/ubuntu/kafka_node/kafka_2.13-3.2.3/config/server.properties
command: /home/ubuntu/kafka_node/kafka_2.13-3.5.0/bin/kafka-server-start.sh /home/ubuntu/kafka_node/kafka_2.13-3.5.0/config/server.properties
```

### Ansible Commands
Expand Down Expand Up @@ -548,10 +548,10 @@ Using a text editor, save the code below in a file called `client.yaml`. It will
apt update
apt install -y default-jdk
mkdir kafka_node
cd kafka_node/ && wget https://dlcdn.apache.org/kafka/3.2.3/kafka_2.13-3.2.3.tgz && tar -xzf kafka_2.13-3.2.3.tgz && cd kafka_2.13-3.2.3
cd kafka_node/ && wget https://dlcdn.apache.org/kafka/3.5.0/kafka_2.13-3.5.0.tgz && tar -xzf kafka_2.13-3.5.0.tgz && cd kafka_2.13-3.5.0
- name: Create a topic
command: /home/ubuntu/kafka_node/kafka_2.13-3.2.3/bin/kafka-topics.sh --create --topic test-topic --bootstrap-server {{kf_1_ip}}:9092,{{kf_2_ip}}:9092,{{kf_3_ip}}:9092 --replication-factor 3 --partitions 64
command: /home/ubuntu/kafka_node/kafka_2.13-3.5.0/bin/kafka-topics.sh --create --topic test-topic --bootstrap-server {{kf_1_ip}}:9092,{{kf_2_ip}}:9092,{{kf_3_ip}}:9092 --replication-factor 3 --partitions 64
```

### Ansible Commands
Expand Down Expand Up @@ -591,7 +591,7 @@ PLAY RECAP *********************************************************************
To describe the topic created ssh on the client instance and run the following command.
```console
ssh ubuntu@client_ip
cd kafka_node/kafka_2.13-3.2.3
cd kafka_node/kafka_2.13-3.5.0
./bin/kafka-topics.sh --topic test-topic --bootstrap-server kf_1_ip:9092,kf_2_ip:9092,kf_3_ip:9092 --describe
```
Replace the `client_ip`, `kf_1_ip`, `kf_2_ip`, `kf_3_ip` with the IP of client, kafka1, kafka2 and kafka3 respectively generated in inventory file present at location `/tmp/inventory`.
Expand Down Expand Up @@ -682,7 +682,7 @@ ubuntu@ip-172-31-31-117:~/kafka_node/kafka_2.13-3.2.3$ ./bin/kafka-console-produ
Open a new terminal on the client machine and run the consumer client using the following command.
```console
ssh ubuntu@client_ip
cd kafka_node/kafka_2.13-3.2.3
cd kafka_node/kafka_2.13-3.5.0
./bin/kafka-console-consumer.sh --topic test-topic --bootstrap-server kf_1_ip:9092,kf_2_ip:9092,kf_3_ip:9092
```
Replace the `client_ip`, `kf_1_ip`, `kf_2_ip`, `kf_3_ip` with the IP of client, kafka1, kafka2 and kafka3 respectively generated in inventory file present at location `/tmp/inventory`.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,9 @@ Run the commands shown to download and setup Kafka on node 1:
```console
mkdir kafka_node1
cd kafka_node1
wget https://dlcdn.apache.org/kafka/3.2.3/kafka_2.13-3.2.3.tgz
tar -xzf kafka_2.13-3.2.3.tgz
cd kafka_2.13-3.2.3
wget https://dlcdn.apache.org/kafka/3.5.0/kafka_2.13-3.5.0.tgz
tar -xzf kafka_2.13-3.5.0.tgz
cd kafka_2.13-3.5.0
```
Use a file editor of your choice and replace the contents in `config/server.properties` with the contents below:.
Replace `zk_1_ip`,`zk_2_ip` and `zk_3_ip` with the IP addresses of the 3 Zookeeper nodes you setup.
Expand Down Expand Up @@ -56,9 +56,9 @@ Run the commands shown to download and setup Kafka on node 2:
```console
mkdir kafka_node2
cd kafka_node2
wget https://dlcdn.apache.org/kafka/3.2.3/kafka_2.13-3.2.3.tgz
tar -xzf kafka_2.13-3.2.3.tgz
cd kafka_2.13-3.2.3
wget https://dlcdn.apache.org/kafka/3.5.0/kafka_2.13-3.5.0.tgz
tar -xzf kafka_2.13-3.5.0.tgz
cd kafka_2.13-3.5.0
```
Use a file editor of your choice and replace the contents in `config/server.properties` with the contents below:.
Replace `zk_1_ip`,`zk_2_ip` and `zk_3_ip` with the IP addresses of the 3 Zookeeper nodes you setup.
Expand Down Expand Up @@ -92,9 +92,9 @@ Run the commands shown to download and setup Kafka on node 3:
```console
mkdir kafka_node3
cd kafka_node3
wget https://dlcdn.apache.org/kafka/3.2.3/kafka_2.13-3.2.3.tgz
tar -xzf kafka_2.13-3.2.3.tgz
cd kafka_2.13-3.2.3
wget https://dlcdn.apache.org/kafka/3.5.0/kafka_2.13-3.5.0.tgz
tar -xzf kafka_2.13-3.5.0.tgz
cd kafka_2.13-3.5.0
```
Use a file editor of your choice and replace the contents in `config/server.properties` with the contents below:.
Replace `zk_1_ip`,`zk_2_ip` and `zk_3_ip` with the IP addresses of the 3 Zookeeper nodes you setup.
Expand Down

0 comments on commit 7f2a986

Please sign in to comment.