Yet-Another-Kafka-YAK2.0

YAK2.0 can also be termed as Yet Another Kafka 2.0.

This project aims to create Kafka like software or platform, for producers and consumers.

Table of Contents

Introduction

Prerequisites and Techstack

Steps for execution

Usage

Introduction

In this project we have used Python and RabbitMQ to mimic the functionalities of Kafka.
We have successfully implemented producers, consumers, 3 brokers, topic log, etc...
This project can also handle multiple or any random number of producers and consumers.

(back to top)

Clone the Yet-Another-Kafka--YAK-2.0 repository

 git clone https://github.com/ankitacoder3/Yet-Another-Kafka-YAK2.0.git

Start RabbitMQ in terminal/command prompt. By typing
```
rabbitmq-server.bat 
```

Open a new terminal and start consumer. By typing

python consumer.py test n

Or

python consumer.py test --from-beginning

Open another new terminal and start producer. By typing
```
python producer1.py test 101
```
Check the terminal where consumer is running, to check for messages received.
Now, open the 'broker' folder, which is under 'YAK' folder.
In that folder you can check the 'topic-log' file for the log activity.
In the same folder, check the '101' folder (where 101 is the key of leader broker).
Open the 'test-lead' file (where test is the topic). You can see that the message has been appended in file.
Hence, the messages have been successfully sent, received and stored. A file also maintains all the activities done as log.

To run multiple producers and consumers, open a new terminal for each new consumer and producer.
To run a consumer type the following:
- To view the recent message sent
```
python consumer.py <topic_name> n
```
- To view all messages sent from beginning or starting of topic
```
python consumer.py <topic_name> --from-beginning
```
To run a producer type the following
```
python producers1.py <topic_name> <key_of_leader_broker>
```
where, key_of_leader_broker = 101 or 102 or 103

(back to top)

Thank You.