C37-Platform

The C37 platform automates the deployment and management of a robust Kafka-based streaming ecosystem on Kubernetes and Docker environments. By leveraging Helm Charts and Docker Compose, it simplifies the setup and scalability of the Confluent Kafka ecosystem and is suitable for development, testing, and production environments. This platform integrates essential components such as Kafka, Kafka Connect, Schema Registry, REST Proxy, and a Kafka UI, offering comprehensive monitoring and management capabilities.

In this project, Zookeeper was not used as Kafka > 3.2 supports internal clustering with KRaft for production environments.

The project was inspired by the Helm Charts cp-helm-charts by Confluent Inc. and kafka-k8s by Rafael Natali. Its aim is to utilize components of the Confluent Platform and to demonstrate and test the deployment of Kafka on Kubernetes.

Key Components

• Kafka: A distributed streaming platform that forms the backbone of our ecosystem.

• Kafka Connect: Facilitates the streaming of data between Kafka and other systems.

• Kafka REST Proxy: Provides RESTful access to Kafka clusters, enabling easy integration with other applications.

• ksqlDB: Enables real-time data processing, allowing for complex queries and data transformation.

• Schema Registry: Manages Avro schemas for Kafka topics, ensuring data consistency.

• Kafka UI: A web interface for cluster management, offering a visual overview of your Kafka environment.

Project Structure

The project structure is designed for ease of use and scalability, including the following main components and directories:

• bin/: Contains executable scripts for project initialization and maintenance.

• helm-charts/: Helm charts for deploying services on Kubernetes.

• kafka-connect/: Plugins for Kafka Connect, facilitating seamless data integration.

• kafka-utility/: Utilities for Kafka management tasks, enhancing operational efficiency.

Usage

Important

This project uses podman and podman-compose as default containerization tools, but provides flexibility for users using docker and docker-compose. By simply setting the environment variables: export DOCKER=docker export DOCKER_COMPOSE=docker-compose

Prerequisites

This project was rigorously tested on a MacBook Pro with an Apple M2 Max chip, 32 GB of RAM under macOS Sonoma 14.2.1, Docker Desktop 4.27.2, and Kubernetes v1.29.1.

Mandatory

• make: For executing instructions in the Makefile.

• docker and docker-compose: For running services in containers. Note that Docker Desktop integrates docker-compose.

• kubernetes: For managing containerized applications across clusters, facilitating deployment, scaling, and automation through declarative configurations.

• helm: For deploying on Kubernetes.

Recommendation

• kcat (formerly known as kafkacat): Essential for local Kafka testing. It provides a command-line interface to produce and consume messages from Kafka topics.

• k9s: For efficient management of Kubernetes clusters. It offers a user-friendly console interface that simplifies navigating, observing, and managing your clusters.

Note	While this setup is tested on macOS with specific hardware, efforts have been made to ensure compatibility across various platforms. For users on Windows or Linux, please refer to the respective installation guides for each tool, adjusting commands and setups as necessary.

Deployment

Helm

Deploying with Helm offers a streamlined approach to managing Kafka clusters on Kubernetes. The charts are optimized for KRaft mode, eliminating the need for Zookeeper.

• make helm-install or make hi: Deploy the Kafka ecosystem on Kubernetes.

• make helm-update or make hu: Apply updates or configuration changes.

• make helm-uninstall or make hd: Remove the deployment from Kubernetes.

Docker

Docker and Docker-Compose provide a flexible alternative for environments better suited to container management.

• make docker-build or make db: Build Docker images for the stack.

• make docker-up or make du: Start the entire stack using Docker-Compose.

• make docker-down or make dd: Tear down the stack, removing containers and networks.

• make docker-restart or make dr: Swiftly restart the environment by combining docker-down and docker-up.

Community and Contributions

We welcome contributions and feedback from the community! If you’re interested in contributing to the C37-Platform, please check out our GitHub repository for open issues or to submit a pull request. For any questions or suggestions, feel free to open an issue on GitHub.

Conclusion

The C37-Platform provides a comprehensive solution for deploying and managing the Confluent Kafka ecosystem on both Kubernetes and Docker environments. With support for Helm charts and Docker Compose, it offers flexibility in deployment options, catering to a wide range of applications from development to production. We hope this platform empowers developers and system administrators alike to efficiently manage their Kafka-based streaming solutions.