VKitM is a Apache Kafka proxy which uses the native Kafka protocol.
Its name is due to how it has been designed: the application acts as an intermediary between the Kafka client and the Kafka cluster, modifying the requests and responses from/to the client. Actually, the client thinks that the connection is to a Kafka cluster (by now, with one node) and doesn't know what's behind (an entirely Kafka cluster).
Right now, the implementation includes a subset of the API Keys but can produce and consume messages.
- Kafka client connects to a single Kafka broker. Actually, it's connecting to a VKitM.
- VKitM modifies the request, depending of the API Key.
- The modified request is sent to the actual Kafka cluster, if applicable.
- VKitM receives the response from the Kafka brokers.
- Based on the kind of response it has to send to the client, VKitM changes data related with brokers, partition leaders and other metadata in the final response.
- The client receives the response thinking that there is just one node in the cluster and continues.
DISCLAIMER: This application has been implemented in order to provide another option when a proxy is required. If you use VKitM with a different purpose from the one has been designed it's under your own responsibility.
There are multiple Kafka proxies out there but most of them are HTTP REST based. A well-known REST Proxy is this one.
However, most of them lack of some of the following features I'd like to provide in VKitM:
- Use the native Kafka protocol directly, improving the performance.
- Be able to modify the messages from the clients on the fly when we cannot do it in the client itself (due to business cases or whatever) including interceptors.
- Define custom ACLs to complement those that already exist (or maybe not) in the Kafka Cluster.
- Change the kind of protocol in your clients (secured or not). I.e.: using TLS connection from the clients to VKitM and from VKitM to Kafka brokers in plaintext (or vice versa) or even change the certificates they use.
gradle clean build
java -jar build/libs/vkitm-{VERSION}.jar src/main/resources/application.conf
NOTE: modify src/main/resources/application.conf file, if applies, to your custom configuration.
You can run VKitM inside a Docker container. Use the image hosted in Docker Hub or just build your own one.
docker build -t mmolimar/vkitm --build-arg VERSION=<VERSION> .
docker run -p <HOST_PORT>:<CONTAINER_PORT> -v /path/to/conf/directory:/vkitm/conf mmolimar/vkitm
NOTE: path /path/to/conf/directory must contain a file named application.conf with the VKitM configuration.
- Implement most of the Kafka protocol API Keys.
- Develop a VKitM cluster, avoiding SPOF (Single Point Of Failure) which now exists.
- Source Code: https://github.com/mmolimar/vkitm
- Issue Tracker: https://github.com/mmolimar/vkitm/issues
Released under the Apache License, version 2.0.