Building connectors for Apache Kafka is hard. Chances are that you just read the previous sentence, and you subconsciously nooded with your head. The reason this happens is that Kafka Connect, which is the runtime platform behind the executing connectors, uses a not so trivial software architecture. There is also a lack of a proper documentation that teaches how the development framework works, how it connects with the runtime, and which best practices you must follow.
For situations like this, your best bet is to get hold of an existing code and try to do the same, hoping your own connector will be written the best way possible. By all means, this is an excellent strategy. But most of the time, it is hard to understand the code, as the connector you took as an example might have more code focused on solving the technical problem that the connector is aiming to solve than only the code part related to building custom connectors.
This is the reason this project exists. This is a minimalistic repository that contains a source connector, whose focus is to show how the development framework from Kafka Connect works. From this repository, you can easily derive your own connector and write only the code that matter for your technical use case. This project also provides you with the resources to build, deploy, test, and debug the code on-premises, as well as deploying it in the cloud.
The first thing you need to do to use this connector is to build it.
- Install the following dependencies:
- Build the Kafka Connect connector file.
mvn clean package💡 A file named target/my-first-kafka-connector-1.0.jar will be created. This is your connector for Kafka Connect.
With the connector properly built, you need to have a local environment to test it. This project includes a Docker Compose file that can spin up container instances for Apache Kafka and Kafka Connect.
- Install the following dependencies:
- Start the containers using Docker Compose.
docker compose up -dWait until the containers kafka and connect are started and healthy.
Nothing is actually happening since the connector hasn't been deployed. Once you deploy the connector, it will start generating sample data from an artificial source and write this data off into three Kafka topics.
- Deploy the connector.
curl -X POST -H "Content-Type:application/json" -d @examples/my-first-kafka-connector.json http://localhost:8083/connectors- Check if the connector is producing data to Kafka topics.
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic source-1 --from-beginningkafka-console-consumer.sh --bootstrap-server localhost:9092 --topic source-2 --from-beginningkafka-console-consumer.sh --bootstrap-server localhost:9092 --topic source-3 --from-beginning💡 All three topics should have sample data continuously generated for them.
This is actually an optional step, but if you wish to debug the connector code to learn its behavior by watching the code executing line by line, you can do so by using remote debugging. The Kafka Connect container created in the Docker Compose file was changed to rebind the port 8888 to enable support for JDWP. The instructions below assume that you are using Visual Studio Code for debugging. However, most IDEs for Java should provide support for JDWP. Please check their documentation manuals about how to attach their debugger to the remote process.
- Create a file named
.vscode/launch.jsonwith the following content:
{
"version": "0.2.0",
"configurations": [
{
"name": "Debug Connector",
"type": "java",
"request": "attach",
"hostName": "localhost",
"port": 8888
}
]
}- Set one or multiple breakpoints throughout the code.
- Launch a new debugging session to attach to the container.
- Play with the connector to trigger the live debugging.
Use the following command to undeploy the connector from Kafka Connect:
curl -X DELETE http://localhost:8083/connectors/my-first-kafka-connector- Stop the containers using Docker Compose.
docker compose downOnce you have played with the connector locally, you can also deploy the connector in the cloud. This project contains the code necessary for you to automatically deploy this connector in AWS using Terraform. To deploy the connector in AWS, you will need:
You will also need to have the credentials from your AWS account properly configured in your system. You can do this by running the command aws configure using the AWS CLI. More information on how to do this here.
Follow these steps to execute the deployment.
- Go to the
deploy-awsfolder.
cd deploy-aws- Initialize the Terraform plugins.
terraform init- Execute the deployment.
terraform apply -auto-approveIt may take several minutes for this deployment to finish, depending on your network speed, AWS region selected, and other factors. On average, you can expect something like 45 minutes.
🚨 Please note that the Terraform code will create 35 resources in your AWS account. It includes a VPC, subnets, security groups, IAM roles, CloudWatch log streams, an S3 bucket, a MSK cluster, an MSK Connect instance, and one EC2 instance. For this reason, be sure to execute the ninth step to destroy these resources, so you don't end up with an unexpected bill.
Once the deployment completes, you should see the following output:
Outputs:
execute_this_to_access_the_bastion_host = "ssh ec2-user@<PUBLIC_IP> -i cert.pem"- SSH into the bastion host.
ssh ec2-user@<PUBLIC_IP> -i cert.pem💡 The following steps assume you are connected to the bastion host.
- List the Kafka endpoints stored in the
/home/ec2-user/bootstrap-serversfile.
more /home/ec2-user/bootstrap-servers-
Copy one of the endpoints shown from the command above.
-
Check if the connector is writing data to the topics.
kafka-console-consumer.sh --bootstrap-server <ENDPOINT_COPIED_FROM_STEP_SIX> --topic source-1 --from-beginningkafka-console-consumer.sh --bootstrap-server <ENDPOINT_COPIED_FROM_STEP_SIX> --topic source-2 --from-beginningkafka-console-consumer.sh --bootstrap-server <ENDPOINT_COPIED_FROM_STEP_SIX> --topic source-3 --from-beginning💡 All three topics should have sample data continuously generated for them.
- Exit the connection with the bastion host.
exit- Destroy all the resources created by Terraform.
terraform destroy -auto-approveSee CONTRIBUTING for more information.
This project is licensed under the MIT-0 License. See the LICENSE file.
