diff --git a/course-definition.yml b/course-definition.yml index a16c16c..66e364b 100644 --- a/course-definition.yml +++ b/course-definition.yml @@ -57,6 +57,15 @@ extensions: [fetch-api]: https://kafka.apache.org/protocol.html#The_Messages_Fetch + - slug: "topic-partition-metadata" + name: "Topic Partition Metadata" + description_markdown: | + In this challenge extension you'll add support for fetching metadata about topic partitions by implementing the [DescribeTopicPartition][describe-topic-partition] API. + + Along the way you'll learn about how Kafka stores cluster metadata on disk, how to read topic metadata from disk and more. + + [describe-topic-partition]: https://kafka.apache.org/protocol.html#The_Messages_DescribeTopicPartitions + stages: - slug: "vi6" name: "Bind to a port" @@ -522,3 +531,218 @@ stages: 🚧 **We're still working on tests for this stage!**. marketing_md: |- In this stage, you'll implement the Fetch response for a topic with multiple messages, reading them from disk. + + # Describe Topic Partitions + + - slug: "yk1" + primary_extension_slug: "topic-partition-metadata" + name: "Include DescribeTopicPartition in APIVersions" + difficulty: medium + description_md: |- + In this stage, you'll add an entry for the `DescribeTopicPartition` API to the APIVersions response. + + 🚧 **We're still working on instructions for this stage**. You can find notes on how the tester works below. + + In the meantime, please use + [this link](https://forum.codecrafters.io/new-topic?category=Challenges&tags=challenge%3Akafka&title=Question+about+YK1+%28Include+DescribeTopicPartition+in+APIVersions%29&body=%3Cyour+question+here%3E) + to ask questions on the forum. + + ### Tests + + The tester will execute your program like this: + + ```bash + $ ./your_program.sh + ``` + + + It'll then connect to your server on port 9092 and send a valid `APIVersions` (v4) request. + + The tester will validate that: + + - The first 4 bytes of your response (the "message length") are valid. + - The correlation ID in the response header matches the correlation ID in the request header. + - The error code in the response body is `0` (No Error). + - The response body contains at least three entries of APIKeyResponses. + - The response for the API key `18` (API_VERSIONS) has a `MaxVersion` of at least 4, and a `MinVersion` of at least 0. + - The response for the API key `75` (DESCRIBE_TOPIC_PARTITION) has a `MaxVersion` of at least 0, and a `MinVersion` of at least 0. + + ### Notes + + - You don't have to implement support for the `DescribeTopicPartition` request in this stage. We'll get to this in later stages. + - You'll still need to include the entry for `APIVersions` in your response to pass the previous stage. + - The `MaxVersion` for the `DescribeTopicPartition` and `APIVersions` are different. For `APIVersions`, it is 4. For `DescribeTopicPartition`, it is 0. + marketing_md: |- + In this stage, you'll add the DescribeTopicPartition API to the APIVersions response. + + - slug: "vt6" + primary_extension_slug: "topic-partition-metadata" + name: "DescribeTopicPartition with unknown topic" + difficulty: medium + description_md: |- + In this stage, you'll implement the `DescribeTopicPartition` response for an unknown topic. + + 🚧 **We're still working on instructions for this stage**. You can find notes on how the tester works below. + + In the meantime, please use + [this link](https://forum.codecrafters.io/new-topic?category=Challenges&tags=challenge%3Akafka&title=Question+about+VT6+%28DescribeTopicPartition+with+unknown+topic%29&body=%3Cyour+question+here%3E) + to ask questions on the forum. + + ### Tests + + The tester will execute your program like this: + + ```bash + $ ./your_program.sh + ``` + + It'll then connect to your server on port 9092 and send a `DescribeTopicPartition` (v0) request. The request will contain a single topic which 1 partition. + + The tester will validate that: + + - The first 4 bytes of your response (the "message length") are valid. + - The correlation ID in the response header matches the correlation ID in the request header. + - The error code in the response body is `3` (UNKNOWN_TOPIC_OR_PARTITION). + - The response body should be valid DescribeTopicPartitionResponse. + - The `topic_name` field in the response should be equal to the topic name sent in the request. + - The `topic_id` field in the response should be equal to `00000000-0000-0000-0000-000000000000`. + - The partition response should be empty. (As there are no partitions assigned to this non-existent topic.) + + ### Notes + + - You'll need to parse the `DescribeTopicPartition` request in this stage to get the topic name to send in the response. + - The official docs for the `DescribeTopicPartition` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_DescribeTopicPartition). + + marketing_md: |- + In this stage, you'll implement the DescribeTopicPartition response for an unknown topic. + + - slug: "ea7" + primary_extension_slug: "topic-partition-metadata" + name: "DescribeTopicPartition with single topic" + difficulty: very hard + description_md: |- + In this stage, you'll implement the `DescribeTopicPartition` response for a single topic. + + 🚧 **We're still working on instructions for this stage**. You can find notes on how the tester works below. + + In the meantime, please use + [this link](https://forum.codecrafters.io/new-topic?category=Challenges&tags=challenge%3Akafka&title=Question+about+EA7+%28DescribeTopicPartition+with+single+topic%29&body=%3Cyour+question+here%3E) + to ask questions on the forum. + + ### Tests + + The tester will execute your program like this: + + ```bash + $ ./your_program.sh + ``` + + It'll then connect to your server on port 9092 and send a `DescribeTopicPartition` (v0) request. The request will contain an single topic name. The topic exists and has a single partition assigned to it. + + The tester will validate that: + + - The first 4 bytes of your response (the "message length") are valid. + - The correlation ID in the response header matches the correlation ID in the request header. + - The error code in the response body is `0` (NO_ERROR). + - The response body should be valid DescribeTopicPartitionResponse. + - The `topic_name` field in the response should be equal to the topic name sent in the request. + - The `topic_id` field in the response should be equal to the topic UUID of the topic. + - The partition response should contain details of the partition assigned to the topic. + - The error code in the partition response should be `0` (NO_ERROR). + - The `partition_index` field in the partition response should be equal to the partition ID of the partition assigned to the topic. + + ### Notes + + - You'll need to parse the `DescribeTopicPartition` request in this stage to get the topic name to send in the response. + - The official docs for the `DescribeTopicPartition` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_DescribeTopicPartition). + + marketing_md: |- + In this stage, you'll implement the DescribeTopicPartition response for a single topic. + + + + - slug: "ku4" + primary_extension_slug: "topic-partition-metadata" + name: "DescribeTopicPartition with single topic but multiple partitions" + difficulty: hard + description_md: |- + In this stage, you'll implement the `DescribeTopicPartition` response for a single topic with multiple partitions. + + 🚧 **We're still working on instructions for this stage**. You can find notes on how the tester works below. + + In the meantime, please use + [this link](https://forum.codecrafters.io/new-topic?category=Challenges&tags=challenge%3Akafka&title=Question+about+KU4+%28DescribeTopicPartition+with+single+topic+but+multiple+partitions%29&body=%3Cyour+question+here%3E) + to ask questions on the forum. + + ### Tests + + The tester will execute your program like this: + + ```bash + $ ./your_program.sh + ``` + + It'll then connect to your server on port 9092 and send a `DescribeTopicPartition` (v0) request. The request will contain an single topic name. The topic exists and has 2 partitions assigned to it. + + The tester will validate that: + + - The first 4 bytes of your response (the "message length") are valid. + - The correlation ID in the response header matches the correlation ID in the request header. + - The error code in the response body is `0` (NO_ERROR). + - The response body should be valid DescribeTopicPartitionResponse. + - The `topic_name` field in the response should be equal to the topic name sent in the request. + - The `topic_id` field in the response should be equal to the topic UUID of the topic. + - The partition response should contain details of the 2 partitions assigned to the topic. + - There should be 2 partition responses. + - The error code in both the partition responses should be `0` (NO_ERROR). + - The `partition_index` field in both the partition responses should be equal to the partition ID of the partition assigned to the topic. + + ### Notes + + - You'll need to parse the `DescribeTopicPartition` request in this stage to get the topic name to send in the response. + - The official docs for the `DescribeTopicPartition` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_DescribeTopicPartition). + + marketing_md: |- + In this stage, you'll implement the DescribeTopicPartition response for a single topic with multiple partitions. + + - slug: "wq2" + primary_extension_slug: "topic-partition-metadata" + name: "DescribeTopicPartition with multiple topics" + difficulty: very hard + description_md: |- + In this stage, you'll implement the `DescribeTopicPartition` response for multiple topics. + + 🚧 **We're still working on instructions for this stage**. You can find notes on how the tester works below. + + In the meantime, please use + [this link](https://forum.codecrafters.io/new-topic?category=Challenges&tags=challenge%3Akafka&title=Question+about+WQ2+%28DescribeTopicPartition+with+multiple+topics%29&body=%3Cyour+question+here%3E) + to ask questions on the forum. + + ### Tests + + The tester will execute your program like this: + + ```bash + $ ./your_program.sh + ``` + + It'll then connect to your server on port 9092 and send a `DescribeTopicPartition` (v0) request. The request will contain 3 topic names. The topics exist and have 1 or 2 partitions assigned to each. + + The tester will validate that: + + - The first 4 bytes of your response (the "message length") are valid. + - The correlation ID in the response header matches the correlation ID in the request header. + - The error code in the topic response body is `0` (NO_ERROR). + - The `topic_name` field in the topic response should be equal to the topic name sent in the request. + - The `topic_id` field in the topic response should be equal to the topic UUID of the topic. + - The partition response should contain details of the partitions assigned to the topic. + - The error code in all the partition responses should be `0` (NO_ERROR). + - The `partition_index` field in all the partition responses should be equal to the partition ID of the partition assigned to the topic. + + ### Notes + + - You'll need to parse the `DescribeTopicPartition` request in this stage to get the topic name to send in the response. + - The official docs for the `DescribeTopicPartition` request can be found [here](https://kafka.apache.org/protocol.html#The_Messages_DescribeTopicPartition). + + marketing_md: |- + In this stage, you'll implement the DescribeTopicPartition response for multiple topics.