diff --git a/docs/products/kafka/kafka-mirrormaker/concepts/mirrormaker2-tuning.md b/docs/products/kafka/kafka-mirrormaker/concepts/mirrormaker2-tuning.md index 19c7a339..2fde5669 100644 --- a/docs/products/kafka/kafka-mirrormaker/concepts/mirrormaker2-tuning.md +++ b/docs/products/kafka/kafka-mirrormaker/concepts/mirrormaker2-tuning.md @@ -1,50 +1,95 @@ --- -title: MirrorMaker 2 common parameters +title: Configuration parameters for Aiven for Apache Kafka® MirrorMaker 2 --- -MirrorMaker 2 (MM2) offers a suite of parameters to help with data replication and monitoring within Apache Kafka® ecosystems. -This topic outlines common parameters you can adjust, along with tips for -validating MM2's performance. - -1. Increase the value of `kafka_mirrormaker.tasks_max_per_cpu` in the - advanced options. Setting this to match the number of partitions can - enhance performance. -1. Ensure the interval seconds for the following settings match. You - can reduce these intervals for more frequent data synchronization: - - Advanced options: - - `kafka_mirrormaker.emit_checkpoints_interval_seconds` - - `kafka_mirrormaker.sync_group_offsets_interval_seconds` - - Replication flow: - - `Sync interval in seconds`. -1. To exclude internal topics, add these patterns to your topic - blacklist: - - `.*[\-\.]internal` `.*\.replica` `__.*` `connect.*` -1. Depending on your use case, consider adjusting these parameters: - - `kafka_mirrormaker.consumer_fetch_min_bytes` - - `kafka_mirrormaker.producer_batch_size` - - `kafka_mirrormaker.producer_buffer_memory` - - `kafka_mirrormaker.producer_linger_ms` - - `kafka_mirrormaker.producer_max_request_size` - -## MirrorMaker 2 validation tips - -To ensure MirrorMaker 2 is up-to-date with message processing, monitor -these: - -1. **Consumer lag metric**: Monitor the `kafka.consumer_lag` metric. - -1. **Dashboard metrics**: If MirrorMaker 2 stops adding records to a - topic, the `jmx.kafka.connect.mirror.record_count` metric stops - increasing, showing a flat line on the dashboard. - -1. **Retrieve latest messages with \`kt\`**: Use - [kt](https://github.com/fgeller/kt) to retrieve the latest messages - from all partitions with the following command: - - ``` - kt consume -auth ./mykafka.conf \ - -brokers SERVICE-PROJECT.aivencloud.com:PORT \ - -topic topicname -offsets all=newest:newest | \ - jq -c -s 'sort_by(.partition) | .[] | \ - {partition: .partition, value: .value, timestamp: .timestamp}' - ``` +Learn about the configuration layers in Aiven for Apache Kafka® MirrorMaker 2, including service, replication flow, and integration settings. +Optimize data replication and performance in your Kafka ecosystem. + +## Configuration layers + +Aiven for Apache Kafka® MirrorMaker 2 configurations are organized into three layers: +**service**, **replication flow**, and **integration**. Each layer controls a specific +aspect of the replication process. + +### Service configurations + +Service configurations control the behavior of nodes and workers in the +Aiven for Apache Kafka® MirrorMaker 2 cluster. + +**Example of a service configuration**: + +- Parameter: [`kafka_mirrormaker.emit_checkpoints_enabled`](https://aiven.io/docs/products/kafka/kafka-mirrormaker/reference/advanced-params#kafka_mirrormaker_emit_checkpoints_enabled) +- Description: Enables or disables periodically emitting consumer group offset + checkpoints to the target cluster. +- Impact: + - Automatically restarts the workers. + - Restarts all connectors and tasks. + +### Replication-flow configurations + +Replication-flow configurations manage the behavior of connectors, such as Source, Sink, +Checkpoint, and Heartbeat. + +**Example of a replication-flow configuration**: + +- Parameter: [`topics`](https://registry.terraform.io/providers/aiven/aiven/latest/docs/resources/mirrormaker_replication_flow) +- Description: Specifies a list of topics or regular expressions to replicate. + For more information, see the [topics included in a replication flow](/docs/products/kafka/kafka-mirrormaker/concepts/replication-flow-topics-regex). +- Impact: + - Automatically restarts the affected connectors. + - Restarts their associated tasks. + +### Integration configurations + +Integration configurations fine-tune the interaction between producers and consumers +within connectors. + +**Example of an integration configuration**: + +- Parameter: [`consumer_fetch_min_bytes`](https://registry.terraform.io/providers/aiven/aiven/latest/docs/resources/service_integration#nested-schema-for-kafka_mirrormaker_user_configkafka_mirrormaker) +- Description: Sets the minimum amount of data the server should return for a fetch + request. +- Impact: + - Automatically restarts the workers. + - Restarts all connectors and tasks. + +:::note +Most configuration parameters are derived from +[KIP-382: MirrorMaker 2.0 - Configuration Properties](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95650722#KIP382:MirrorMaker2.0-ConnectorConfigurationProperties). Refer to this resource for additional details. +::: + +## Common parameters + +This section describes common parameters that can be adjusted to optimize the performance +and behavior of Aiven for Apache Kafka MirrorMaker 2 replication. + +1. **Optimize task allocation**: + Increase the value of + [`kafka_mirrormaker.tasks_max_per_cpu`](/docs/products/kafka/kafka-mirrormaker/reference/advanced-params#kafka_mirrormaker_tasks_max_per_cpu) + in the advanced configuration. + Setting this to match the number of partitions can improve performance. + +1. **Align interval settings**: + Ensure the following interval settings match to achieve more frequent and synchronized + data replication: + - **Advanced configurations**: + - [`kafka_mirrormaker.emit_checkpoints_interval_seconds`](/docs/products/kafka/kafka-mirrormaker/reference/advanced-params#kafka_mirrormaker_emit_checkpoints_interval_seconds) + - [`kafka_mirrormaker.sync_group_offsets_interval_seconds`](/docs/products/kafka/kafka-mirrormaker/reference/advanced-params#kafka_mirrormaker_sync_group_offsets_interval_seconds) + - **Replication flow**: + - `Sync interval in seconds` + +1. **Exclude internal topics**: + Add these patterns to your topic blacklist to exclude internal topics: + - `.*[\-\.]internal` + - `.*\.replica` + - `__.*` + - `connect.*` + +1. **Adjust integration parameters**: + Modify these integration parameters based on your use case to improve producer and + consumer performance: + - `consumer_fetch_min_bytes` + - `producer_batch_size` + - `producer_buffer_memory` + - `producer_linger_ms` + - `producer_max_request_size` diff --git a/docs/products/kafka/kafka-mirrormaker/howto/monitor-replication-execution.md b/docs/products/kafka/kafka-mirrormaker/howto/monitor-replication-execution.md new file mode 100644 index 00000000..4095a7d8 --- /dev/null +++ b/docs/products/kafka/kafka-mirrormaker/howto/monitor-replication-execution.md @@ -0,0 +1,27 @@ +--- +title: Monitor replication execution +--- +Apache Kafka® MirrorMaker 2 uses Kafka® Connect for monitoring and state management, helping you track replication flows and address issues. + +## Monitoring tips + +Follow these tips to ensure that replication is up-to-date with message processing: + +1. **Monitor consumer lag:** Use the `kafka.consumer_lag` metric to track replication + progress and identify delays. + +1. **Track dashboard metrics:** Check the `jmx.kafka.connect.mirror.record_count` metric. + If MirrorMaker 2 stops adding records to a topic, this metric will show a flat line, + indicating no new records are being replicated. + +1. **Retrieve the latest messages using `kt`:** Use the + [**kt**](https://github.com/fgeller/kt) tool to fetch the latest messages from all + partitions. Run the following command: + + ```bash + kt consume -auth ./mykafka.conf \ + -brokers SERVICE-PROJECT.aivencloud.com:PORT \ + -topic topicname -offsets all=newest:newest | \ + jq -c -s 'sort_by(.partition) | .[] | \ + {partition: .partition, value: .value, timestamp: .timestamp}' + ``` diff --git a/sidebars.ts b/sidebars.ts index 1492d1bb..6d1e0243 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -1060,6 +1060,7 @@ const sidebars: SidebarsConfig = { items: [ 'products/kafka/kafka-mirrormaker/howto/integrate-external-kafka-cluster', 'products/kafka/kafka-mirrormaker/howto/setup-replication-flow', + 'products/kafka/kafka-mirrormaker/howto/monitor-replication-execution', 'products/kafka/kafka-mirrormaker/howto/remove-mirrormaker-prefix', 'products/kafka/kafka-mirrormaker/howto/datadog-customised-metrics', 'products/kafka/kafka-mirrormaker/howto/log-analysis-offset-sync-tool',