HowTo topics for Kafka tiered storage

.github/vale/dicts/aiven.dic

@@ -80,6 +80,8 @@ failover
 fileset
 filesets
 Flink
+Forecast
+Forecasted
 FusionAuth
 Gantt
 geocoder

_toc.yml

@@ -410,6 +410,16 @@ entries:
               - file: docs/products/kafka/howto/get-topic-partition-details
               - file: docs/products/kafka/howto/schema-registry
               - file: docs/products/kafka/howto/change-retention-period
+          - file: docs/products/kafka/howto/kafka-tiered-storage-get-started
+            title: Tiered storage
+            entries:
+            - file: docs/products/kafka/howto/enable-kafka-tiered-storage
+              title: Enable tiered storage
+            - file: docs/products/kafka/howto/configure-topic-tiered-storage
+              title: Configure tiered storage for topic
+            - file: docs/products/kafka/howto/tiered-storage-overview
+              title: Tiered storage overview
+
 
       - file: docs/products/kafka/reference
         title: Reference

docs/products/kafka/howto/configure-topic-tiered-storage.rst

@@ -0,0 +1,101 @@
+Enable and configure tiered storage for topics
+===========================================================================
+
+Aiven for Apache Kafka® offers flexibility in configuring tiered storage and setting retention policies. This guide will walk you through the process of configuring tiered storage for individual topic and configuring local retention policies. 
+
+.. important:: 
+
+   Aiven for Apache Kafka® tiered storage is an early availability feature, which means it has some restrictions on the functionality and service level agreement. It is intended for non-production environments, but you can test it with production-like workloads to assess the performance. To enable this feature, navigate to the :doc:`Feature preview </docs/platform/howto/feature-preview>` page within your user profile.
+
+Prerequisite
+------------
+* :doc:`Tiered storage enabled for the Aiven for Apache Kafka service </docs/products/kafka/howto/enable-kafka-tiered-storage>`.
+
+Configure tiered storage for topics via Aiven Console
+-------------------------------------------------------
+
+1. Access `Aiven console <https://console.aiven.io/>`_, select your project, and select your Aiven for Apache Kafka service.
+2. From the left sidebar, select **Topics**.
+3. Here, you have the option to either add a new topic with tiered storage configuration or modify an existing topic to use tiered storage.
+
+For a new topic
+~~~~~~~~~~~~~~~
+
+1. From the **Topics** page, select **Add topic**.
+2. Enable advanced configurations by setting the **Do you want to enable advanced configuration?** option to **Yes**.
+3. In the **Topic advanced configuration** drop-down, choose ``remote_storage_enable``. This action will reveal the **Remote storage enabled** drop-down. 
+4. Select **True** to activate tiered storage for the topic.
+5. Additionally, you can also set the values for ``local_retention_ms`` and ``local_retention_bytes`` using the respective options from the drop-down list.
+
+.. important:: 
+    If the values for ``local_retention_bytes`` and ``local_retention_ms`` are not set, they default to -2 or take the configuration from the service level.
+
+    When set to -2, the retention in local storage will match the total retention. In this scenario, the data segments sent to remote storage are also retained locally. The remote storage will contain older data segments than in the local storage only when the total retention is set to be greater than the local retention. 
+
+6. Select **Add topic** to save your changes and add the topic with tiered storage.
+
+For an existing topic
+~~~~~~~~~~~~~~~~~~~~~
+
+1. From the **Topics** page, select the topic for which you wish to enable tiered storage.
+2. Use the ellipsis or open the topic and choose **Modify**.
+3. In the **Modify** page, choose ``remote_storage_enable`` from the drop-down list, followed by selecting **True** from the **Remote storage enable** drop-down.
+4. Additionally, you can also set the values for ``local_retention_ms`` and ``local_retention_bytes`` using the respective options from the drop-down list.
+
+.. important:: 
+    If the values for ``local_retention_bytes`` and ``local_retention_ms`` are not set, they default to -2 or take the configuration from the service level. 
+
+    When set to -2, the retention in local storage will match the total retention. In this scenario, the data segments sent to remote storage are also retained locally. The remote storage will contain older data segments than in the local storage only when the total retention is set to be greater than the local retention. 
+
+
+5. Select **Update** to save your changes and activate tiered storage.
+
+
+Enable tiered storage for topics via Aiven CLI
+------------------------------------------------
+
+Using the :doc:`Aiven CLI </docs/tools/cli>`, you can enable tiered storage for specific Apache Kafka® topics and set retention policies.
+
+Enable tiered storage for a topic
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+To enable tiered storage for a Kafka topic, execute the following command:
+
+.. code-block:: bash
+
+   avn service topic-create \
+       --project demo-kafka-project \
+       --service-name demo-kafka-service \
+       --topic exampleTopic \
+       --partitions 1 \
+       --replication 2 \
+       --remote-storage-enable
+
+In this example:
+
+- ``demo-kafka-project`` is the name of your project.
+- ``demo-kafka-service`` is the name of your Aiven for Apache Kafka® service.
+- ``exampleTopic`` is the name of the topic you are creating with tiered storage enabled.
+- The topic will have 1 partition and a replication factor of 2.
+
+Configure retention policies for a topic with tiered storage
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+After enabling tiered storage, you can configure the retention policies for local storage:
+
+.. code-block:: bash
+
+   avn service topic-update \
+       --project demo-kafka-project \
+       --service-name demo-kafka-service \
+       --topic exampleTopic \
+       --local-retention-ms 100 \
+       --local-retention-bytes 10
+
+This command sets the local retention time to 100 milliseconds and the local retention size to 10 bytes for the topic named ``exampleTopic`` in the ``demo-kafka-service`` of the ``demo-kafka-project``.
+
+.. important:: 
+    If the values for ``local_retention_bytes`` and ``local_retention_ms`` are not set, they default to -2 or inherit the configuration from the service level. 
+
+    When set to -2, the retention in local storage will match the total retention. Consequently, data segments sent to remote storage are also retained locally. The remote storage will contain older data segments than the local storage, only if the total retention exceeds the local retention. 
+
+
+

docs/products/kafka/howto/enable-kafka-tiered-storage.rst

@@ -0,0 +1,108 @@
+Enable tiered storage for Aiven for Apache Kafka®
+=====================================================
+Learn how to enable tiered storage capability of Aiven for Apache Kafka®. This topic provides step-by-step instructions for maximizing storage efficiency using either the `Aiven console <https://console.aiven.io/>`_ or the :doc:`Aiven CLI </docs/tools/cli>`.
+
+.. important:: 
+
+   Aiven for Apache Kafka® tiered storage is an early availability feature, which means it has some restrictions on the functionality and service level agreement. It is intended for non-production environments, but you can test it with production-like workloads to assess the performance. To enable this feature, navigate to the :doc:`Feature preview </docs/platform/howto/feature-preview>` page within your user profile.
+
+Prerequisites
+--------------
+* Aiven account and a project set up in the Aiven Console
+* Aiven for Apache Kafka® service with Apache Kafka version 3.6. Tiered storage on Aiven for Apache Kafka is currently not available on all plans and regions. View the `plans and pricing page <https://aiven.io/pricing?product=kafka>`_ for a comprehensive list of supported plans and regions.
+* Aiven CLI
+
+
+Enable tiered storage via Aiven Console
+------------------------------------------
+Follow these steps to enable tiered storage for your service using the Aiven Console. 
+
+1. Access the  `Aiven console <https://console.aiven.io/>`_, and select your project.
+2. Create a new Aiven for Apache Kafka service or choose an existing one.  
+
+   - If you are :doc:`creating a new service </docs/platform/howto/create_new_service>`:
+
+     a. On the **Create Apache Kafka® service** page, scroll down to the **Tiered storage** section.
+     b. To enable tiered storage, select the **Enable tiered storage** toggle.
+     c. In the **Service summary**, you can view the pricing for tiered storage. 
+
+   - If you are using an existing service:
+
+     a. Go to the service's **Overview** page, scroll down to the **Tiered storage** section.
+     b. To enable tiered storage, select the **Enable tiered storage** toggle.
+
+
+3. Select the **Activate tiered storage** to save your settings and enable tiered storage for the service. 
+
+Once you have enabled tiered storage and it's in use, access the :doc:`Tiered storage overview </docs/products/kafka/howto/tiered-storage-overview>` on the left sidebar to get an overview of the overall usage and cost details.
+
+.. note:: 
+
+   If tiered storage is not yet enabled for your service, clicking **Tiered storage** from the sidebar provides you with the option to activate tiered storage.
+
+.. warning:: 
+   If you power off a service with tiered storage active, all remote data will be permanently deleted. You will not be billed for tiered storage usage during the powered-off period.
+
+
+Configuring default retention policies at service-level
+`````````````````````````````````````````````````````````````````````````````
+
+1. Access `Aiven console <https://console.aiven.io/>`_, select your project, and choose your Aiven for Apache Kafka service.
+2. On the **Overview** page, navigate to **Advanced configuration** and select **Change**.
+3. In the **Edit advanced configuration** view, choose **Add configuration option**.
+4. To set the retention policy for Aiven for Apache Kafka tiered storage, select ``kafka.log_local_retention_ms`` for time-specific retention or ``kafka.log_local_retention_bytes`` for size-specific retention.
+5. Select **Save advanced configuration** to apply your changes.
+
+Additionally, you can configure the retention policies from the :ref:`Tiered storage overview <modify-retention-polices>` page.
+
+Enable tiered storage via Aiven CLI 
+-----------------------------------------
+Follow these steps to enable tiered storage for your Aiven for Apache Kafka service using the :doc:`Aiven CLI </docs/tools/cli>`:
+
+1. Retrieve the project information using the following command: 
+
+   .. code-block:: bash
+
+        avn project details
+
+
+   If you need details for a specific project, use:
+
+   .. code-block:: bash
+
+        avn project details --project <your_project_name>
+
+2. Get the name of the Aiven for the Apache Kafka service for which you want to enable tiered storage by using the following command: 
+
+   .. code-block:: bash
+
+       avn service list
+
+   Make a note of the ``SERVICE_NAME`` corresponding to your Aiven for Apache Kafka service.
+
+3. Enable tiered storage using the command below:
+
+   .. code-block:: bash
+
+        avn service update \
+           --project demo-kafka-project \
+           demo-kafka-service \
+           -c tiered_storage.enabled=true
+
+
+
+
+In this command:
+
+* ``--project demo-kafka-project`` refers to the name of your project. In this example, it's ``demo-kafka-project``.
+* ``demo-kafka-service`` denotes the Aiven for Apache Kafka® service you intend to update. 
+* ``-c tiered_storage.enabled=true`` is the configuration flag that activates tiered storage for your Aiven for Apache Kafka service.
+
+
+
+
+
+
+
+
+

docs/products/kafka/howto/kafka-tiered-storage-get-started.rst

@@ -0,0 +1,38 @@
+
+Get started with tiered storage for Aiven for Apache Kafka® 
+====================================================================
+
+Aiven for Apache Kafka®'s tiered storage optimizes resources by keeping recent data—typically the most accessed—on faster local disks. As data becomes less active, it's transferred to more economical, slower storage, balancing performance with cost efficiency.
+
+For an in-depth understanding of tiered storage, how it works, and its benefits, see :doc:`Tiered storage in Aiven for Apache Kafka® </docs/products/kafka/concepts/kafka-tiered-storage>`.
+
+.. important:: 
+
+   Aiven for Apache Kafka® tiered storage is an early availability feature, which means it has some restrictions on the functionality and service level agreement. It is intended for non-production environments, but you can test it with production-like workloads to assess the performance. To enable this feature, navigate to the :doc:`Feature preview </docs/platform/howto/feature-preview>` page within your user profile.
+
+Enable tiered storage for service
+----------------------------------
+To use tiered storage, you need to first :doc:`enable </docs/products/kafka/howto/enable-kafka-tiered-storage>` it for your Aiven for Apache Kafka service® service. This foundational step ensures that the necessary infrastructure is in place.
+
+.. note:: 
+
+    Tiered storage for Aiven for Apache Kafka® is supported starting from Apache Kafka® version 3.6 and is not available for startup-2 plans.
+
+
+Configure tiered storage per topic
+----------------------------------
+Once the tiered storage is enabled at the service level, you can configure it for individual topics. In the Aiven for Apache Kafka Topics page, topics using tiered storage will display **Active** in the **Tiered storage** column.
+
+For detailed instructions, see :doc:`Configuring tiered storage for topics </docs/products/kafka/howto/configure-topic-tiered-storage>`.
+
+
+Tiered storage usage overview
+------------------------------
+Gain insights into tiered storage usage from the **Tiered Storage Overview** page in your Aiven for Apache Kafka service. This includes details on billing, settings, and specific storage aspects.
+
+For more information, see :doc:`Tiered Storage Overview in Aiven Console </docs/products/kafka/howto/tiered-storage-overview>`.
+
+
+
+
+

docs/products/kafka/howto/tiered-storage-overview.rst

@@ -0,0 +1,73 @@
+Tiered storage overview in Aiven Console
+========================================
+
+Aiven for Apache Kafka® offers a comprehensive overview of tiered storage, allowing you to understand its usage and make informed decisions. This overview provides insights into various aspects of tiered storage, including billing, settings, and storage details.
+
+.. important:: 
+
+   Aiven for Apache Kafka® tiered storage is an early availability feature, which means it has some restrictions on the functionality and service level agreement. It is intended for non-production environments, but you can test it with production-like workloads to assess the performance. To enable this feature, navigate to the :doc:`Feature preview </docs/platform/howto/feature-preview>` page within your user profile.
+
+
+Access tiered storage overview
+--------------------------------
+
+1. In the `Aiven console <https://console.aiven.io/>`_, choose your project and select your Aiven for Apache Kafka service.
+2. From the left sidebar, select **Tiered Storage**.
+
+   - If you haven't enabled tiered storage for your service, you'll have the option to enable it.
+   - If tiered storage is enabled but not configured for any topics, you have the option to set it up for topics directly. For more details, see :doc:`Enable and configure tiered storage for topics </docs/products/kafka/howto/configure-topic-tiered-storage>`. 
+
+3. Once configured, you can view an overview of tiered storage and its associated details.
+
+
+Key insights of tiered storage
+------------------------------
+
+Get a quick snapshot of the essential metrics and details related to tiered storage:
+
+- **Current billing expenses in USD**: Stay informed about your current tiered storage expenses.
+- **Forecasted month cost in USD**: Estimate your upcoming monthly costs based on current usage.
+- **Data tiered in bytes**: View the volume of data that has been tiered.
+- **Storage overview**: Understand the specifics of the storage mediums in use, including details about the used object storage and SSD storage.
+
+
+Current tiered storage configurations
+---------------------------------------------
+
+This section provides an overview of the current local cache details and retention policy configurations for tiered storage:
+
+- **Default local retention time (ms)**: Shows the current local data retention set in milliseconds.
+- **Default local retention bytes**: Shows the configured volume of data, in bytes, for local retention.
+
+.. _modify-retention-polices:
+
+Modify retention policies 
+`````````````````````````````````
+
+1. In the **Tiered storage settings** section, select the ellipsis (three dots) and select **Update tiered storage settings**.
+2. Within **Update tiered storage settings** page, adjust the values for:
+
+   - Local Cache
+   - Default Local Retention Time (ms)
+   - Default Local Retention Bytes
+
+3. Confirm your adjustments by selecting **Save changes**.
+
+
+Graphical view of tiered storage costs
+------------------------------------------
+
+Gain a visual understanding of your tiered storage expenses:
+
+- **Hourly expense**: Visualize your hourly expenses through graphical representation.
+- **Total cost and forecast**: Gain insights into your overall costs and receive a forecast. This forecast is an estimate based on your most recent usage patterns and trends. It provides an indicative projection and should not be considered as an exact or fixed value.
+
+
+Detailed storage overview
+-------------------------
+
+Explore the specifics of your storage usage and configurations:
+
+- **Used object storage and SSD storage**:Take a deep dive into the storage mediums in use.
+- **Filter by topic**: Narrow down your view to specific topics for focused insights.
+