[8.12] Expand/fix custom presets docs (backport #870)

docs/en/ingest-management/elastic-agent/configuration/outputs/output-elasticsearch.asciidoc

@@ -488,42 +488,41 @@ The memory queue waits for the output to acknowledge or drop events. If
 the queue is full, no new events can be inserted into the memory queue. Only
 after the signal from the output will the queue free up space for more events to be accepted.
 
-The memory queue is controlled by the parameters `flush.min_events` and `flush.timeout`. If
-`flush.timeout` is `0s` or `flush.min_events` is `0` or `1` then events can be sent by the output as
+The memory queue is controlled by the parameters `queue.mem.flush.min_events` and `queue.mem.flush.timeout`. If
+`queue.mem.flush.timeout` is `0s` or `queue.mem.flush.min_events` is `0` or `1` then events can be sent by the output as
 soon as they are available. If the output supports a `bulk_max_size` parameter it controls the
 maximum batch size that can be sent.
 
-If `flush.min_events` is greater than `1` and `flush.timeout` is greater than `0s`, events will only
-be sent to the output when the queue contains at least `flush.min_events` events or the
-`flush.timeout` period has expired. In this mode the maximum size batch that that can be sent by the
-output is `flush.min_events`. If the output supports a `bulk_max_size` parameter, values of
-`bulk_max_size` greater than `flush.min_events` have no effect. The value of `flush.min_events`
+If `queue.mem.flush.min_events` is greater than `1` and `queue.mem.flush.timeout` is greater than `0s`, events will only
+be sent to the output when the queue contains at least `queue.mem.flush.min_events` events or the
+`queue.mem.flush.timeout` period has expired. In this mode the maximum size batch that that can be sent by the
+output is `queue.mem.flush.min_events`. If the output supports a `bulk_max_size` parameter, values of
+`bulk_max_size` greater than `queue.mem.flush.min_events` have no effect. The value of `queue.mem.flush.min_events`
 should be evenly divisible by `bulk_max_size` to avoid sending partial batches to the output.
 
 This sample configuration forwards events to the output if 512 events are available or the oldest
 available event has been waiting for 5s in the queue:
 
 [source,yaml]
 ------------------------------------------------------------------------------
-queue.mem:
-  events: 4096
-  flush.min_events: 512
-  flush.timeout: 5s
+  queue.mem.events: 4096
+  queue.mem.flush.min_events: 512
+  queue.mem.flush.timeout: 5s
 ------------------------------------------------------------------------------
 
 [cols="2*<a"]
 |===
 | Setting | Description
 
-include::output-shared-settings.asciidoc[tag=events-setting]
+include::output-shared-settings.asciidoc[tag=queue.mem.events-setting]
 
 // =============================================================================
 
-include::output-shared-settings.asciidoc[tag=flush.min_events-setting]
+include::output-shared-settings.asciidoc[tag=queue.mem.flush.min_events-setting]
 
 // =============================================================================
 
-include::output-shared-settings.asciidoc[tag=flush.timeout-setting]
+include::output-shared-settings.asciidoc[tag=queue.mem.flush.timeout-setting]
 
 // =============================================================================
 

docs/en/ingest-management/elastic-agent/configuration/outputs/output-logstash.asciidoc

@@ -173,42 +173,41 @@ The memory queue waits for the output to acknowledge or drop events. If
 the queue is full, no new events can be inserted into the memory queue. Only
 after the signal from the output will the queue free up space for more events to be accepted.
 
-The memory queue is controlled by the parameters `flush.min_events` and `flush.timeout`. If
-`flush.timeout` is `0s` or `flush.min_events` is `0` or `1` then events can be sent by the output as
+The memory queue is controlled by the parameters `queue.mem.flush.min_events` and `queue.mem.flush.timeout`. If
+`queue.mem.flush.timeout` is `0s` or `queue.mem.flush.min_events` is `0` or `1` then events can be sent by the output as
 soon as they are available. If the output supports a `bulk_max_size` parameter it controls the
 maximum batch size that can be sent.
 
-If `flush.min_events` is greater than `1` and `flush.timeout` is greater than `0s`, events will only
-be sent to the output when the queue contains at least `flush.min_events` events or the
-`flush.timeout` period has expired. In this mode the maximum size batch that that can be sent by the
-output is `flush.min_events`. If the output supports a `bulk_max_size` parameter, values of
-`bulk_max_size` greater than `flush.min_events` have no effect. The value of `flush.min_events`
+If `queue.mem.flush.min_events` is greater than `1` and `queue.mem.flush.timeout` is greater than `0s`, events will only
+be sent to the output when the queue contains at least `queue.mem.flush.min_events` events or the
+`queue.mem.flush.timeout` period has expired. In this mode the maximum size batch that that can be sent by the
+output is `queue.mem.flush.min_events`. If the output supports a `bulk_max_size` parameter, values of
+`bulk_max_size` greater than `queue.mem.flush.min_events` have no effect. The value of `queue.mem.flush.min_events`
 should be evenly divisible by `bulk_max_size` to avoid sending partial batches to the output.
 
 This sample configuration forwards events to the output if 512 events are available or the oldest
 available event has been waiting for 5s in the queue:
 
 [source,yaml]
 ------------------------------------------------------------------------------
-queue.mem:
-  events: 4096
-  flush.min_events: 512
-  flush.timeout: 5s
+  queue.mem.events: 4096
+  queue.mem.flush.min_events: 512
+  queue.mem.flush.timeout: 5s
 ------------------------------------------------------------------------------
 
 [cols="2*<a"]
 |===
 | Setting | Description
 
-include::output-shared-settings.asciidoc[tag=events-setting]
+include::output-shared-settings.asciidoc[tag=queue.mem.events-setting]
 
 // =============================================================================
 
-include::output-shared-settings.asciidoc[tag=flush.min_events-setting]
+include::output-shared-settings.asciidoc[tag=queue.mem.flush.min_events-setting]
 
 // =============================================================================
 
-include::output-shared-settings.asciidoc[tag=flush.timeout-setting]
+include::output-shared-settings.asciidoc[tag=queue.mem.flush.timeout-setting]
 
 // =============================================================================
 

docs/en/ingest-management/elastic-agent/configuration/outputs/output-shared-settings.asciidoc

@@ -43,39 +43,39 @@ escaping.
 
 // =============================================================================
 
-// tag::events-setting[]
+// tag::queue.mem.events-setting[]
 |
-[id="{type}-events-setting"]
-`events`
+[id="{type}-queue.mem.events-setting"]
+`queue.mem.events`
 
-| The number of events the queue can store. This value should be evenly divisible by `flush.min_events` to avoid sending partial batches to the output.
+| The number of events the queue can store. This value should be evenly divisible by `queue.mem.flush.min_events` to avoid sending partial batches to the output.
 
 *Default:* `3200 events`
-// end::events-setting[]
+// end::queue.mem.events-setting[]
 
 // =============================================================================
 
-// tag::flush.min_events-setting[]
+// tag::queue.mem.flush.min_events-setting[]
 |
-[id="{type}-flush.min_events-setting"]
-`flush.min_events`
+[id="{type}-queue.mem.flush.min_events-setting"]
+`queue.mem.flush.min_events`
 
-| The minimum number of events required for publishing. If this value is set to 0 or 1, events are available to the output immediately. If this value is greater than 1 the output must wait for the queue to accumulate this minimum number of events or for `flush.timeout`` to expire before publishing. When greater than 1 this value also defines the maximum possible batch that can be sent by the output.
+| The minimum number of events required for publishing. If this value is set to 0 or 1, events are available to the output immediately. If this value is greater than 1 the output must wait for the queue to accumulate this minimum number of events or for `queue.mem.flush.timeout` to expire before publishing. When greater than 1 this value also defines the maximum possible batch that can be sent by the output.
 
 *Default:* `1600 events`
-// end::flush.min_events-setting[]
+// end::queue.mem.flush.min_events-setting[]
 
 // =============================================================================
 
-// tag::flush.timeout-setting[]
+// tag::queue.mem.flush.timeout-setting[]
 |
-[id="{type}-flush.timeout-setting"]
-`flush.timeout`
+[id="{type}-queue.mem.flush.timeout-setting"]
+`queue.mem.flush.timeout`
 
-| (int) The maximum wait time for `flush.min_events` to be fulfilled. If set to 0s, events are available to the output immediately.
+| (int) The maximum wait time for `queue.mem.flush.min_events` to be fulfilled. If set to 0s, events are available to the output immediately.
 
 *Default:* `10s`
-// end::flush.timeout-setting[]
+// end::queue.mem.flush.timeout-setting[]
 
 // =============================================================================
 

docs/en/ingest-management/fleet/fleet-settings-output-elasticsearch.asciidoc

@@ -89,6 +89,31 @@ output is set in the <<agent-policy,agent policy>>.
 | Choose one of the menu options to tune your {agent} performance when sending data to an {es} output. You can optimize for throughput, scale, latency, or you can choose a balanced (the default) set of performance specifications. Refer to <<es-output-settings-performance-tuning-settings,Performance tuning settings>> for details about the setting values and their potential impact on performance.
 
 You can also use the <<es-output-settings-yaml-config,Advanced YAML configuration>> field to set custom values. Note that if you adjust any of the performance settings described in the following **Advanced YAML configuration** section, the **Performance tuning** option automatically changes to `Custom` and cannot be changed.
+
+Performance tuning preset values take precedence over any settings that may be defined separately. If you want to change any setting, you need to use the `Custom` **Performance tuning** option and specify the settings in the **Advanced YAML configuration** field.
+
+For example, if you would like to use the balanced preset values except that you prefer a higher compression level, you can do so as follows:
+
+. In {fleet}, open the **Settings** tab.
+. In the **Outputs** section, select **Add output** to create a new output, or select the edit icon to edit an existing output.
+. In the **Add new output** or the **Edit output** flyout, set **Performance tuning** to `Custom`.
+. Refer to the list of <<es-output-settings-performance-tuning-settings,performance tuning preset values>>, and add the settings you prefer into the **Advanced YAML configuration** field. For the `balanced` presets, the yaml configuration would be as shown:
++
+[source,yaml]
+----
+bulk_max_size: 1600
+worker: 1
+queue.mem.events: 3200
+queue.mem.flush.min_events: 1600
+queue.mem.flush.timeout: 10s
+compression_level: 1
+connection_idle_timeout: 3s
+----
+
+. Adjust any settings as preferred. For example, you can update the `compression_level` setting to `4`.
+
+When you create an {agent} policy using this output, the output will use the balanced preset options except with the higher compression level, as specified.
+
 |===
 
 [[es-output-settings-yaml-config]]
@@ -114,20 +139,19 @@ include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[
 
 // =============================================================================
 
-include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=events-setting]
+include::../elastic-agent/configuration/outputs/output-elasticsearch.asciidoc[tag=max_retries-setting]
 
 // =============================================================================
 
-include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=flush.min_events-setting]
+include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=queue.mem.events-setting]
 
 // =============================================================================
 
-include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=flush.timeout-setting]
+include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=queue.mem.flush.min_events-setting]
 
 // =============================================================================
 
-include::../elastic-agent/configuration/outputs/output-elasticsearch.asciidoc[tag=max_retries-setting]
-
+include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=queue.mem.flush.timeout-setting]
 
 // =============================================================================
 
@@ -166,32 +190,32 @@ include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[
 |3200
 |4100
 
-|`flush.min_events`
+|`queue.mem.flush.min_events`
 |1600
 |1600
 |1600
 |2050
 
-|`flush.timeout`
+|`queue.mem.flush.timeout`
 |10
 |5
 |20
 |1
 
-|`compression`
+|`compression_level`
 |1
 |1
 |1
 |1
 
-|`idle_timeout`
+|`connection_idle_timeout`
 |3
 |15
 |1
 |60
 |===
 
-For descriptions of each setting, refer to <<es-output-settings-yaml-config,Advanced YAML configuration>>. For the  `queue.mem.events`, `flush.min_events` and `flush.timeout` settings, refer to the {filebeat-ref}/configuring-internal-queue.html[internal queue configuration settings] in the {filebeat} documentation.
+For descriptions of each setting, refer to <<es-output-settings-yaml-config,Advanced YAML configuration>>. For the  `queue.mem.events`, `queue.mem.flush.min_events` and `queue.mem.flush.timeout` settings, refer to the {filebeat-ref}/configuring-internal-queue.html[internal queue configuration settings] in the {filebeat} documentation.
 
 `Balanced` represents the new default setting (out of the box behaviour). Relative to `Balanced`, `Optimized for throughput` setting will improve EPS by 4 times, `Optimized for Scale` will perform on par and `Optimized for Latency` will show a 20% degredation in EPS (Events Per Second). These relative performance numbers were calculated from a performance testbed which operates in a controlled setting ingesting a large log file. 
 
@@ -203,7 +227,7 @@ These presets apply only to agents on version 8.12.0 or later.
 .Performance tuning: EPS data
 [cols="1,1,1,1,1,1,1"]
 |===
-|worker |bulk_max_size |queue.mem_events |flush.min_events |flush.timeout |idle_timeout |Relative EPS
+|worker |bulk_max_size |queue.mem_events |queue.mem.flush.min_events |queue.mem.flush.timeout |connection_idle_timeout |Relative EPS
 
 |1
 |1600

docs/en/ingest-management/fleet/fleet-settings-output-kafka.asciidoc

@@ -511,18 +511,6 @@ output.console:
 
 // =============================================================================
 
-include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=events-setting]
-
-// =============================================================================
-
-include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=flush.min_events-setting]
-
-// =============================================================================
-
-include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=flush.timeout-setting]
-
-// =============================================================================
-
 // tag::keep_alive-setting[]
 |
 [id="{type}-keep_alive-setting"]
@@ -569,6 +557,18 @@ is false.
 
 // =============================================================================
 
+include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=queue.mem.events-setting]
+
+// =============================================================================
+
+include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=queue.mem.flush.min_events-setting]
+
+// =============================================================================
+
+include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=queue.mem.flush.timeout-setting]
+
+// =============================================================================
+
 |===
 
 :type!:

docs/en/ingest-management/fleet/fleet-settings-output-logstash.asciidoc

@@ -142,18 +142,6 @@ include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[
 
 // =============================================================================
 
-include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=events-setting]
-
-// =============================================================================
-
-include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=flush.min_events-setting]
-
-// =============================================================================
-
-include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=flush.timeout-setting]
-
-// =============================================================================
-
 // tag::index-setting[]
 |
 [id="{type}-index-setting"]
@@ -180,6 +168,18 @@ include::../elastic-agent/configuration/outputs/output-logstash.asciidoc[tag=pro
 
 // =============================================================================
 
+include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=queue.mem.events-setting]
+
+// =============================================================================
+
+include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=queue.mem.flush.min_events-setting]
+
+// =============================================================================
+
+include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=queue.mem.flush.timeout-setting]
+
+// =============================================================================
+
 include::../elastic-agent/configuration/outputs/output-logstash.asciidoc[tag=slow_start-setting]
 
 // =============================================================================