Merge pull request #17 from axoflow/otel-dest-batch-and-workers-options

OpenTelemetry destination: add batch and workers options
axoflow · Sep 4, 2023 · 5b63d7d · 5b63d7d
2 parents 899a407 + 3ae32fa
commit 5b63d7d
Show file tree

Hide file tree

Showing 18 changed files with 42 additions and 93 deletions.
diff --git a/...destinations/configuring-destinations-amqp/reference-destination-amqp/_index.md b/...destinations/configuring-destinations-amqp/reference-destination-amqp/_index.md
@@ -25,9 +25,7 @@ The `amqp()` destination has the following options:
 
 {{% include-headless "chunk/option-destination-batch-bytes.md" %}}
 
-{{% include-headless "chunk/option-destination-batch-lines.md" %}}
-
-{{% include-headless "chunk/option-destination-batch-timeout.md" %}}
+{{% include-headless "chunk/option-destination-threaded-batching.md" %}}
 
 
 ## body() {#amqp-option-body}

diff --git a/...tinations-elasticsearch-http/reference-destination-elasticsearch-http/_index.md b/...tinations-elasticsearch-http/reference-destination-elasticsearch-http/_index.md
@@ -243,3 +243,4 @@ In case the server on the specified URL returns a redirect request, {{% param "p
 
 {{< include-headless "chunk/option-destination-description-workers.md" >}}
 
+{{% include-headless "chunk/http-load-balance-workers.md" %}}
diff --git a/...figuring-destinations-http-nonjava/reference-destination-http-nonjava/_index.md b/...figuring-destinations-http-nonjava/reference-destination-http-nonjava/_index.md
@@ -12,19 +12,11 @@ The `http` destination of {{% param "product.abbrev" %}} can directly post log m
 For details on how this option influences HTTP batch mode, see [http: Posting messages over HTTP without Java]({{< relref "/chapter-destinations/configuring-destinations-http-nonjava/_index.md" >}})
 
 
-
-{{% include-headless "chunk/option-destination-batch-lines.md" %}}
+{{% include-headless "chunk/option-destination-threaded-batching.md" %}}
 
 For details on how this option influences HTTP batch mode, see [http: Posting messages over HTTP without Java]({{< relref "/chapter-destinations/configuring-destinations-http-nonjava/_index.md" >}})
 
 
-
-{{% include-headless "chunk/option-d-b-t-http-spl-sent-pub.md" %}}
-
-For details on how this option influences HTTP batch mode, see [http: Posting messages over HTTP without Java]({{< relref "/chapter-destinations/configuring-destinations-http-nonjava/_index.md" >}})
-
-
-
 ## body()
 
 |          |                    |
@@ -344,4 +336,7 @@ In case the server on the specified URL returns a redirect request, {{% param "p
 
 {{% include-headless "chunk/option-destination-http-use-system-cert-store.md" %}}
 
-{{< include-headless "chunk/option-destination-http-workers.md" >}}
+{{< include-headless "chunk/option-destination-threaded-workers.md" >}}
+
+{{% include-headless "chunk/http-load-balance-workers.md" %}}
+
diff --git a/...ations/configuring-destinations-kafka-c/reference-destination-kafka-c/_index.md b/...ations/configuring-destinations-kafka-c/reference-destination-kafka-c/_index.md
@@ -18,7 +18,7 @@ The following options are required: `bootstrap-servers()`, `topic()`. Note that
 {{< include-headless "chunk/kafka-c-impl-required-options-note.md" >}}
 
 
-{{% include-headless "chunk/option-destination-batch-lines.md" %}}
+{{% include-headless "chunk/option-destination-threaded-batching.md" %}}
 
 {{< include-headless "wnt/n-options-only-effective-for-sync-send(yes).md" >}}
 
@@ -28,12 +28,6 @@ If you set `sync-send()` to `"yes"`, the number you specify for `batch-lines()`
 
 {{% /alert %}}
 
-
-
-{{% include-headless "chunk/option-destination-batch-timeout.md" %}}
-
-{{< include-headless "wnt/n-options-only-effective-for-sync-send(yes).md" >}}
-
 {{% alert title="Note" color="info" %}}
 
 When setting `batch-timeout()`, consider the value of the `transaction.timeout.ms` Kafka property. If in case of timeout (that is, if {{% param "product.abbrev" %}} does not receive `batch-lines()` amount of messages) the value of `batch-timeout()` exceeds the value of `transaction.timeout.ms`, {{% param "product.abbrev" %}} will not send out messages in time.
@@ -210,15 +204,13 @@ For details on formatting messages in JSON format, see [format-json]({{< relref
 {{< include-headless "chunk/option-destination-ts-format.md" >}}
 
 
-## workers() {#kafka-option-kafka-topic}
+{{< include-headless "chunk/option-destination-threaded-workers.md" >}} 
 
-|          |         |
-| -------- | ------- |
-| Type:    | integer |
-| Default: | 1       |
+{{% alert title="Note" color="info" %}}
 
-*Description:* The workers are only responsible for formatting the messages that need to be delivered to the Kafka clients. Configure this option only if your Kafka clients have many threads and they do not receive enough messages.
+The workers are only responsible for formatting the messages that need to be delivered to the Kafka clients. Configure this option only if your Kafka clients have many threads and they do not receive enough messages.
 
+{{% /alert %}}
 
 {{% alert title="Note" color="info" %}}
 

diff --git a/...ations/configuring-destinations-mongodb/reference-destination-mongodb/_index.md b/...ations/configuring-destinations-mongodb/reference-destination-mongodb/_index.md
@@ -63,9 +63,7 @@ Hazard of data loss! The {{% param "product.abbrev" %}} application does not ver
 
 {{% include-headless "chunk/option-destination-batch-bytes.md" %}}
 
-{{% include-headless "chunk/option-destination-batch-lines.md" %}}
-
-{{% include-headless "chunk/option-destination-batch-timeout.md" %}}
+{{% include-headless "chunk/option-destination-threaded-batching.md" %}}
 
 {{< include-headless "chunk/option-destination-frac-digits.md" >}}
 
@@ -102,18 +100,7 @@ For MongoDB operations, {{% param "product.abbrev" %}} uses a one-minute timeout
 
 {{< include-headless "wnt/note-typehinting.md" >}}
 
-
-## workers()
-
-|          |         |
-| -------- | ------- |
-| Type:    | integer |
-| Default: | 1       |
-
-
-*Description:* Specifies the number of worker threads (at least 1) that {{% param "product.abbrev" %}} uses to send messages to the server. Increasing the number of worker threads can drastically improve the performance of the destination.
-
-{{< include-headless "wnt/warning-diskbuffer-workers.md" >}}
+{{< include-headless "chunk/option-destination-threaded-workers.md" >}}
 
 ## write-concern() {#mongodb-option-write-concern}
 

diff --git a/...stinations/configuring-destinations-redis/reference-destination-redis/_index.md b/...stinations/configuring-destinations-redis/reference-destination-redis/_index.md
@@ -17,9 +17,8 @@ The `redis()` destination has the following options:
 
 *Description:* The password used for authentication on a password-protected Redis server. Available in {{% param "product.abbrev" %}} version 3.10 and later.
 
-{{% include-headless "chunk/option-destination-batch-lines.md" %}}
+{{% include-headless "chunk/option-destination-threaded-batching.md" %}}
 
-{{% include-headless "chunk/option-destination-batch-timeout.md" %}}
 
 ## command() {#redis-option-command}
 
@@ -42,9 +41,7 @@ Note the following points when using the `redis()` destination:
 
 {{% include-headless "chunk/option-destination-batch-bytes.md" %}}
 
-{{% include-headless "chunk/option-destination-batch-lines.md" %}}
-
-{{% include-headless "chunk/option-destination-batch-timeout.md" %}}
+{{% include-headless "chunk/option-destination-threaded-batching.md" %}}
 
 {{< include-headless "chunk/option-destination-hook.md" >}}
 
@@ -72,13 +69,5 @@ Note the following points when using the `redis()` destination:
 
 {{% include-headless "chunk/option-source-time-reopen.md" %}}
 
-
-## workers()
-
-|          |         |
-| -------- | ------- |
-| Type:    | integer |
-| Default: | 1       |
-
-{{< include-headless "chunk/option-destination-description-workers.md" >}}
+{{< include-headless "chunk/option-destination-threaded-workers.md" >}}
 
diff --git a/...ations/configuring-destinations-riemann/reference-destination-riemann/_index.md b/...ations/configuring-destinations-riemann/reference-destination-riemann/_index.md
@@ -97,13 +97,11 @@ Note that the time format parameter requires:
 {{% include-headless "chunk/option-destination-batch-bytes.md" %}}
 
 
-{{% include-headless "chunk/option-destination-batch-lines.md" %}}
+{{% include-headless "chunk/option-destination-threaded-batching.md" %}}
 
 If an error occurs while sending the messages to the server, {{% param "product.abbrev" %}} will try to resend every message from the batch. If it does not succeed (you can set the number of retry attempts in the `retries()` option), {{% param "product.abbrev" %}} drops every message in the batch.
 
 
-{{% include-headless "chunk/option-destination-batch-timeout.md" %}}
-
 {{< include-headless "chunk/option-destination-hook.md" >}}
 
 ## host() {#riemann-option-host}

diff --git a/...destinations/configuring-destinations-smtp/reference-destination-smtp/_index.md b/...destinations/configuring-destinations-smtp/reference-destination-smtp/_index.md
@@ -16,9 +16,7 @@ The `smtp()` sends email messages using SMTP, without needing external applicati
 
 {{% include-headless "chunk/option-destination-batch-bytes.md" %}}
 
-{{% include-headless "chunk/option-destination-batch-lines.md" %}}
-
-{{% include-headless "chunk/option-destination-batch-timeout.md" %}}
+{{% include-headless "chunk/option-destination-threaded-batching.md" %}}
 
 {{% include-headless "chunk/smtp-option-from.md" %}}
 

diff --git a/...r-destinations/configuring-destinations-sql/reference-destination-sql/_index.md b/...r-destinations/configuring-destinations-sql/reference-destination-sql/_index.md
@@ -8,9 +8,7 @@ This driver sends messages into an SQL database. The `sql()` destination has the
 
 {{% include-headless "chunk/option-destination-batch-bytes.md" %}}
 
-{{% include-headless "chunk/option-destination-batch-lines.md" %}}
-
-{{% include-headless "chunk/option-destination-batch-timeout.md" %}}
+{{% include-headless "chunk/option-destination-threaded-batching.md" %}}
 
 ## columns() {#sql-option-columns}
 

diff --git a/...stinations/configuring-destinations-stomp/reference-destination-stomp/_index.md b/...stinations/configuring-destinations-stomp/reference-destination-stomp/_index.md
@@ -39,9 +39,7 @@ The `stomp()` destination has the following options:
 
 {{% include-headless "chunk/option-destination-batch-bytes.md" %}}
 
-{{% include-headless "chunk/option-destination-batch-lines.md" %}}
-
-{{% include-headless "chunk/option-destination-batch-timeout.md" %}}
+{{% include-headless "chunk/option-destination-threaded-batching.md" %}}
 
 {{< include-headless "chunk/option-destination-hook.md" >}}
 

diff --git a/...hapter-destinations/destination-discord/reference-destination-discord/_index.md b/...hapter-destinations/destination-discord/reference-destination-discord/_index.md
@@ -24,18 +24,11 @@ For details on how this option influences HTTP batch mode, see [http: Posting me
 
 
 
-{{% include-headless "chunk/option-destination-batch-lines.md" %}}
+{{% include-headless "chunk/option-destination-threaded-batching.md" %}}
 
 For details on how this option influences HTTP batch mode, see [http: Posting messages over HTTP without Java]({{< relref "/chapter-destinations/configuring-destinations-http-nonjava/_index.md" >}})
 
 
-
-{{% include-headless "chunk/option-d-b-t-http-spl-sent-pub.md" %}}
-
-For details on how this option influences HTTP batch mode, see [http: Posting messages over HTTP without Java]({{< relref "/chapter-destinations/configuring-destinations-http-nonjava/_index.md" >}})
-
-
-
 {{% include-headless "chunk/option-destination-tls-ca-dir.md" %}}
 
 {{% include-headless "chunk/topic-tls-block-http.md" %}}
@@ -259,4 +252,6 @@ For more information, see [Discord: Rate Limits](https://discord.com/developers/
 
 {{% include-headless "chunk/option-destination-http-use-system-cert-store.md" %}}
 
-{{< include-headless "chunk/option-destination-http-workers.md" >}}
+{{< include-headless "chunk/option-destination-threaded-workers.md" >}}
+
+{{% include-headless "chunk/http-load-balance-workers.md" %}}
diff --git a/...nt/chapter-destinations/destination-slack/reference-destination-slack/_index.md b/...nt/chapter-destinations/destination-slack/reference-destination-slack/_index.md
@@ -46,18 +46,11 @@ For details on how this option influences batch mode, see [http: Posting message
 
 
 
-{{% include-headless "chunk/option-destination-batch-lines.md" %}}
+{{% include-headless "chunk/option-destination-threaded-batching.md" %}}
 
 For details on how this option influences batch mode, see [http: Posting messages over HTTP without Java]({{< relref "/chapter-destinations/configuring-destinations-http-nonjava/_index.md" >}})
 
 
-
-{{% include-headless "chunk/option-destination-batch-timeout.md" %}}
-
-For details on how this option influences batch mode, see [http: Posting messages over HTTP without Java]({{< relref "/chapter-destinations/configuring-destinations-http-nonjava/_index.md" >}})
-
-
-
 {{% include-headless "chunk/option-destination-tls-ca-dir.md" %}}
 
 {{% include-headless "chunk/topic-tls-block-http.md" %}}
@@ -248,4 +241,6 @@ To handle HTTP error responses, if the HTTP server returns 5xx codes, {{% param
 
 {{% include-headless "chunk/option-destination-http-use-system-cert-store.md" %}}
 
-{{< include-headless "chunk/option-destination-http-workers.md" >}}
+{{< include-headless "chunk/option-destination-threaded-workers.md" >}}
+
+{{% include-headless "chunk/http-load-balance-workers.md" %}}
diff --git a/...ntro/destination-sumologic-options/destination-sumologic-options-http/_index.md b/...ntro/destination-sumologic-options/destination-sumologic-options-http/_index.md
@@ -10,7 +10,7 @@ In addition, the `sumologic-http()` destination also has the following options.
 
 {{% include-headless "chunk/option-destination-batch-bytes.md" %}}
 
-{{% include-headless "chunk/option-destination-batch-lines.md" %}}
+{{% include-headless "chunk/option-destination-threaded-batching.md" %}}
 
 {{% include-headless "chunk/option-destination-tls-ca-dir.md" %}}
 

diff --git a/content/chapter-destinations/opentelemetry/_index.md b/content/chapter-destinations/opentelemetry/_index.md
@@ -58,7 +58,9 @@ log non_otel_to_otel_tls {
 
 {{< include-headless "chunk/opentelemetry-authentication.md" >}}
 
+{{< include-headless "chunk/option-destination-threaded-batching.md" >}}
+
+{{< include-headless "chunk/option-destination-threaded-workers.md" >}}
+
 <!-- FIXME xinclude other common options
- threaded_dest_driver_general_option
-  | threaded_dest_driver_batch_option
-  | threaded_dest_driver_workers_option -->
+ threaded_dest_driver_general_option-->
diff --git a/content/headless/chunk/option-destination-description-workers.md b/content/headless/chunk/option-destination-description-workers.md
@@ -5,6 +5,3 @@
 *Description:* Specifies the number of worker threads (at least 1) that {{% param "product.abbrev" %}} uses to send messages to the server. Increasing the number of worker threads can drastically improve the performance of the destination.
 
 {{< include-headless "wnt/warning-diskbuffer-workers.md" >}}
-
-{{% include-headless "chunk/http-load-balance-workers.md" %}}
-
diff --git a/content/headless/chunk/option-destination-threaded-batching.md b/content/headless/chunk/option-destination-threaded-batching.md
@@ -0,0 +1,6 @@
+---
+---
+
+{{% include-headless "chunk/option-destination-batch-lines.md" %}}
+
+{{% include-headless "chunk/option-destination-batch-timeout.md" %}}
diff --git a/.../chunk/option-destination-http-workers.md → ...nk/option-destination-threaded-workers.md b/.../chunk/option-destination-http-workers.md → ...nk/option-destination-threaded-workers.md
diff --git a/content/headless/wnt/n-options-only-effective-for-sync-send(yes).md b/content/headless/wnt/n-options-only-effective-for-sync-send(yes).md
@@ -3,6 +3,6 @@
 <!-- DISCLAIMER: This file is based on the syslog-ng Open Source Edition documentation https://github.com/balabit/syslog-ng-ose-guides/commit/2f4a52ee61d1ea9ad27cb4f3168b95408fddfdf2 and is used under the terms of The syslog-ng Open Source Edition Documentation License. The file has been modified by Axoflow. -->
 {{% alert title="Note" color="info" %}}
 
-The {{% param "product.abbrev" %}} configuration accepts this option with `sync-send()` set to both `"yes"` or `"no"`, but the option will only take effect if you set `sync-send()` to `"yes"`.
+The {{% param "product.abbrev" %}} configuration accepts `batch-lines()` with `sync-send()` set to both `"yes"` or `"no"`, but the option will only take effect if you set `sync-send()` to `"yes"`.
 
 {{% /alert %}}
Original file line number	Diff line number	Diff line change
Expand Up		@@ -243,3 +243,4 @@ In case the server on the specified URL returns a redirect request, {{% param "p

		{{< include-headless "chunk/option-destination-description-workers.md" >}}

		{{% include-headless "chunk/http-load-balance-workers.md" %}}
Original file line number	Diff line number	Diff line change
Expand Up		@@ -5,6 +5,3 @@
		Description: Specifies the number of worker threads (at least 1) that {{% param "product.abbrev" %}} uses to send messages to the server. Increasing the number of worker threads can drastically improve the performance of the destination.

		{{< include-headless "wnt/warning-diskbuffer-workers.md" >}}

		{{% include-headless "chunk/http-load-balance-workers.md" %}}