diff --git a/content/chapter-destinations/configuring-destinations-elasticsearch-http/_index.md b/content/chapter-destinations/configuring-destinations-elasticsearch-http/_index.md index d651bff9..025b7c6f 100644 --- a/content/chapter-destinations/configuring-destinations-elasticsearch-http/_index.md +++ b/content/chapter-destinations/configuring-destinations-elasticsearch-http/_index.md @@ -15,7 +15,7 @@ HTTPS connection, as well as password- and certificate-based authentication is s d_elasticsearch_http { elasticsearch-http( index("") - url("https://your-elasticsearch-server1:9200/_bulk" "https://your-elasticsearch-server2:9200/_bulk") + url("https://your-elasticsearch-server1:9200/_bulk") type("") ); }; @@ -35,7 +35,7 @@ HTTPS connection, as well as password- and certificate-based authentication is s ## Example: Sending log data to Elasticsearch {#example-destination-elasticsearch-http} -The following example defines a `elasticsearch-http()` destination, with only the required options. +The following example defines an `elasticsearch-http()` destination, with only the required options. ```shell destination d_elasticsearch_http { diff --git a/content/chapter-destinations/configuring-destinations-elasticsearch-http/elasticsearch-http-batch-mode/_index.md b/content/chapter-destinations/configuring-destinations-elasticsearch-http/elasticsearch-http-batch-mode/_index.md index 225ed1fe..9682dfdb 100644 --- a/content/chapter-destinations/configuring-destinations-elasticsearch-http/elasticsearch-http-batch-mode/_index.md +++ b/content/chapter-destinations/configuring-destinations-elasticsearch-http/elasticsearch-http-batch-mode/_index.md @@ -15,10 +15,10 @@ In the following example, a batch consists of 100 messages, or a maximum of 512 ```shell destination d_elasticsearch-http { - elasticsearch-http(url("http://your-elasticsearch-server:9200/_bulk") + elasticsearch-http( + url("http://your-elasticsearch-server:9200/_bulk") index("") type("") - url("http://your-elasticsearch-server:9200/_bulk") batch-lines(100) batch-bytes(512Kb) batch-timeout(10000) @@ -26,9 +26,6 @@ In the following example, a batch consists of 100 messages, or a maximum of 512 }; ``` - - - ## Load balancing between multiple Elasticsearch indexers {#elasticsearch-http-load-balancing} {{< include-headless "chunk/destination-load-balancing-url.md" >}} @@ -50,5 +47,3 @@ The following destination sends log messages to 3 different Elasticsearch indexe ``` {{% include-headless "chunk/http-load-balance-workers.md" %}} - - diff --git a/content/chapter-destinations/destination-opensearch/_index.md b/content/chapter-destinations/destination-opensearch/_index.md index e07e5cbd..a123c334 100644 --- a/content/chapter-destinations/destination-opensearch/_index.md +++ b/content/chapter-destinations/destination-opensearch/_index.md @@ -22,12 +22,9 @@ HTTPS connection, as well as password- and certificate-based authentication is s }; ``` -{{< include-headless "chunk/option-dest-http-proxy.md" >}} +## Example: Sending log data to OpenSearch {#example-destination-opensearch} - -## Example: Sending log data to Elasticsearch {#example-destination-elasticsearch-http} - -The following example defines a `elasticsearch-http()` destination, with only the required options. +The following example defines an `opensearch()` destination, with only the required options. ```shell destination opensearch { @@ -45,7 +42,7 @@ The following example defines a `elasticsearch-http()` destination, with only th }; ``` -The following example uses mutually-authenticated HTTPS connection, templated index, and also sets the `type()` and some other options. +The following example uses mutually-authenticated HTTPS connection, templated index, and also sets some other options. ```shell destination opensearch_https { @@ -53,7 +50,6 @@ The following example uses mutually-authenticated HTTPS connection, templated in url("https://node01.example.com:9200/_bulk") index("test-${YEAR}${MONTH}${DAY}") time-zone("UTC") - type("test") workers(4) batch-lines(16) timeout(10) diff --git a/content/chapter-destinations/destination-opensearch/batch-mode/_index.md b/content/chapter-destinations/destination-opensearch/batch-mode/_index.md new file mode 100644 index 00000000..640bc5da --- /dev/null +++ b/content/chapter-destinations/destination-opensearch/batch-mode/_index.md @@ -0,0 +1,47 @@ +--- +title: "Batch mode and load balancing" +weight: 100 +--- + + +The `opensearch()` destination automatically sends multiple log messages in a single HTTP request, increasing the rate of messages that your Elasticsearch deployment can consume. For details on adjusting and fine-tuning the batch mode, see the following section. + +{{% include-headless "chunk/concept-batch-size.md" %}} + +## Example: HTTP batch mode {#example-elasticsearch-http-batch-mode} + +In the following example, a batch consists of 100 messages, or a maximum of 512 kilobytes, and is sent every 20 seconds (20000 milliseconds). + +```shell + destination d_opensearch { + opensearch( + url("http://your-server:9200/_bulk") + index("") + batch-lines(100) + batch-bytes(512Kb) + batch-timeout(10000) + ); + }; +``` + +## Load balancing between multiple indexers + +{{< include-headless "chunk/destination-load-balancing-url.md" >}} + +## Example: HTTP load balancing + +The following destination sends log messages to 3 different indexer nodes. Each node is assigned a separate worker thread. A batch consists of 100 messages, or a maximum of 512 kilobytes, and is sent every 20 seconds (20000 milliseconds). + +```shell + destination d_opensearch { + opensearch( + url("http://your-elasticsearch-server1:9200/_bulk" "http://your-elasticsearch-server2:9200/_bulk" "http://your-elasticsearch-server3:9200/_bulk") + batch-lines(100) + batch-bytes(512Kb) + batch-timeout(20000) + persist-name("opensearch-load-balance") + ); + }; +``` + +{{% include-headless "chunk/http-load-balance-workers.md" %}} diff --git a/content/chapter-destinations/destination-opensearch/reference-destination-opensearch/_index.md b/content/chapter-destinations/destination-opensearch/reference-destination-opensearch/_index.md index 4ef13da0..1a48fa61 100644 --- a/content/chapter-destinations/destination-opensearch/reference-destination-opensearch/_index.md +++ b/content/chapter-destinations/destination-opensearch/reference-destination-opensearch/_index.md @@ -10,7 +10,7 @@ This destination is available in {{% param "product.abbrev" %}} version 4.4 and {{% include-headless "chunk/option-destination-batch-bytes.md" %}} -For details on how this option influences batch mode, see {{% xref "/chapter-destinations/configuring-destinations-elasticsearch-http/elasticsearch-http-batch-mode/_index.md" %}} +For details on how this option influences batch mode, see {{% xref "/chapter-destinations/destination-opensearch/batch-mode/_index.md" %}}. ## batch-lines() @@ -21,23 +21,34 @@ For details on how this option influences batch mode, see {{% xref "/chapter-des {{% include-headless "chunk/option-description-destination-batch-lines.md" %}} -For details on how this option influences batch mode, see {{% xref "/chapter-destinations/configuring-destinations-elasticsearch-http/elasticsearch-http-batch-mode/_index.md" %}} - - +For details on how this option influences batch mode, see {{% xref "/chapter-destinations/destination-opensearch/batch-mode/_index.md" %}}. {{% include-headless "chunk/option-destination-batch-timeout.md" %}} -For details on how this option influences batch mode, see {{% xref "/chapter-destinations/configuring-destinations-elasticsearch-http/elasticsearch-http-batch-mode/_index.md" %}} - - +For details on how this option influences batch mode, see {{% xref "/chapter-destinations/destination-opensearch/batch-mode/_index.md" %}}. {{% include-headless "chunk/option-destination-tls-ca-dir.md" %}} {{% include-headless "chunk/topic-tls-block-http.md" %}} -{{% include-headless "chunk/example-tls-block-elasticsearch-http-ca-dir.md" %}} - - +### Declaration: + +```shell + destination d_opensearch { + opensearch( + url("http://your-server:9200/_bulk") + index("example-index") + tls( + ca-dir("dir") + cert-file("cert") + cipher-suite("cipher") + key-file("key") + peer-verify(yes|no) + ssl-version() + ) + ); + }; +``` ## ca-file() {#opensearch-options-ca-file} @@ -50,25 +61,36 @@ For details on how this option influences batch mode, see {{% xref "/chapter-des {{% include-headless "chunk/topic-tls-block-http.md" %}} -{{% include-headless "chunk/example-tls-block-elasticsearch-http-ca-file.md" %}} - - +### Declaration: + +```shell + destination d_opensearch { + opensearch( + url("http://your-server:9200/_bulk") + index("example-index") + tls( + ca-file("ca") + cert-file("cert") + cipher-suite("cipher") + key-file("key") + peer-verify(yes|no) + ssl-version() + ) + ); + }; +``` {{% include-headless "chunk/option-destination-tls-cert-file.md" %}} {{% include-headless "chunk/topic-tls-block-http.md" %}} -{{% include-headless "chunk/example-tls-block-elasticsearch-http.md" %}} - - +{{% include-headless "chunk/example-tls-block-opensearch.md" %}} {{% include-headless "chunk/option-destination-tls-cipher-suite.md" %}} {{% include-headless "chunk/topic-tls-block-http.md" %}} -{{% include-headless "chunk/example-tls-block-elasticsearch-http.md" %}} - - +{{% include-headless "chunk/example-tls-block-opensearch.md" %}} ## custom-id() {#opensearch-options-custom-id} @@ -79,8 +101,6 @@ For details on how this option influences batch mode, see {{% xref "/chapter-des *Description:* Sets the specified value as the ID of the OpenSearch index (`_id`). - - ## delimiter() {#opensearch-options-delimiter} | | | @@ -90,14 +110,12 @@ For details on how this option influences batch mode, see {{% xref "/chapter-des *Description:* By default, {{% param "product.abbrev" %}} separates the log messages of the batch with a newline character. You can specify a different delimiter by using the `delimiter()` option. -For details on how this option influences batch mode, see {{% xref "/chapter-destinations/configuring-destinations-elasticsearch-http/elasticsearch-http-batch-mode/_index.md" %}} - +For details on how this option influences batch mode, see {{% xref "/chapter-destinations/destination-opensearch/batch-mode/_index.md" %}}. {{< include-headless "chunk/option-destination-diskbuffer.md" >}} {{< include-headless "chunk/option-destination-hook.md" >}} - ## index() | | | @@ -119,7 +137,7 @@ This destination supports only unencrypted key files (that is, the private key c {{% include-headless "chunk/topic-tls-block-http.md" %}} -{{% include-headless "chunk/example-tls-block-elasticsearch-http.md" %}} +{{% include-headless "chunk/example-tls-block-opensearch.md" %}} @@ -132,30 +150,23 @@ This destination supports only unencrypted key files (that is, the private key c *Description:* The password that {{% param "product.abbrev" %}} uses to authenticate on the server where it sends the messages. - - {{< include-headless "chunk/option-peer-verify-simple.md" >}} {{% include-headless "chunk/topic-tls-block-http.md" %}} -{{% include-headless "chunk/example-tls-block-elasticsearch-http.md" %}} - +{{% include-headless "chunk/example-tls-block-opensearch.md" %}} {{% include-headless "chunk/option-persist-name.md" %}} -{{% include-headless "chunk/option-destination-http-proxy.md" %}} - - +{{< include-headless "chunk/option-dest-http-proxy.md" >}} {{% include-headless "chunk/option-destination-retries.md" %}} To handle HTTP error responses, if the HTTP server returns 5xx codes, {{% param "product.abbrev" %}} will attempt to resend messages until the number of attempts reaches `retries`. If the HTTP server returns 4xx codes, {{% param "product.abbrev" %}} will drop the messages. - - ## ssl-version() {#opensearch-options-ssl-version} | | | @@ -167,15 +178,12 @@ To handle HTTP error responses, if the HTTP server returns 5xx codes, {{% param {{% include-headless "chunk/topic-tls-block-http.md" %}} -{{% include-headless "chunk/example-tls-block-elasticsearch-http.md" %}} - +{{% include-headless "chunk/example-tls-block-opensearch.md" %}} {{% include-headless "chunk/option-destination-throttle.md" %}} - {{% include-headless "chunk/option-source-time-reopen.md" %}} - ## timeout() {#opensearch-options-timeout} | | | @@ -185,8 +193,6 @@ To handle HTTP error responses, if the HTTP server returns 5xx codes, {{% param *Description:* The value (in seconds) to wait for an operation to complete, and attempt to reconnect the server if exceeded. - - ## url() | | | @@ -194,7 +200,7 @@ To handle HTTP error responses, if the HTTP server returns 5xx codes, {{% param | Type: | URL or list of URLs, for example, url("site1" "site2") | | Default: | N/A | -*Description:* Specifies the hostname or IP address and optionally the port number of the OpenSearch indexer. Use a colon (`:`) after the address to specify the port number of the server. For example: `http://your-OpenSearch-indexer.server:8088/_bulk` +*Description:* Specifies the hostname or IP address and optionally the port number of the OpenSearch indexer. Use a colon (`:`) after the address to specify the port number of the server. For example: `http://your-opensearch-indexer.server:8088/_bulk` This option is mandatory for this destination. @@ -204,8 +210,6 @@ In case the server on the specified URL returns a redirect request, {{% param "p {{< include-headless "chunk/destination-load-balancing-url.md" >}} - - ## user() | | | @@ -215,10 +219,8 @@ In case the server on the specified URL returns a redirect request, {{% param "p *Description:* The username that {{% param "product.abbrev" %}} uses to authenticate on the server where it sends the messages. - {{% include-headless "chunk/option-destination-http-use-system-cert-store.md" %}} - ## workers() | | | diff --git a/content/headless/chunk/concept-batch-size.md b/content/headless/chunk/concept-batch-size.md index 103e1ddb..8b97a89a 100644 --- a/content/headless/chunk/concept-batch-size.md +++ b/content/headless/chunk/concept-batch-size.md @@ -6,9 +6,8 @@ The `batch-bytes()`, `batch-lines()`, and `batch-timeout()` options of the destination determine how many log messages {{% param "product.abbrev" %}} sends in a batch. The `batch-lines()` option determines the maximum number of messages {{% param "product.abbrev" %}} puts in a batch in. This can be limited based on size and time: - - {{% param "product.abbrev" %}} sends a batch every `batch-timeout()` milliseconds, even if the number of messages in the batch is less than `batch-lines()`. That way the destination receives every message in a timely manner even if suddenly there are no more messages. +- {{% param "product.abbrev" %}} sends a batch every `batch-timeout()` milliseconds, even if the number of messages in the batch is less than `batch-lines()`. That way the destination receives every message in a timely manner even if suddenly there are no more messages. - - {{% param "product.abbrev" %}} sends the batch if the total size of the messages in the batch reaches `batch-bytes()` bytes. +- {{% param "product.abbrev" %}} sends the batch if the total size of the messages in the batch reaches `batch-bytes()` bytes. To increase the performance of the destination, increase the number of worker threads for the destination using the `workers()` option, or adjust the `batch-bytes()`, `batch-lines()`, `batch-timeout()` options. - diff --git a/content/headless/chunk/example-tls-block-opensearch.md b/content/headless/chunk/example-tls-block-opensearch.md new file mode 100644 index 00000000..a05a7489 --- /dev/null +++ b/content/headless/chunk/example-tls-block-opensearch.md @@ -0,0 +1,25 @@ +--- +--- + + +### Declaration: + +```shell + destination d_elasticsearch-http { + elasticsearch-http( + url("http://your-elasticsearch-server:9200/_bulk") + type("") + index("example-index") + tls( + ca-dir("dir") + ca-file("ca") + cert-file("cert") + cipher-suite("cipher") + key-file("key") + peer-verify(yes|no) + ssl-version() + ) + ); + }; +``` +