From cc3896bb4a4020c04538c56600c6d012fe6bf4a6 Mon Sep 17 00:00:00 2001 From: Eldar Nash Date: Sat, 30 Sep 2023 16:58:09 +0200 Subject: [PATCH] [4.4][opensearch] Adds initial draft docs --- .../destination-opensearch/_index.md | 70 ++++++ .../_index.md | 231 ++++++++++++++++++ 2 files changed, 301 insertions(+) create mode 100644 content/chapter-destinations/destination-opensearch/_index.md create mode 100644 content/chapter-destinations/destination-opensearch/reference-destination-opensearch/_index.md diff --git a/content/chapter-destinations/destination-opensearch/_index.md b/content/chapter-destinations/destination-opensearch/_index.md new file mode 100644 index 00000000..e07e5cbd --- /dev/null +++ b/content/chapter-destinations/destination-opensearch/_index.md @@ -0,0 +1,70 @@ +--- +title: "opensearch: Sending messages to OpenSearch" +weight: 900 +--- + + +Available in {{% param "product.abbrev" %}} version 4.4 and later. + +The `opensearch()` destination can directly post log messages to [OpenSearch](https://opensearch.org/) using its HTTP endpoint. + +HTTPS connection, as well as password- and certificate-based authentication is supported. The content of the events is sent in JSON format. + + +## Declaration: + +```shell + d_opensearch { + opensearch( + index("") + url("https://your-opensearch-endpoint:9200/_bulk") + ); + }; +``` + +{{< include-headless "chunk/option-dest-http-proxy.md" >}} + + +## Example: Sending log data to Elasticsearch {#example-destination-elasticsearch-http} + +The following example defines a `elasticsearch-http()` destination, with only the required options. + +```shell + destination opensearch { + opensearch( + index("") + url("http://my-elastic-server:9200/_bulk") + ); + }; + + + log { + source(s_file); + destination(d_opensearch_http); + flags(flow-control); + }; +``` + +The following example uses mutually-authenticated HTTPS connection, templated index, and also sets the `type()` and some other options. + +```shell + destination opensearch_https { + opensearch( + url("https://node01.example.com:9200/_bulk") + index("test-${YEAR}${MONTH}${DAY}") + time-zone("UTC") + type("test") + workers(4) + batch-lines(16) + timeout(10) + tls( + ca-file("ca.pem") + cert-file("syslog_ng.crt.pem") + key-file("syslog_ng.key.pem") + peer-verify(yes) + ) + ); + }; +``` + +This driver is actually a reusable configuration snippet configured to send log messages using the `http()` driver using a template. For details on using or writing such configuration snippets, see {{% xref "/chapter-configuration-file/large-configs/config-blocks/_index.md" %}}. You can find the source of this configuration snippet on [GitHub](https://github.com/syslog-ng/syslog-ng/tree/master/scl/opensearch). diff --git a/content/chapter-destinations/destination-opensearch/reference-destination-opensearch/_index.md b/content/chapter-destinations/destination-opensearch/reference-destination-opensearch/_index.md new file mode 100644 index 00000000..4ef13da0 --- /dev/null +++ b/content/chapter-destinations/destination-opensearch/reference-destination-opensearch/_index.md @@ -0,0 +1,231 @@ +--- +title: "opensearch() destination options" +weight: 300 +--- + + +The `opensearch` destination of {{% param "product.abbrev" %}} can directly post log messages to an OpenSearch deployment using the OpenSearch Bulk API over the HTTP and Secure HTTP (HTTPS) protocols. The `opensearch` destination has the following options. The required options are: `index()` and `url()`. + +This destination is available in {{% param "product.abbrev" %}} version 4.4 and later. + +{{% include-headless "chunk/option-destination-batch-bytes.md" %}} + +For details on how this option influences batch mode, see {{% xref "/chapter-destinations/configuring-destinations-elasticsearch-http/elasticsearch-http-batch-mode/_index.md" %}} + +## batch-lines() + +| | | +| -------- | ------ | +| Type: | number | +| Default: | 25 | + +{{% include-headless "chunk/option-description-destination-batch-lines.md" %}} + +For details on how this option influences batch mode, see {{% xref "/chapter-destinations/configuring-destinations-elasticsearch-http/elasticsearch-http-batch-mode/_index.md" %}} + + + +{{% include-headless "chunk/option-destination-batch-timeout.md" %}} + +For details on how this option influences batch mode, see {{% xref "/chapter-destinations/configuring-destinations-elasticsearch-http/elasticsearch-http-batch-mode/_index.md" %}} + + + +{{% include-headless "chunk/option-destination-tls-ca-dir.md" %}} + +{{% include-headless "chunk/topic-tls-block-http.md" %}} + +{{% include-headless "chunk/example-tls-block-elasticsearch-http-ca-dir.md" %}} + + + +## ca-file() {#opensearch-options-ca-file} + +| | | +| ---------------- | -------- | +| Accepted values: | Filename | +| Default: | none | + +*Description:* Name of a file that contains an X.509 CA certificate (or a certificate chain) in PEM format. The {{% param "product.abbrev" %}} application uses this certificate to validate the certificate of the HTTPS server. If the file contains a certificate chain, the file must begin with the certificate of the host, followed by the CA certificate that signed the certificate of the host, and any other signing CAs in order. + +{{% include-headless "chunk/topic-tls-block-http.md" %}} + +{{% include-headless "chunk/example-tls-block-elasticsearch-http-ca-file.md" %}} + + + +{{% include-headless "chunk/option-destination-tls-cert-file.md" %}} + +{{% include-headless "chunk/topic-tls-block-http.md" %}} + +{{% include-headless "chunk/example-tls-block-elasticsearch-http.md" %}} + + + +{{% include-headless "chunk/option-destination-tls-cipher-suite.md" %}} + +{{% include-headless "chunk/topic-tls-block-http.md" %}} + +{{% include-headless "chunk/example-tls-block-elasticsearch-http.md" %}} + + + +## custom-id() {#opensearch-options-custom-id} + +| | | +| ---------------- | ------------ | +| Accepted values: | string | +| Default: | empty string | + +*Description:* Sets the specified value as the ID of the OpenSearch index (`_id`). + + + +## delimiter() {#opensearch-options-delimiter} + +| | | +| ---------------- | ----------------- | +| Accepted values: | string | +| Default: | newline character | + +*Description:* By default, {{% param "product.abbrev" %}} separates the log messages of the batch with a newline character. You can specify a different delimiter by using the `delimiter()` option. + +For details on how this option influences batch mode, see {{% xref "/chapter-destinations/configuring-destinations-elasticsearch-http/elasticsearch-http-batch-mode/_index.md" %}} + + +{{< include-headless "chunk/option-destination-diskbuffer.md" >}} + +{{< include-headless "chunk/option-destination-hook.md" >}} + + +## index() + +| | | +| ---------------- | ------------------ | +| Accepted values: | string or template | +| Default: | None | + +*Description:* The name of the OpenSearch index where OpenSearch will store the messages received from {{% param "product.abbrev" %}}. This option is mandatory for this destination. + +You can use macros and template functions, but you must ensure that the resolved template contains only characters that OpenSearch permits in the name of the index. The {{% param "product.abbrev" %}} application does not validate the name of the index. For details on the characters permitted in the name of OpenSearch indices, see the documentation of OpenSearch. + + +{{% include-headless "chunk/option-destination-log-fifo-size.md" %}} + + +{{% include-headless "chunk/option-destination-tls-key-file.md" %}} + +This destination supports only unencrypted key files (that is, the private key cannot be password-protected). + +{{% include-headless "chunk/topic-tls-block-http.md" %}} + +{{% include-headless "chunk/example-tls-block-elasticsearch-http.md" %}} + + + +## password() + +| | | +| -------- | ------ | +| Type: | string | +| Default: | | + +*Description:* The password that {{% param "product.abbrev" %}} uses to authenticate on the server where it sends the messages. + + + +{{< include-headless "chunk/option-peer-verify-simple.md" >}} + +{{% include-headless "chunk/topic-tls-block-http.md" %}} + +{{% include-headless "chunk/example-tls-block-elasticsearch-http.md" %}} + + +{{% include-headless "chunk/option-persist-name.md" %}} + + + + +{{% include-headless "chunk/option-destination-http-proxy.md" %}} + + + +{{% include-headless "chunk/option-destination-retries.md" %}} + +To handle HTTP error responses, if the HTTP server returns 5xx codes, {{% param "product.abbrev" %}} will attempt to resend messages until the number of attempts reaches `retries`. If the HTTP server returns 4xx codes, {{% param "product.abbrev" %}} will drop the messages. + + + +## ssl-version() {#opensearch-options-ssl-version} + +| | | +| -------- | ------------------------------ | +| Type: | string | +| Default: | None, uses the libcurl default | + +*Description:* Specifies the permitted SSL/TLS version. Possible values: `sslv2`, `sslv3`, `tlsv1`, `tlsv1_0`, `tlsv1_1`, `tlsv1_2`, `tlsv1_3`. + +{{% include-headless "chunk/topic-tls-block-http.md" %}} + +{{% include-headless "chunk/example-tls-block-elasticsearch-http.md" %}} + + +{{% include-headless "chunk/option-destination-throttle.md" %}} + + +{{% include-headless "chunk/option-source-time-reopen.md" %}} + + +## timeout() {#opensearch-options-timeout} + +| | | +| -------- | ------------------ | +| Type: | number [seconds] | +| Default: | 10 | + +*Description:* The value (in seconds) to wait for an operation to complete, and attempt to reconnect the server if exceeded. + + + +## url() + +| | | +| -------- | ------------------------------------------------------ | +| Type: | URL or list of URLs, for example, url("site1" "site2") | +| Default: | N/A | + +*Description:* Specifies the hostname or IP address and optionally the port number of the OpenSearch indexer. Use a colon (`:`) after the address to specify the port number of the server. For example: `http://your-OpenSearch-indexer.server:8088/_bulk` + +This option is mandatory for this destination. + +Make sure that the URL ends with `_bulk`, this is the OpenSearch API endpoint that properly parses the messages sent by {{% param "product.abbrev" %}}. + +In case the server on the specified URL returns a redirect request, {{% param "product.abbrev" %}} automatically follows maximum 3 redirects. Only HTTP and HTTPS based redirections are supported. + +{{< include-headless "chunk/destination-load-balancing-url.md" >}} + + + +## user() + +| | | +| -------- | ------ | +| Type: | string | +| Default: | | + +*Description:* The username that {{% param "product.abbrev" %}} uses to authenticate on the server where it sends the messages. + + +{{% include-headless "chunk/option-destination-http-use-system-cert-store.md" %}} + + +## workers() + +| | | +| -------- | ------- | +| Type: | integer | +| Default: | 4 | + +{{< include-headless "chunk/option-destination-description-workers.md" >}} + +{{% include-headless "chunk/http-load-balance-workers.md" %}}