Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[4.4][opensearch] Adds initial draft docs #24

Merged
merged 2 commits into from
Oct 3, 2023
Merged

[4.4][opensearch] Adds initial draft docs #24

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cc3896b
[4.4][opensearch] Adds initial draft docs
eldarnash Sep 30, 2023
5830c40
Create separate files for opensearch
eldarnash Oct 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions content/chapter-destinations/configuring-destinations-elasticsearch-http/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ HTTPS connection, as well as password- and certificate-based authentication is s
d_elasticsearch_http {
elasticsearch-http(
index("<elasticsearch-index-to-store-messages>")
url("https://your-elasticsearch-server1:9200/_bulk" "https://your-elasticsearch-server2:9200/_bulk")
url("https://your-elasticsearch-server1:9200/_bulk")
type("<type-of-the-index>")
);
};
Expand All @@ -35,7 +35,7 @@ HTTPS connection, as well as password- and certificate-based authentication is s

## Example: Sending log data to Elasticsearch {#example-destination-elasticsearch-http}

The following example defines a `elasticsearch-http()` destination, with only the required options.
The following example defines an `elasticsearch-http()` destination, with only the required options.

```shell
destination d_elasticsearch_http {
Expand Down
9 changes: 2 additions & 7 deletions ...iguring-destinations-elasticsearch-http/elasticsearch-http-batch-mode/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,20 +15,17 @@ In the following example, a batch consists of 100 messages, or a maximum of 512

```shell
destination d_elasticsearch-http {
elasticsearch-http(url("http://your-elasticsearch-server:9200/_bulk")
elasticsearch-http(
url("http://your-elasticsearch-server:9200/_bulk")
index("<elasticsearch-index-to-store-messages>")
type("")
url("http://your-elasticsearch-server:9200/_bulk")
batch-lines(100)
batch-bytes(512Kb)
batch-timeout(10000)
);
};
```




## Load balancing between multiple Elasticsearch indexers {#elasticsearch-http-load-balancing}

{{< include-headless "chunk/destination-load-balancing-url.md" >}}
Expand All @@ -50,5 +47,3 @@ The following destination sends log messages to 3 different Elasticsearch indexe
```

{{% include-headless "chunk/http-load-balance-workers.md" %}}


66 changes: 66 additions & 0 deletions content/chapter-destinations/destination-opensearch/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
---
title: "opensearch: Sending messages to OpenSearch"
weight: 900
---
<!-- DISCLAIMER: This file is based on the syslog-ng Open Source Edition documentation https://github.com/balabit/syslog-ng-ose-guides/commit/2f4a52ee61d1ea9ad27cb4f3168b95408fddfdf2 and is used under the terms of The syslog-ng Open Source Edition Documentation License. The file has been modified by Axoflow. -->

Available in {{% param "product.abbrev" %}} version 4.4 and later.

The `opensearch()` destination can directly post log messages to [OpenSearch](https://opensearch.org/) using its HTTP endpoint.

HTTPS connection, as well as password- and certificate-based authentication is supported. The content of the events is sent in JSON format.


## Declaration:

```shell
d_opensearch {
opensearch(
index("<opensearch-index-to-store-messages>")
url("https://your-opensearch-endpoint:9200/_bulk")
);
};
```

## Example: Sending log data to OpenSearch {#example-destination-opensearch}

The following example defines an `opensearch()` destination, with only the required options.

```shell
destination opensearch {
opensearch(
index("<name-of-the-index>")
url("http://my-elastic-server:9200/_bulk")
);
};


log {
source(s_file);
destination(d_opensearch_http);
flags(flow-control);
};
```

The following example uses mutually-authenticated HTTPS connection, templated index, and also sets some other options.

```shell
destination opensearch_https {
opensearch(
url("https://node01.example.com:9200/_bulk")
index("test-${YEAR}${MONTH}${DAY}")
time-zone("UTC")
workers(4)
batch-lines(16)
timeout(10)
tls(
ca-file("ca.pem")
cert-file("syslog_ng.crt.pem")
key-file("syslog_ng.key.pem")
peer-verify(yes)
)
);
};
```

This driver is actually a reusable configuration snippet configured to send log messages using the `http()` driver using a template. For details on using or writing such configuration snippets, see {{% xref "/chapter-configuration-file/large-configs/config-blocks/_index.md" %}}. You can find the source of this configuration snippet on [GitHub](https://github.com/syslog-ng/syslog-ng/tree/master/scl/opensearch).
47 changes: 47 additions & 0 deletions content/chapter-destinations/destination-opensearch/batch-mode/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
---
title: "Batch mode and load balancing"
weight: 100
---
<!-- DISCLAIMER: This file is based on the syslog-ng Open Source Edition documentation https://github.com/balabit/syslog-ng-ose-guides/commit/2f4a52ee61d1ea9ad27cb4f3168b95408fddfdf2 and is used under the terms of The syslog-ng Open Source Edition Documentation License. The file has been modified by Axoflow. -->

The `opensearch()` destination automatically sends multiple log messages in a single HTTP request, increasing the rate of messages that your Elasticsearch deployment can consume. For details on adjusting and fine-tuning the batch mode, see the following section.

{{% include-headless "chunk/concept-batch-size.md" %}}

## Example: HTTP batch mode {#example-elasticsearch-http-batch-mode}

In the following example, a batch consists of 100 messages, or a maximum of 512 kilobytes, and is sent every 20 seconds (20000 milliseconds).

```shell
destination d_opensearch {
opensearch(
url("http://your-server:9200/_bulk")
index("<index-to-store-messages>")
batch-lines(100)
batch-bytes(512Kb)
batch-timeout(10000)
);
};
```

## Load balancing between multiple indexers

{{< include-headless "chunk/destination-load-balancing-url.md" >}}

## Example: HTTP load balancing

The following destination sends log messages to 3 different indexer nodes. Each node is assigned a separate worker thread. A batch consists of 100 messages, or a maximum of 512 kilobytes, and is sent every 20 seconds (20000 milliseconds).

```shell
destination d_opensearch {
opensearch(
url("http://your-elasticsearch-server1:9200/_bulk" "http://your-elasticsearch-server2:9200/_bulk" "http://your-elasticsearch-server3:9200/_bulk")
batch-lines(100)
batch-bytes(512Kb)
batch-timeout(20000)
persist-name("opensearch-load-balance")
);
};
```

{{% include-headless "chunk/http-load-balance-workers.md" %}}
233 changes: 233 additions & 0 deletions ...-destinations/destination-opensearch/reference-destination-opensearch/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,233 @@
---
title: "opensearch() destination options"
weight: 300
---
<!-- DISCLAIMER: This file is based on the syslog-ng Open Source Edition documentation https://github.com/balabit/syslog-ng-ose-guides/commit/2f4a52ee61d1ea9ad27cb4f3168b95408fddfdf2 and is used under the terms of The syslog-ng Open Source Edition Documentation License. The file has been modified by Axoflow. -->

The `opensearch` destination of {{% param "product.abbrev" %}} can directly post log messages to an OpenSearch deployment using the OpenSearch Bulk API over the HTTP and Secure HTTP (HTTPS) protocols. The `opensearch` destination has the following options. The required options are: `index()` and `url()`.

This destination is available in {{% param "product.abbrev" %}} version 4.4 and later.

{{% include-headless "chunk/option-destination-batch-bytes.md" %}}

For details on how this option influences batch mode, see {{% xref "/chapter-destinations/destination-opensearch/batch-mode/_index.md" %}}.

## batch-lines()

| | |
| -------- | ------ |
| Type: | number |
| Default: | 25 |

{{% include-headless "chunk/option-description-destination-batch-lines.md" %}}

For details on how this option influences batch mode, see {{% xref "/chapter-destinations/destination-opensearch/batch-mode/_index.md" %}}.

{{% include-headless "chunk/option-destination-batch-timeout.md" %}}

For details on how this option influences batch mode, see {{% xref "/chapter-destinations/destination-opensearch/batch-mode/_index.md" %}}.

{{% include-headless "chunk/option-destination-tls-ca-dir.md" %}}

{{% include-headless "chunk/topic-tls-block-http.md" %}}

### Declaration:

```shell
destination d_opensearch {
opensearch(
url("http://your-server:9200/_bulk")
index("example-index")
tls(
ca-dir("dir")
cert-file("cert")
cipher-suite("cipher")
key-file("key")
peer-verify(yes|no)
ssl-version(<the permitted SSL/TLS version>)
)
);
};
```

## ca-file() {#opensearch-options-ca-file}

| | |
| ---------------- | -------- |
| Accepted values: | Filename |
| Default: | none |

*Description:* Name of a file that contains an X.509 CA certificate (or a certificate chain) in PEM format. The {{% param "product.abbrev" %}} application uses this certificate to validate the certificate of the HTTPS server. If the file contains a certificate chain, the file must begin with the certificate of the host, followed by the CA certificate that signed the certificate of the host, and any other signing CAs in order.

{{% include-headless "chunk/topic-tls-block-http.md" %}}

### Declaration:

```shell
destination d_opensearch {
opensearch(
url("http://your-server:9200/_bulk")
index("example-index")
tls(
ca-file("ca")
cert-file("cert")
cipher-suite("cipher")
key-file("key")
peer-verify(yes|no)
ssl-version(<the permitted SSL/TLS version>)
)
);
};
```

{{% include-headless "chunk/option-destination-tls-cert-file.md" %}}

{{% include-headless "chunk/topic-tls-block-http.md" %}}

{{% include-headless "chunk/example-tls-block-opensearch.md" %}}

{{% include-headless "chunk/option-destination-tls-cipher-suite.md" %}}

{{% include-headless "chunk/topic-tls-block-http.md" %}}

{{% include-headless "chunk/example-tls-block-opensearch.md" %}}

## custom-id() {#opensearch-options-custom-id}

| | |
| ---------------- | ------------ |
| Accepted values: | string |
| Default: | empty string |

*Description:* Sets the specified value as the ID of the OpenSearch index (`_id`).

## delimiter() {#opensearch-options-delimiter}

| | |
| ---------------- | ----------------- |
| Accepted values: | string |
| Default: | newline character |

*Description:* By default, {{% param "product.abbrev" %}} separates the log messages of the batch with a newline character. You can specify a different delimiter by using the `delimiter()` option.

For details on how this option influences batch mode, see {{% xref "/chapter-destinations/destination-opensearch/batch-mode/_index.md" %}}.

{{< include-headless "chunk/option-destination-diskbuffer.md" >}}

{{< include-headless "chunk/option-destination-hook.md" >}}

## index()

| | |
| ---------------- | ------------------ |
| Accepted values: | string or template |
| Default: | None |

*Description:* The name of the OpenSearch index where OpenSearch will store the messages received from {{% param "product.abbrev" %}}. This option is mandatory for this destination.

You can use macros and template functions, but you must ensure that the resolved template contains only characters that OpenSearch permits in the name of the index. The {{% param "product.abbrev" %}} application does not validate the name of the index. For details on the characters permitted in the name of OpenSearch indices, see the documentation of OpenSearch.


{{% include-headless "chunk/option-destination-log-fifo-size.md" %}}


{{% include-headless "chunk/option-destination-tls-key-file.md" %}}

This destination supports only unencrypted key files (that is, the private key cannot be password-protected).

{{% include-headless "chunk/topic-tls-block-http.md" %}}

{{% include-headless "chunk/example-tls-block-opensearch.md" %}}



## password()

| | |
| -------- | ------ |
| Type: | string |
| Default: | |

*Description:* The password that {{% param "product.abbrev" %}} uses to authenticate on the server where it sends the messages.

{{< include-headless "chunk/option-peer-verify-simple.md" >}}

{{% include-headless "chunk/topic-tls-block-http.md" %}}

{{% include-headless "chunk/example-tls-block-opensearch.md" %}}

{{% include-headless "chunk/option-persist-name.md" %}}


<span id="http-proxy"></span>

{{< include-headless "chunk/option-dest-http-proxy.md" >}}

{{% include-headless "chunk/option-destination-retries.md" %}}

To handle HTTP error responses, if the HTTP server returns 5xx codes, {{% param "product.abbrev" %}} will attempt to resend messages until the number of attempts reaches `retries`. If the HTTP server returns 4xx codes, {{% param "product.abbrev" %}} will drop the messages.

## ssl-version() {#opensearch-options-ssl-version}

| | |
| -------- | ------------------------------ |
| Type: | string |
| Default: | None, uses the libcurl default |

*Description:* Specifies the permitted SSL/TLS version. Possible values: `sslv2`, `sslv3`, `tlsv1`, `tlsv1_0`, `tlsv1_1`, `tlsv1_2`, `tlsv1_3`.

{{% include-headless "chunk/topic-tls-block-http.md" %}}

{{% include-headless "chunk/example-tls-block-opensearch.md" %}}

{{% include-headless "chunk/option-destination-throttle.md" %}}

{{% include-headless "chunk/option-source-time-reopen.md" %}}

## timeout() {#opensearch-options-timeout}

| | |
| -------- | ------------------ |
| Type: | number [seconds] |
| Default: | 10 |

*Description:* The value (in seconds) to wait for an operation to complete, and attempt to reconnect the server if exceeded.

## url()

| | |
| -------- | ------------------------------------------------------ |
| Type: | URL or list of URLs, for example, url("site1" "site2") |
| Default: | N/A |

*Description:* Specifies the hostname or IP address and optionally the port number of the OpenSearch indexer. Use a colon (`:`) after the address to specify the port number of the server. For example: `http://your-opensearch-indexer.server:8088/_bulk`

This option is mandatory for this destination.

Make sure that the URL ends with `_bulk`, this is the OpenSearch API endpoint that properly parses the messages sent by {{% param "product.abbrev" %}}.

In case the server on the specified URL returns a redirect request, {{% param "product.abbrev" %}} automatically follows maximum 3 redirects. Only HTTP and HTTPS based redirections are supported.

{{< include-headless "chunk/destination-load-balancing-url.md" >}}

## user()

| | |
| -------- | ------ |
| Type: | string |
| Default: | |

*Description:* The username that {{% param "product.abbrev" %}} uses to authenticate on the server where it sends the messages.

{{% include-headless "chunk/option-destination-http-use-system-cert-store.md" %}}

## workers()

| | |
| -------- | ------- |
| Type: | integer |
| Default: | 4 |

{{< include-headless "chunk/option-destination-description-workers.md" >}}

{{% include-headless "chunk/http-load-balance-workers.md" %}}
Loading