From a35642403b5d4078666dfd51962ec9ce49f2a9ef Mon Sep 17 00:00:00 2001 From: Robert Fekete Date: Fri, 12 Apr 2024 16:02:32 +0200 Subject: [PATCH 01/13] [4.7] documents concurrent-requests() --- content/chapter-sources/opentelemetry/_index.md | 2 ++ .../chapter-sources/source-syslog-ng-otlp/_index.md | 2 ++ .../chunk/option-source-concurrent-requests.md | 12 ++++++++++++ 3 files changed, 16 insertions(+) create mode 100644 content/headless/chunk/option-source-concurrent-requests.md diff --git a/content/chapter-sources/opentelemetry/_index.md b/content/chapter-sources/opentelemetry/_index.md index b67008a6..1c0493ce 100644 --- a/content/chapter-sources/opentelemetry/_index.md +++ b/content/chapter-sources/opentelemetry/_index.md @@ -33,6 +33,8 @@ log otel_forward_mode_alts { {{< include-headless "chunk/grpc-authentication.md" >}} +{{< include-headless "chunk/option-source-concurrent-requests.md" >}} + ## `port()` {#port} The port number to receive incoming connections. Default value: 4317 diff --git a/content/chapter-sources/source-syslog-ng-otlp/_index.md b/content/chapter-sources/source-syslog-ng-otlp/_index.md index 8dd85fd8..0cc90738 100644 --- a/content/chapter-sources/source-syslog-ng-otlp/_index.md +++ b/content/chapter-sources/source-syslog-ng-otlp/_index.md @@ -40,6 +40,8 @@ The `syslog-ng-otlp()` source has the following options. *Description:* Enable or disable the chained hostname format. For details, see the [chain-hostnames() global option]({{< relref "/chapter-global-options/reference-options/_index.md#global-options-chain-hostnames" >}}). +{{< include-headless "chunk/option-source-concurrent-requests.md" >}} + {{< include-headless "chunk/option-source-default-facility.md" >}} {{% include-headless "chunk/option-source-default-level-journal.md" %}} diff --git a/content/headless/chunk/option-source-concurrent-requests.md b/content/headless/chunk/option-source-concurrent-requests.md new file mode 100644 index 00000000..cbca9091 --- /dev/null +++ b/content/headless/chunk/option-source-concurrent-requests.md @@ -0,0 +1,12 @@ +--- +--- + + +## concurrent-requests() + +| | | +| -------- | --------------- | +| Type: | integer | +| Default: | 2 | + +*Description:* Configures the maximal number of in-flight gRPC requests per worker. Setting this value in the range of 10s or 100s is recommended when there are a high number of clients sending simultaneously. Ideally, `workers() * concurrent-requests()` should be greater than or equal to the number of clients, but this can increase the memory usage. From c70b062a005bede27ca8a355dbfcae6fb38ff2b6 Mon Sep 17 00:00:00 2001 From: Robert Fekete Date: Fri, 12 Apr 2024 16:22:53 +0200 Subject: [PATCH 02/13] [4.7] Adds log-fetch-limit to opentelemetry and syslog-ng-otlp sources --- content/chapter-sources/opentelemetry/_index.md | 2 ++ content/chapter-sources/source-syslog-ng-otlp/_index.md | 2 ++ 2 files changed, 4 insertions(+) diff --git a/content/chapter-sources/opentelemetry/_index.md b/content/chapter-sources/opentelemetry/_index.md index 1c0493ce..3bf72e48 100644 --- a/content/chapter-sources/opentelemetry/_index.md +++ b/content/chapter-sources/opentelemetry/_index.md @@ -35,6 +35,8 @@ log otel_forward_mode_alts { {{< include-headless "chunk/option-source-concurrent-requests.md" >}} +{{% include-headless "chunk/option-source-log-fetch-limit.md" %}} + ## `port()` {#port} The port number to receive incoming connections. Default value: 4317 diff --git a/content/chapter-sources/source-syslog-ng-otlp/_index.md b/content/chapter-sources/source-syslog-ng-otlp/_index.md index 0cc90738..53de439e 100644 --- a/content/chapter-sources/source-syslog-ng-otlp/_index.md +++ b/content/chapter-sources/source-syslog-ng-otlp/_index.md @@ -67,6 +67,8 @@ The `syslog-ng-otlp()` source has the following options. {{< include-headless "chunk/option-source-keep-timestamp.md" >}} +{{% include-headless "chunk/option-source-log-fetch-limit.md" %}} + {{% include-headless "chunk/option-source-file-log-iw-size.md" %}} {{< include-headless "chunk/option-source-log-prefix.md" >}} From 169e5218ed04733c1ad89883156c3a4007230e53 Mon Sep 17 00:00:00 2001 From: Robert Fekete Date: Fri, 12 Apr 2024 16:23:20 +0200 Subject: [PATCH 03/13] [4.7] Documents grpc channel-args --- .../destination-loki/_index.md | 2 ++ .../destination-syslog-ng-otlp/_index.md | 2 ++ .../google-bigquery/_index.md | 2 ++ .../opentelemetry/_index.md | 2 ++ .../chapter-sources/opentelemetry/_index.md | 2 ++ .../source-syslog-ng-otlp/_index.md | 2 ++ .../chunk/option-grpc-channel-args.md | 19 +++++++++++++++++++ 7 files changed, 31 insertions(+) create mode 100644 content/headless/chunk/option-grpc-channel-args.md diff --git a/content/chapter-destinations/destination-loki/_index.md b/content/chapter-destinations/destination-loki/_index.md index b68aff74..5f12e3a2 100644 --- a/content/chapter-destinations/destination-loki/_index.md +++ b/content/chapter-destinations/destination-loki/_index.md @@ -43,6 +43,8 @@ The `loki()` destination has the following options. {{% include-headless "chunk/option-destination-batch-timeout.md" %}} +{{< include-headless "chunk/option-grpc-channel-args.md" >}} + {{< include-headless "chunk/option-destination-grpc-keep-alive.md" >}} ## labels() diff --git a/content/chapter-destinations/destination-syslog-ng-otlp/_index.md b/content/chapter-destinations/destination-syslog-ng-otlp/_index.md index a6cb76ec..80d4cee1 100644 --- a/content/chapter-destinations/destination-syslog-ng-otlp/_index.md +++ b/content/chapter-destinations/destination-syslog-ng-otlp/_index.md @@ -31,6 +31,8 @@ The `syslog-ng-otlp()` destination has the following options. {{% include-headless "chunk/option-destination-batch-timeout.md" %}} +{{< include-headless "chunk/option-grpc-channel-args.md" >}} + {{< include-headless "chunk/option-destination-grpc-compression.md" >}} {{< include-headless "chunk/option-destination-diskbuffer.md" >}} diff --git a/content/chapter-destinations/google-bigquery/_index.md b/content/chapter-destinations/google-bigquery/_index.md index 83cea287..0b385daa 100644 --- a/content/chapter-destinations/google-bigquery/_index.md +++ b/content/chapter-destinations/google-bigquery/_index.md @@ -64,6 +64,8 @@ By default, the `batch-bytes()` option of the `bigquery()` destination is 10 MB. {{% include-headless "chunk/option-destination-batch-timeout.md" %}} +{{< include-headless "chunk/option-grpc-channel-args.md" >}} + {{< include-headless "chunk/option-destination-grpc-compression.md" >}} ## dataset() diff --git a/content/chapter-destinations/opentelemetry/_index.md b/content/chapter-destinations/opentelemetry/_index.md index 02dd83c9..eed20d6d 100644 --- a/content/chapter-destinations/opentelemetry/_index.md +++ b/content/chapter-destinations/opentelemetry/_index.md @@ -64,6 +64,8 @@ log non_otel_to_otel_tls { {{< include-headless "chunk/option-destination-threaded-batching.md" >}} +{{< include-headless "chunk/option-grpc-channel-args.md" >}} + {{< include-headless "chunk/option-destination-grpc-compression.md" >}} {{< include-headless "chunk/option-destination-threaded-workers.md" >}} diff --git a/content/chapter-sources/opentelemetry/_index.md b/content/chapter-sources/opentelemetry/_index.md index 3bf72e48..e31c309f 100644 --- a/content/chapter-sources/opentelemetry/_index.md +++ b/content/chapter-sources/opentelemetry/_index.md @@ -33,6 +33,8 @@ log otel_forward_mode_alts { {{< include-headless "chunk/grpc-authentication.md" >}} +{{< include-headless "chunk/option-grpc-channel-args.md" >}} + {{< include-headless "chunk/option-source-concurrent-requests.md" >}} {{% include-headless "chunk/option-source-log-fetch-limit.md" %}} diff --git a/content/chapter-sources/source-syslog-ng-otlp/_index.md b/content/chapter-sources/source-syslog-ng-otlp/_index.md index 53de439e..f850ba15 100644 --- a/content/chapter-sources/source-syslog-ng-otlp/_index.md +++ b/content/chapter-sources/source-syslog-ng-otlp/_index.md @@ -40,6 +40,8 @@ The `syslog-ng-otlp()` source has the following options. *Description:* Enable or disable the chained hostname format. For details, see the [chain-hostnames() global option]({{< relref "/chapter-global-options/reference-options/_index.md#global-options-chain-hostnames" >}}). +{{< include-headless "chunk/option-grpc-channel-args.md" >}} + {{< include-headless "chunk/option-source-concurrent-requests.md" >}} {{< include-headless "chunk/option-source-default-facility.md" >}} diff --git a/content/headless/chunk/option-grpc-channel-args.md b/content/headless/chunk/option-grpc-channel-args.md new file mode 100644 index 00000000..94a422e6 --- /dev/null +++ b/content/headless/chunk/option-grpc-channel-args.md @@ -0,0 +1,19 @@ +--- +--- + + +## channel-args() + +| | | +| -------- | --------------- | +| Type: | See description | +| Default: | - | + +*Description:* The `channel-args()` option is available in gRPC-based drivers. It accepts name-value pairs and sets channel arguments defined in the [GRPC Core library documentation](https://grpc.github.io/grpc/core/group__grpc__arg__keys.html). For example: + +```shell + channel-args( + "grpc.loadreporting" => 1 + "grpc.minimal_stack" => 0 + ) +``` From 03ead33bc904ec28f061dbf8203487a90b90c45c Mon Sep 17 00:00:00 2001 From: Robert Fekete Date: Fri, 12 Apr 2024 16:42:45 +0200 Subject: [PATCH 04/13] [4.7] Documents *arr() and jellyfin() sources --- content/chapter-sources/arr/_index.md | 35 ++++++++++++++++++++++ content/chapter-sources/jellyfin/_index.md | 30 +++++++++++++++++++ 2 files changed, 65 insertions(+) create mode 100644 content/chapter-sources/arr/_index.md create mode 100644 content/chapter-sources/jellyfin/_index.md diff --git a/content/chapter-sources/arr/_index.md b/content/chapter-sources/arr/_index.md new file mode 100644 index 00000000..80e415a2 --- /dev/null +++ b/content/chapter-sources/arr/_index.md @@ -0,0 +1,35 @@ +--- +title: Arr logs +linktitle: "*arr() logs" +weight: 100 +driver: "lidarr(), prowlarr(), radarr(), readarr(), sonarr(), whisparr()" +short_description: "Collect Pi-hole FTL logs" +--- + + +Starting with version 4.7.0, {{% param "product_name" %}} can collect logs of the [Lidarr, Prowlarr, Radarr, Readarr, and Sonarr](https://github.com/Servarr/Wiki) (often referred to as "*Arr" or "*Arrs") applications. + +Use the new `*arr()` sources to read various *arr logs: + +- `lidarr()` +- `prowlarr()` +- `radarr()` +- `readarr()` +- `sonarr()` +- `whisparr()` + +Example minimal config: + +```shell +source s_radarr { + radarr( + dir("/path/to/my/radarr/log/dir") + ); +}; +``` + +The logging module is stored in the `` name-value pair, for example: `.radarr.module` => `ImportListSyncService`. + +You can modify the prefix with the `prefix()` option. + +This driver is actually a reusable configuration snippet. For details on using or writing such configuration snippets, see {{% xref "/chapter-configuration-file/large-configs/config-blocks/_index.md" %}}. You can find the source of this configuration snippet on [GitHub](https://github.com/syslog-ng/syslog-ng/blob/master/scl/arr/arr.conf). diff --git a/content/chapter-sources/jellyfin/_index.md b/content/chapter-sources/jellyfin/_index.md new file mode 100644 index 00000000..ba6864ac --- /dev/null +++ b/content/chapter-sources/jellyfin/_index.md @@ -0,0 +1,30 @@ +--- +title: Jellyfin logs +linktitle: "Jellifin logs" +weight: 1280 +driver: "jellyfin()" +short_description: "Collect Jellyfin logs" +--- + + +Starting with version 4.7.0, you can use the `jellyfin()` source to read [Jellyfin](https://jellyfin.org/) logs from its log file output. + +Example minimal configuration: + +```shell +source s_jellyfin { + jellyfin( + base-dir("/path/to/my/jellyfin/root/log/dir") + filename-pattern("log_*.log") + ); +}; +``` + +For more details about Jellyfin logging, see: + +- https://jellyfin.org/docs/general/administration/configuration/#main-configuration +- https://jellyfin.org/docs/general/administration/configuration/#log-directory + +As the `jellyfin()` source is based on the [`wildcard-file()` source]({{< relref "/chapter-sources/configuring-sources-wildcard-file/_index.md" >}}), you can use the [`wildcard-file()` source options]({{< relref "/chapter-sources/configuring-sources-wildcard-file/reference-source-wildcard-file/_index.md" >}}). + +This driver is actually a reusable configuration snippet. For details on using or writing such configuration snippets, see {{% xref "/chapter-configuration-file/large-configs/config-blocks/_index.md" %}}. You can find the source of this configuration snippet on [GitHub](https://github.com/syslog-ng/syslog-ng/blob/master/scl/jellyfin/jellyfin.conf). From b885a87043dbd635eab0f11ec9ead2bd0eb5d552 Mon Sep 17 00:00:00 2001 From: Robert Fekete Date: Fri, 12 Apr 2024 17:57:36 +0200 Subject: [PATCH 05/13] [4.7] Updates transport macro with local values --- .../reference-macros/_index.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/content/chapter-manipulating-messages/customizing-message-format/reference-macros/_index.md b/content/chapter-manipulating-messages/customizing-message-format/reference-macros/_index.md index de507f22..d1c336e9 100644 --- a/content/chapter-manipulating-messages/customizing-message-format/reference-macros/_index.md +++ b/content/chapter-manipulating-messages/customizing-message-format/reference-macros/_index.md @@ -413,6 +413,17 @@ Available in {{% param "product.abbrev" %}} version 4.5 and later. - `mqtt`: `mqtt()` driver - `hypr-api`: `hypr-audit-source()` driver +- Locally created logs (in version 4.7 and newer): + - `local+unix-stream` + - `local+unix-dgram` + - `local+file` + - `local+pipe` + - `local+program` + - `local+devkmsg` + - `local+journal` + - `local+afstreams` + - `local+openbsd` + {{% include-headless "chunk/macro-tz.md" %}} {{% include-headless "chunk/macro-tzoffset.md" %}} From d5d9821672bdede656b4446295e6446495caf0ba Mon Sep 17 00:00:00 2001 From: Robert Fekete Date: Fri, 12 Apr 2024 17:57:58 +0200 Subject: [PATCH 06/13] [4.7][loki] Adds tenant-id --- .../destination-loki/_index.md | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/content/chapter-destinations/destination-loki/_index.md b/content/chapter-destinations/destination-loki/_index.md index 5f12e3a2..65a852e0 100644 --- a/content/chapter-destinations/destination-loki/_index.md +++ b/content/chapter-destinations/destination-loki/_index.md @@ -75,6 +75,27 @@ Default value: *Description:* Specifies a template defining the logformat to be used in the destination. Macros are described in {{% xref "/chapter-manipulating-messages/customizing-message-format/reference-macros/_index.md" %}}. For details on template functions, see {{% xref "/chapter-manipulating-messages/customizing-message-format/reference-template-functions/_index.md" %}}. +## tenant-id() + +| | | +| -------- | -------------------------------------------------- | +| Type: | string | +| Default: | - | + +*Description:* Available in version 4.7 and newer. Sets the tenant ID for multi-tenant scenarios. For example: + +```shell +loki( + url("localhost:9096") + labels( + "app" => "$PROGRAM", + "host" => "$HOST", + ) + + tenant-id("testTenant") +); +``` + ## timestamp() | | | From 3cd0c3d41b3cbd2bae6926b6d27ded20a5ed0cfa Mon Sep 17 00:00:00 2001 From: Robert Fekete Date: Fri, 12 Apr 2024 18:00:55 +0200 Subject: [PATCH 07/13] [4.7][s3] Credentials can come from env variables --- content/chapter-destinations/destination-s3/_index.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/content/chapter-destinations/destination-s3/_index.md b/content/chapter-destinations/destination-s3/_index.md index 9625fe3d..14a37e83 100644 --- a/content/chapter-destinations/destination-s3/_index.md +++ b/content/chapter-destinations/destination-s3/_index.md @@ -65,6 +65,8 @@ The following options are specific to the `s3()` destination. *Description:* The `ACCESS_KEY` of the service account used to access the S3 bucket. (Together with [`secret-key()`](#secret-key).) +Starting with version 4.7, you can use the `AWS_...` environment variables or credentials files from the `~/.aws/` directory instead of this option. For details, see the [official documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html). + ## bucket() | | | @@ -184,6 +186,8 @@ If you configure an invalid value, the default is used. *Description:* The `SECRET_KEY` of the service account used to access the S3 bucket. (Together with [`access-key()`](#access-key).) +Starting with version 4.7, you can use the `AWS_...` environment variables or credentials files from the `~/.aws/` directory instead of this option. For details, see the [official documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html). + ## storage-class() | | | From 9bd864d0653af1ae6388abe23adf9a5ad3029d89 Mon Sep 17 00:00:00 2001 From: Robert Fekete Date: Fri, 12 Apr 2024 18:06:52 +0200 Subject: [PATCH 08/13] [4.7] Adds the tags-head template function --- .../reference-template-functions/_index.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/content/chapter-manipulating-messages/customizing-message-format/reference-template-functions/_index.md b/content/chapter-manipulating-messages/customizing-message-format/reference-template-functions/_index.md index 2e12caa0..d6a9daf6 100644 --- a/content/chapter-manipulating-messages/customizing-message-format/reference-template-functions/_index.md +++ b/content/chapter-manipulating-messages/customizing-message-format/reference-template-functions/_index.md @@ -1238,6 +1238,25 @@ $(tag ) - If you do not set the `value-if-set` and `value-if-unset` arguments, the `$(tag)` template function acts as a boolean and expands to `0` or `1`, depending on whether the message has the specified tag set. - If the `value-if-set` and `value-if-unset` arguments are set, `$(tag)` returns a string: the second argument (``) if the message has ``, and the third argument (``) if the message doesn't have ``. +## tags-head {#template-function-tags-head} + +Available in {{% param "product.abbrev" %}} version 4.7 and later. + +*Syntax:* + +```shell +$(tags-head ... ) +``` + +*Description:* This template function accepts multiple tag names, and returns the first one that is set. + +Example config: + +```shell +# resolves to "bar" if "bar" tag is set, but "foo" is not +template("$(tags-head foo bar baz)") +``` + ## template {#template-function-template} *Syntax:* From 79c57cf1a12be4ded87c71dbf4e4fb33911cd70e Mon Sep 17 00:00:00 2001 From: Robert Fekete Date: Mon, 15 Apr 2024 12:59:03 +0200 Subject: [PATCH 09/13] [4.7] MQTT updates and fixes --- content/chapter-sources/source-mqtt/_index.md | 9 ++++++++- .../source-mqtt/source-mqtt-lim/_index.md | 13 +++++-------- .../source-mqtt/source-mqtt-options/_index.md | 13 ------------- .../source-mqtt/source-mqtt-prer/_index.md | 2 +- 4 files changed, 14 insertions(+), 23 deletions(-) diff --git a/content/chapter-sources/source-mqtt/_index.md b/content/chapter-sources/source-mqtt/_index.md index 547ee826..af945892 100644 --- a/content/chapter-sources/source-mqtt/_index.md +++ b/content/chapter-sources/source-mqtt/_index.md @@ -28,14 +28,21 @@ The rest of this chapter and its sections build on your familiarity with the MQT }; ``` +Starting with {{% param "product.abbrev" %}} version 4.7, `mqtt()` source automatically sets the `${MQTT_TOPIC}` name-value pair for the received messages. This can be useful when the name of the topic contains MQTT (`$`, `+`, `#`) wildcards. For example: +```shell +log { + source { mqtt(topic("#")); }; + destination { stdout(template("${MQTT_TOPIC} - ${MESSAGE}\n")); }; +}; +``` ## Example: Using the mqtt() source in your configuration The following example illustrates an `mqtt()` source configured to fetch messages from the MQTT broker running on `localhost:4444` using the `test/test topic`, and send them to the `localhost:4445` address. ```shell - @version: 3.35 + @version: current @include "scl.conf" source s_mqtt { mqtt( diff --git a/content/chapter-sources/source-mqtt/source-mqtt-lim/_index.md b/content/chapter-sources/source-mqtt/source-mqtt-lim/_index.md index 110f8644..53142224 100644 --- a/content/chapter-sources/source-mqtt/source-mqtt-lim/_index.md +++ b/content/chapter-sources/source-mqtt/source-mqtt-lim/_index.md @@ -1,15 +1,12 @@ --- -title: "Limitations to using the mqtt() destination" +title: "Limitations to using the mqtt() source" weight: 300 --- Using the `mqtt()` source of {{% param "product.abbrev" %}} has the following limitations: - - You can only use the `mqtt()` source with {{% param "product.abbrev" %}} version 3.35 or higher. - - - You cannot use the `mqtt()` source without installing the the `eclipse-paho-mqtt-c` library. - - - {{< include-headless "chunk/eclipse-paho-mqtt-c.md" >}} - - - The current implementation of the `mqtt()` source supports versions 3.1 and 3.1.1 of the MQTT protocol +- You can only use the `mqtt()` source with {{% param "product.abbrev" %}} version 3.35 or higher. +- You cannot use the `mqtt()` source without installing the the `eclipse-paho-mqtt-c` library. +- {{< include-headless "chunk/eclipse-paho-mqtt-c.md" >}} +- The current implementation of the `mqtt()` source supports versions 3.1 and 3.1.1 of the MQTT protocol diff --git a/content/chapter-sources/source-mqtt/source-mqtt-options/_index.md b/content/chapter-sources/source-mqtt/source-mqtt-options/_index.md index 0b20f3f6..d5995e4f 100644 --- a/content/chapter-sources/source-mqtt/source-mqtt-options/_index.md +++ b/content/chapter-sources/source-mqtt/source-mqtt-options/_index.md @@ -75,17 +75,4 @@ Required options: `address()`, `fallback-topic()`, and `topic()`. *Description:* Required option. Specifies the MQTT topic. -{{% alert title="Note" color="info" %}} - -The current implementation of the `mqtt()` destination does not support using the following characters for topic names: - - - `$` - - - `+` - - - `#` - -{{% /alert %}} - - {{% include-headless "chunk/option-mqtt-username.md" %}} diff --git a/content/chapter-sources/source-mqtt/source-mqtt-prer/_index.md b/content/chapter-sources/source-mqtt/source-mqtt-prer/_index.md index 5efe36f3..b54e4bf0 100644 --- a/content/chapter-sources/source-mqtt/source-mqtt-prer/_index.md +++ b/content/chapter-sources/source-mqtt/source-mqtt-prer/_index.md @@ -1,5 +1,5 @@ --- -title: "Prerequisites to using the mqtt() destination" +title: "Prerequisites to using the mqtt() drivers" weight: 100 --- From 05c5f6925279b015b23cac42c5cf5f9a4360064d Mon Sep 17 00:00:00 2001 From: Robert Fekete Date: Mon, 15 Apr 2024 13:10:03 +0200 Subject: [PATCH 10/13] [4.7] MQTT updates --- .../customizing-message-format/reference-macros/_index.md | 4 ++++ content/chapter-sources/source-mqtt/_index.md | 2 +- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/content/chapter-manipulating-messages/customizing-message-format/reference-macros/_index.md b/content/chapter-manipulating-messages/customizing-message-format/reference-macros/_index.md index d1c336e9..3d647c5f 100644 --- a/content/chapter-manipulating-messages/customizing-message-format/reference-macros/_index.md +++ b/content/chapter-manipulating-messages/customizing-message-format/reference-macros/_index.md @@ -219,7 +219,11 @@ Note that before AxoSyslog version 3.0, the `${MESSAGE}` macro included the prog Available in {{% param "product.abbrev" %}} version 3.4 and later. +## MQTT_TOPIC +*Description:* The [`mqtt()` source]({{< relref "/chapter-sources/source-mqtt/_index.md" >}}) automatically sets the `${MQTT_TOPIC}` name-value pair for the messages it receives. This is useful when the name of the topic contains MQTT wildcards (`$`, `+`, `#`). + +Available in {{% param "product.abbrev" %}} version 4.7 and later. ## MSG {#macro-msg} diff --git a/content/chapter-sources/source-mqtt/_index.md b/content/chapter-sources/source-mqtt/_index.md index af945892..66d3ebd2 100644 --- a/content/chapter-sources/source-mqtt/_index.md +++ b/content/chapter-sources/source-mqtt/_index.md @@ -28,7 +28,7 @@ The rest of this chapter and its sections build on your familiarity with the MQT }; ``` -Starting with {{% param "product.abbrev" %}} version 4.7, `mqtt()` source automatically sets the `${MQTT_TOPIC}` name-value pair for the received messages. This can be useful when the name of the topic contains MQTT (`$`, `+`, `#`) wildcards. For example: +Starting with {{% param "product.abbrev" %}} version 4.7, `mqtt()` source automatically sets the `${MQTT_TOPIC}` name-value pair for the messages it receives. This is useful when the name of the topic contains MQTT wildcards (`$`, `+`, `#`). For example: ```shell log { From f8df6f684c57055ff9351d44de2d4865095170d9 Mon Sep 17 00:00:00 2001 From: Robert Fekete Date: Thu, 18 Apr 2024 09:15:36 +0200 Subject: [PATCH 11/13] [4.7] Whatsnew update --- .../reference-macros/_index.md | 2 +- content/whats-new/_index.md | 15 +++++++++++++++ 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/content/chapter-manipulating-messages/customizing-message-format/reference-macros/_index.md b/content/chapter-manipulating-messages/customizing-message-format/reference-macros/_index.md index 3d647c5f..2ddf8ce6 100644 --- a/content/chapter-manipulating-messages/customizing-message-format/reference-macros/_index.md +++ b/content/chapter-manipulating-messages/customizing-message-format/reference-macros/_index.md @@ -219,7 +219,7 @@ Note that before AxoSyslog version 3.0, the `${MESSAGE}` macro included the prog Available in {{% param "product.abbrev" %}} version 3.4 and later. -## MQTT_TOPIC +## MQTT_TOPIC {#macro-mqtt-topic} *Description:* The [`mqtt()` source]({{< relref "/chapter-sources/source-mqtt/_index.md" >}}) automatically sets the `${MQTT_TOPIC}` name-value pair for the messages it receives. This is useful when the name of the topic contains MQTT wildcards (`$`, `+`, `#`). diff --git a/content/whats-new/_index.md b/content/whats-new/_index.md index 9ad3944b..2d094af2 100644 --- a/content/whats-new/_index.md +++ b/content/whats-new/_index.md @@ -6,6 +6,19 @@ weight: 10 This page is a changelog that collects the major changes and additions to this documentation. (If you want to know the details about why we have separate documentation for AxoSyslog, our syslog-ng distribution and how it relates to the syslog-ng documentation, read our [syslog-ng documentation and similarities with AxoSyslog Core](https://axoflow.com/axosyslog-core-documentation-syslog-ng/) blog post.) +## Version 4.7 release (2024-04-18) + +- {{% xref "/chapter-sources/arr/_index.md" %}} source +- {{% xref "/chapter-sources/jellyfin/_index.md" %}} source +- `channel-args()` option for gRPC-based drivers, like [`opentelemetry()`]({{< relref "/chapter-destinations/opentelemetry/_index.md#channel-args" >}}) +- `concurrent-requests()` option for the [`opentelemetry()` source]({{< relref "/chapter-sources/opentelemetry/_index.md#concurrent-requests" >}}) and the [`syslog-ng-otlp()` source]({{< relref "/chapter-sources/source-syslog-ng-otlp/_index.md#concurrent-requests" >}}) +- {{% xref "/chapter-destinations/destination-loki/_index.md#tenant-id" %}} option for the Loki destination +- {{% xref "/chapter-manipulating-messages/customizing-message-format/reference-template-functions/_index.md#template-function-tags-head" %}} template function +- {{% xref "/chapter-manipulating-messages/customizing-message-format/reference-macros/_index.md#macro-mqtt-topic" %}} macro +- {{% xref "/chapter-manipulating-messages/customizing-message-format/reference-macros/_index.md#macro-transport" %}} macro updates + +For details, see the [release announcement blog post](https://axoflow.com/axosyslog-release-4-7/). + ## Version 4.6 release (2024-02-01) - [Google BigQuery destination]({{< relref "/chapter-destinations/google-bigquery/_index.md" >}}) @@ -13,6 +26,8 @@ This page is a changelog that collects the major changes and additions to this d - {{% xref "/chapter-manipulating-messages/customizing-message-format/reference-template-functions/_index.md#template-function-tag" %}} template function - `batch-bytes()`, `compression()` and `workers()` options for the [syslog-ng-otlp()]({{< relref "/chapter-destinations/destination-syslog-ng-otlp/_index.md#compression" >}}) and [opentelemetry()]({{< relref "/chapter-destinations/opentelemetry/_index.md#compression" >}}) destinations +For details, see the [release announcement blog post](https://axoflow.com/axosyslog-release-4-6/). + ### New sources - [New macOS sources]({{< relref "/chapter-sources/darwin/_index.md" >}}) From 217ea335bac4d7411be9428ea555596d3e8edffa Mon Sep 17 00:00:00 2001 From: Robert Fekete Date: Thu, 18 Apr 2024 09:15:54 +0200 Subject: [PATCH 12/13] Formatting fixes --- .../concepts-message-representation/_index.md | 34 +++++++++---------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/content/chapter-concepts/concepts-message-representation/_index.md b/content/chapter-concepts/concepts-message-representation/_index.md index ab076bc8..9b309092 100644 --- a/content/chapter-concepts/concepts-message-representation/_index.md +++ b/content/chapter-concepts/concepts-message-representation/_index.md @@ -16,36 +16,36 @@ To parse non-syslog messages, for example, JSON, CSV, or other messages, you can A parsed syslog message has the following parts: - - `Timestamps` - +- `Timestamps` + Two timestamps are associated with every message: one is the timestamp contained within the message (that is, when the sender sent the message), the other is the time when {{% param "product.abbrev" %}} has actually received the message. - - `Severity` - +- `Severity` + The severity of the message. - - `Facility` - +- `Facility` + The facility that sent the message. - - `Tags` - +- `Tags` + Custom text labels added to the message that are mainly used for filtering. None of the current message transport protocols adds tags to the log messages. Tags can be added to the log message only within {{% param "product.abbrev" %}}. The {{% param "product.abbrev" %}} application automatically adds the id of the source as a tag to the incoming messages. Other tags can be added to the message by the pattern database, or using the `tags()` option of the source. - - `IP address of the sender` - +- `IP address of the sender` + The IP address of the host that sent the message. Note that the IP address of the sender is a hard macro and cannot be modified within {{% param "product.abbrev" %}} but the associated hostname can be modified, for example, using rewrite rules. - - `Hard macros` - + - `Hard macros` + Hard macros contain data that is directly derived from the log message, for example, the ${MONTH} macro derives its value from the timestamp. The most important consideration with hard macros is that they are read-only, meaning they cannot be modified using rewrite rules or other means. - - `Soft macros` - - Soft macros (sometimes also called name-value pairs) are either built-in macros automatically generated from the log message (for example, ${HOST}), or custom user-created macros generated by using the {{% param "product.abbrev" %}} pattern database or a CSV-parser. The SDATA fields of RFC5424-formatted log messages become soft macros as well. In contrast with hard macros, soft macros are writable and can be modified within {{% param "product.abbrev" %}}, for example, using rewrite rules. - +- `Soft macros` + + Soft macros (sometimes also called name-value pairs) are either built-in macros automatically generated from the log message (for example, `${HOST}`), or custom user-created macros generated by using the {{% param "product.abbrev" %}} pattern database or a CSV-parser. The SDATA fields of RFC5424-formatted log messages become soft macros as well. In contrast with hard macros, soft macros are writable and can be modified within {{% param "product.abbrev" %}}, for example, using rewrite rules. + {{% alert title="Note" color="info" %}} -It is also possible to set the value of built-in soft macros using parsers, for example, to set the ${HOST} macro from the message using a column of a CSV-parser. +It is also possible to set the value of built-in soft macros using parsers, for example, to set the `${HOST}` macro from the message using a column of a CSV-parser. The data extracted from the log messages using named pattern parsers in the pattern database are also soft macros. {{% /alert %}} {{% alert title="Note" color="info" %}} From baf0b95643d2e998b866fd1f25761f3c80e98058 Mon Sep 17 00:00:00 2001 From: Robert Fekete Date: Thu, 18 Apr 2024 09:19:57 +0200 Subject: [PATCH 13/13] Unrelated typo and formatting fixes --- .../log-statistics-description/_index.md | 34 +++++-------------- 1 file changed, 8 insertions(+), 26 deletions(-) diff --git a/content/chapter-log-statistics/log-statistics-description/_index.md b/content/chapter-log-statistics/log-statistics-description/_index.md index c302e721..f1cf47e1 100644 --- a/content/chapter-log-statistics/log-statistics-description/_index.md +++ b/content/chapter-log-statistics/log-statistics-description/_index.md @@ -18,17 +18,13 @@ The instance ID (destination) of the object, for example, the filename of a file The status of the object. One of the following: - - `a` - active. At the time of quering the statistics, the source or the destination was still alive (it continuously received statistical data). +- `a` - active. At the time of querying the statistics, the source or the destination was still alive (it continuously received statistical data). +- `d` - dynamic. Such objects may not be continuously available, for example, like statistics based on the sender's hostname. These counters only appear above a certain value of `stats-level()` global option: + + - `host`: source host, from `stats-level(2)` + - `program`: program, from `stats-level(3)` + - `sender`: sender host, from `stats-level(3)` - - `d` - dynamic. Such objects may not be continuously available, for example, like statistics based on the sender's hostname. These counters only appear above a certain value of `stats-level()` global option: - - - `host`: source host, from `stats-level(2)` - - - `program`: program, from `stats-level(3)` - - - `sender`: sender host, from `stats-level(3)` - - ## Example: Dynamic counters The following example contains 6 different dynamic values: a sender, a host, and four different programs. @@ -46,14 +42,13 @@ The status of the object. One of the following: src.program;;P-14737;d;stamp;1509121931 src.host;;localhost;d;processed;4 src.host;;localhost;d;stamp;1509121934 - ``` To avoid performance issues or even overloading {{% param "product.abbrev" %}}, you might want to limit the number of registered dynamic counters in the message statistics. To do this, configure the [stats-max-dynamics()]({{< relref "/chapter-global-options/reference-options/_index.md" >}}) global option. - - `o` - This object was once active, but stopped receiving messages. (For example, a dynamic object may disappear and become orphan.) - +- `o` - This object was once active, but stopped receiving messages. (For example, a dynamic object may disappear and become orphan.) + {{% alert title="Note" color="info" %}} The {{% param "product.abbrev" %}} application stores the statistics of the objects when {{% param "product.abbrev" %}} is reloaded. However, if the configuration of {{% param "product.abbrev" %}} was changed since the last reload, the statistics of orphaned objects are deleted. {{% /alert %}} @@ -88,20 +83,8 @@ The type of the statistics: `batch_size_avg`: When batching is enabled, then this shows the current average batch size of the given source or destination. -{{% alert title="Note" color="info" %}} - -In version 7.0.27, {{% param "product.abbrev" %}} only supports the `batch_size_avg` for the `http()` destination. - -{{% /alert %}} - `batch_size_max`: When batching is enabled, the value of `batch_size_max` shows the current largest batch size of the given source or destination. -{{% alert title="Note" color="info" %}} - -In version 7.0.27, {{% param "product.abbrev" %}} only supports the `batch_size_max` for the `http()` destination. - -{{% /alert %}} - `discarded`: The number of messages discarded by the given parser. These are messages that the parser could not parsed, and are therefore not processed. For example: ```shell @@ -163,7 +146,6 @@ Since the `not_matched` metric applies to filters, and filters are expected to d ```shell filter;demo_filter;;a;not_matched;0 - ``` {{% /alert %}}