Skip to content

Commit

Permalink
Merge pull request #90 from axoflow/minor-fixes-241218
Browse files Browse the repository at this point in the history
Minor fixes 241218
  • Loading branch information
fekete-robert authored Dec 18, 2024
2 parents d0bb606 + 463c064 commit 68cd7df
Show file tree
Hide file tree
Showing 9 changed files with 54 additions and 117 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,3 @@ These lines are connected by their second field: `msg=audit(1440927434.124:40347
);
};
```

For another example, see [The grouping-by() parser blog post](https://www.syslog-ng.com/community/b/blog/posts/the-grouping-by-parser-in-syslog-ng-3-8)

Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,6 @@ short_description: "Send metrics to Graphite"

The `graphite()` destination can send metrics to a [Graphite](http://graphite.readthedocs.io/en/latest/index.html) server to store numeric time-series data. There are many ways to feed the Graphite template function with name value pairs. The {{% param "product.abbrev" %}} CSV and PatternDB parsers (for details, see {{% xref "/chapter-parsers/chapter-patterndb/reference-parsers-pattern-databases/syslog-ng-patterndb-using-parsers/_index.md" %}}) can parse log messages and generate name value pairs based on message content. The CSV parser (for details, see {{% xref "/chapter-parsers/csv-parser/_index.md" %}}) can be used for logs that have a constant field based structure, like the Apache web server access logs. The [patterndb() parser] can parse information and can extract important fields from free form log messages, as long as patterns describing the log messages are available. Another way is to send JSON-based log messages (for details, see {{% xref "/chapter-parsers/json-parser/_index.md" %}}) to {{% param "product.abbrev" %}}, like running a simple shell script collecting metrics and running it from cron regularly.

To see an example of how the `graphite()` destination is used to collect statistics coming from `syslog-ng`, see the blog post [Collecting syslog-ng statistics to Graphite](https://syslog-ng.com/blog/collecting-syslog-ng-statistics-to-graphite/).


## Declaration:

```shell
Expand All @@ -21,7 +18,7 @@ To see an example of how the `graphite()` destination is used to collect statist

## Example: Using the graphite() driver {#example-destination-graphite}

To use the <span>graphite()</span> destination, the only mandatory parameter is payload, which specifies the value pairs to send to <span>graphite</span>. In the following example any value pairs starting with <span>"monitor."</span> are forwarded to <span>graphite</span>.
To use the `graphite()` destination, the only mandatory parameter is payload, which specifies the value pairs to send to Graphite. In the following example any value pairs starting with `"monitor."` are forwarded to Graphite.

```shell
destination d_graphite { graphite(payload("--key monitor.*")); };
Expand Down
2 changes: 0 additions & 2 deletions content/chapter-destinations/python-destination/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -229,8 +229,6 @@ The following code is similar to the previous example, but it opens and closes t
};
```

For a more detailed example about sending log messages to an MQTT (Message Queuing Telemetry Transport) server, see the [Writing Python destination in syslog-ng: how to send log messages to MQTT blog post](https://www.syslog-ng.com/community/b/blog/posts/writing-python-destination-in-syslog-ng-how-to-send-log-messages-to-mqtt).

## Example: Print logs in batch mode {#python-destination-batch-mode-example}

The following is a simple destination that uses the `flush()` method to print the messages in batch mode.
Expand Down
118 changes: 53 additions & 65 deletions content/chapter-sources/syslog-ng-source-osquery/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,11 @@ short_description: "Collect and parse osquery result logs"

The [osquery](https://osquery.io) application allows you to ask questions about your machine using an SQL-like language. For example, you can query running processes, logged in users, installed packages and syslog messages as well. You can make queries on demand, and also schedule them to run regularly.

The `osquery()` source of {{% param "product.abbrev" %}} allows you read the results of periodical osquery queries (from the `/var/log/osquery/osqueryd.results.log` file) and automatically parse the messages (if you want to use {{% param "product.abbrev" %}} to [send log messages to osquery, read this blogpost](https://syslog-ng.com/blog/endpoint-visibility-and-monitoring-using-osquery-and-syslog-ng/)). For example, you can:
The `osquery()` source of {{% param "product.abbrev" %}} allows you read the results of periodical osquery queries (from the `/var/log/osquery/osqueryd.results.log` file) and automatically parse the messages. For example, you can:

- Create filters from the fields of the messages.

- Limit which fields to store, or create additional fields (combine multiple fields into one field, and so on).

- Send the messages to a central location, for example, to Elasticsearch, directly from {{% param "product.abbrev" %}}.
- Create filters from the fields of the messages.
- Limit which fields to store, or create additional fields (combine multiple fields into one field, and so on).
- Send the messages to a central location, for example, to Elasticsearch, directly from {{% param "product.abbrev" %}}.

The {{% param "product.abbrev" %}} application automatically adds the `.osquery.` prefix to the name of the fields the extracted from the message.

Expand All @@ -23,14 +21,13 @@ The `osquery()` source is available in {{% param "product.abbrev" %}} version 3.

## Prerequisites:

- To use the `osquery()` driver, the `scl.conf` file must be included in your {{% param "product.abbrev" %}} configuration:
- To use the `osquery()` driver, the `scl.conf` file must be included in your {{% param "product.abbrev" %}} configuration:

```shell
@include "scl.conf"

@include "scl.conf"
```

- {{% param "product.abbrev" %}} must be compiled with JSON-support enabled.
- {{% param "product.abbrev" %}} must be compiled with JSON-support enabled.


The `osquery()` driver is actually a reusable configuration snippet configured to read the osquery log file using the `file()` driver, and process its JSON contents. For details on using or writing such configuration snippets, see {{% xref "/chapter-configuration-file/large-configs/config-blocks/_index.md" %}}. You can find the source of this configuration snippet on [GitHub](https://github.com/axoflow/axosyslog/blob/master/scl/osquery/plugin.conf).
Expand All @@ -41,83 +38,74 @@ The `osquery()` driver is actually a reusable configuration snippet configured t
The following {{% param "product.abbrev" %}} configuration sample uses the default settings of the driver, reading osquery result logs from the `/var/log/osquery/osqueryd.results.log` file, and writes the log messages generated from the traps into a file.
```shell
@version: 3.10
@include "scl.conf"
source s_osquery {
osquery();
};
log {
source(s_osquery);
destination {
file("/var/log/example.log");
};
@version: 3.10
@include "scl.conf"
source s_osquery {
osquery();
};
log {
source(s_osquery);
destination {
file("/var/log/example.log");
};
};
```

Filter for messages related to loading Linux kernel modules:

```shell
@version: 3.10
@include "scl.conf"
source s_osquery {
osquery();
@version: 3.10
@include "scl.conf"
source s_osquery {
osquery();
};
log {
source(s_osquery);
filter f_modules {
"${.osquery.name}" eq "pack_incident-response_kernel_modules"
};
log {
source(s_osquery);
filter f_modules {
"${.osquery.name}" eq "pack_incident-response_kernel_modules"
};
destination {
file("/var/log/example.log");
};
destination {
file("/var/log/example.log");
};
};
```



## Example: Using the osquery() driver with custom configuration

The following {{% param "product.abbrev" %}} configuration sample reads osquery result logs from the `/tmp/osquery_input.log` file, and writes the log messages generated from the traps into a file. Using the `format-json` template, the outgoing message will be a well-formed JSON message.


## Input message:
### Input message

```shell
{"name":"pack_osquery-monitoring_osquery_info","hostIdentifier":"testhost","calendarTime":"Fri Jul 21 10:04:41 2017 UTC","unixTime":"1500631481","decorations":{"host_uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332","username":"myuser"},"columns":{"build_distro":"xenial","build_platform":"ubuntu","config_hash":"43cd1c6a7d0c283e21e026a53e619b2e582e94ee","config_valid":"1","counter":"4","extensions":"active","instance_id":"d0c3eb0d-f8e0-4bea-868b-18a2c61b438d","pid":"19764","resident_size":"26416000","start_time":"1500629552","system_time":"223","user_time":"476","uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332","version":"2.5.0","watcher":"19762"},"action":"added"}
```



## {{% param "product.abbrev" %}} configuration:
### {{% param "product.abbrev" %}} configuration

```shell
@version: 3.10
@include "scl.conf"
source s_osquery {
osquery(
file(/tmp/osquery_input.log)
prefix(.osquery.)
);
};
destination d_file {
file(
"/tmp/output.txt"
template("$(format_json --key .osquery.*)\n")
);
};
log {
source(s_osquery);
destination(d_file);
flags(flow-control);
};
@version: 3.10
@include "scl.conf"
source s_osquery {
osquery(
file(/tmp/osquery_input.log)
prefix(.osquery.)
);
};
destination d_file {
file(
"/tmp/output.txt"
template("$(format_json --key .osquery.*)\n")
);
};
log {
source(s_osquery);
destination(d_file);
flags(flow-control);
};
```

## Outgoing message:
### Outgoing message

```shell
Outgoing message; message='{"_osquery":{"unixTime":"1500631481","name":"pack_osquery-monitoring_osquery_info","hostIdentifier":"testhost","decorations":{"username":"myuser","host_uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332"},"columns":{"watcher":"19762","version":"2.5.0","uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332","user_time":"476","system_time":"223","start_time":"1500629552","resident_size":"26416000","pid":"19764","instance_id":"d0c3eb0d-f8e0-4bea-868b-18a2c61b438d","extensions":"active","counter":"4","config_valid":"1","config_hash":"43cd1c6a7d0c283e21e026a53e619b2e582e94ee","build_platform":"ubuntu","build_distro":"xenial"},"calendarTime":"Fri Jul 21 10:04:41 2017 UTC","action":"added"}}\x0a'
Outgoing message; message='{"_osquery":{"unixTime":"1500631481","name":"pack_osquery-monitoring_osquery_info","hostIdentifier":"testhost","decorations":{"username":"myuser","host_uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332"},"columns":{"watcher":"19762","version":"2.5.0","uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332","user_time":"476","system_time":"223","start_time":"1500629552","resident_size":"26416000","pid":"19764","instance_id":"d0c3eb0d-f8e0-4bea-868b-18a2c61b438d","extensions":"active","counter":"4","config_valid":"1","config_hash":"43cd1c6a7d0c283e21e026a53e619b2e582e94ee","build_platform":"ubuntu","build_distro":"xenial"},"calendarTime":"Fri Jul 21 10:04:41 2017 UTC","action":"added"}}\x0a'
```



To configure a destination to send the log messages to Elasticsearch, see {{% xref "/chapter-destinations/configuring-destinations-elasticsearch2/_index.md" %}}. For other destinations, see {{% xref "/chapter-destinations/_index.md" %}}.

This file was deleted.

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,3 @@ The following action can be used to log the length of an SSH session (the time d
value('value name="MESSAGE" An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to ${DATE}')
)
```

For another example, see [The grouping-by() parser in syslog-ng blog post](https://www.syslog-ng.com/community/b/blog/posts/the-grouping-by-parser-in-syslog-ng-3-8)

10 changes: 0 additions & 10 deletions content/headless/chunk/topic-troubleshooting-unusual-directory.md

This file was deleted.

This file was deleted.

0 comments on commit 68cd7df

Please sign in to comment.