Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minor fixes 241218 #90

Merged
merged 5 commits into from
Dec 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,3 @@ These lines are connected by their second field: `msg=audit(1440927434.124:40347
);
};
```

For another example, see [The grouping-by() parser blog post](https://www.syslog-ng.com/community/b/blog/posts/the-grouping-by-parser-in-syslog-ng-3-8)

Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,6 @@ short_description: "Send metrics to Graphite"

The `graphite()` destination can send metrics to a [Graphite](http://graphite.readthedocs.io/en/latest/index.html) server to store numeric time-series data. There are many ways to feed the Graphite template function with name value pairs. The {{% param "product.abbrev" %}} CSV and PatternDB parsers (for details, see {{% xref "/chapter-parsers/chapter-patterndb/reference-parsers-pattern-databases/syslog-ng-patterndb-using-parsers/_index.md" %}}) can parse log messages and generate name value pairs based on message content. The CSV parser (for details, see {{% xref "/chapter-parsers/csv-parser/_index.md" %}}) can be used for logs that have a constant field based structure, like the Apache web server access logs. The [patterndb() parser] can parse information and can extract important fields from free form log messages, as long as patterns describing the log messages are available. Another way is to send JSON-based log messages (for details, see {{% xref "/chapter-parsers/json-parser/_index.md" %}}) to {{% param "product.abbrev" %}}, like running a simple shell script collecting metrics and running it from cron regularly.

To see an example of how the `graphite()` destination is used to collect statistics coming from `syslog-ng`, see the blog post [Collecting syslog-ng statistics to Graphite](https://syslog-ng.com/blog/collecting-syslog-ng-statistics-to-graphite/).


## Declaration:

```shell
Expand All @@ -21,7 +18,7 @@ To see an example of how the `graphite()` destination is used to collect statist

## Example: Using the graphite() driver {#example-destination-graphite}

To use the <span>graphite()</span> destination, the only mandatory parameter is payload, which specifies the value pairs to send to <span>graphite</span>. In the following example any value pairs starting with <span>"monitor."</span> are forwarded to <span>graphite</span>.
To use the `graphite()` destination, the only mandatory parameter is payload, which specifies the value pairs to send to Graphite. In the following example any value pairs starting with `"monitor."` are forwarded to Graphite.

```shell
destination d_graphite { graphite(payload("--key monitor.*")); };
Expand Down
2 changes: 0 additions & 2 deletions content/chapter-destinations/python-destination/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -229,8 +229,6 @@ The following code is similar to the previous example, but it opens and closes t
};
```

For a more detailed example about sending log messages to an MQTT (Message Queuing Telemetry Transport) server, see the [Writing Python destination in syslog-ng: how to send log messages to MQTT blog post](https://www.syslog-ng.com/community/b/blog/posts/writing-python-destination-in-syslog-ng-how-to-send-log-messages-to-mqtt).

## Example: Print logs in batch mode {#python-destination-batch-mode-example}

The following is a simple destination that uses the `flush()` method to print the messages in batch mode.
Expand Down
118 changes: 53 additions & 65 deletions content/chapter-sources/syslog-ng-source-osquery/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,11 @@ short_description: "Collect and parse osquery result logs"

The [osquery](https://osquery.io) application allows you to ask questions about your machine using an SQL-like language. For example, you can query running processes, logged in users, installed packages and syslog messages as well. You can make queries on demand, and also schedule them to run regularly.

The `osquery()` source of {{% param "product.abbrev" %}} allows you read the results of periodical osquery queries (from the `/var/log/osquery/osqueryd.results.log` file) and automatically parse the messages (if you want to use {{% param "product.abbrev" %}} to [send log messages to osquery, read this blogpost](https://syslog-ng.com/blog/endpoint-visibility-and-monitoring-using-osquery-and-syslog-ng/)). For example, you can:
The `osquery()` source of {{% param "product.abbrev" %}} allows you read the results of periodical osquery queries (from the `/var/log/osquery/osqueryd.results.log` file) and automatically parse the messages. For example, you can:

- Create filters from the fields of the messages.

- Limit which fields to store, or create additional fields (combine multiple fields into one field, and so on).

- Send the messages to a central location, for example, to Elasticsearch, directly from {{% param "product.abbrev" %}}.
- Create filters from the fields of the messages.
- Limit which fields to store, or create additional fields (combine multiple fields into one field, and so on).
- Send the messages to a central location, for example, to Elasticsearch, directly from {{% param "product.abbrev" %}}.

The {{% param "product.abbrev" %}} application automatically adds the `.osquery.` prefix to the name of the fields the extracted from the message.

Expand All @@ -23,14 +21,13 @@ The `osquery()` source is available in {{% param "product.abbrev" %}} version 3.

## Prerequisites:

- To use the `osquery()` driver, the `scl.conf` file must be included in your {{% param "product.abbrev" %}} configuration:
- To use the `osquery()` driver, the `scl.conf` file must be included in your {{% param "product.abbrev" %}} configuration:

```shell
@include "scl.conf"

@include "scl.conf"
```

- {{% param "product.abbrev" %}} must be compiled with JSON-support enabled.
- {{% param "product.abbrev" %}} must be compiled with JSON-support enabled.


The `osquery()` driver is actually a reusable configuration snippet configured to read the osquery log file using the `file()` driver, and process its JSON contents. For details on using or writing such configuration snippets, see {{% xref "/chapter-configuration-file/large-configs/config-blocks/_index.md" %}}. You can find the source of this configuration snippet on [GitHub](https://github.com/axoflow/axosyslog/blob/master/scl/osquery/plugin.conf).
Expand All @@ -41,83 +38,74 @@ The `osquery()` driver is actually a reusable configuration snippet configured t
The following {{% param "product.abbrev" %}} configuration sample uses the default settings of the driver, reading osquery result logs from the `/var/log/osquery/osqueryd.results.log` file, and writes the log messages generated from the traps into a file.

```shell
@version: 3.10
@include "scl.conf"
source s_osquery {
osquery();
};
log {
source(s_osquery);
destination {
file("/var/log/example.log");
};
@version: 3.10
@include "scl.conf"
source s_osquery {
osquery();
};
log {
source(s_osquery);
destination {
file("/var/log/example.log");
};
};
```

Filter for messages related to loading Linux kernel modules:

```shell
@version: 3.10
@include "scl.conf"
source s_osquery {
osquery();
@version: 3.10
@include "scl.conf"
source s_osquery {
osquery();
};
log {
source(s_osquery);
filter f_modules {
"${.osquery.name}" eq "pack_incident-response_kernel_modules"
};
log {
source(s_osquery);
filter f_modules {
"${.osquery.name}" eq "pack_incident-response_kernel_modules"
};
destination {
file("/var/log/example.log");
};
destination {
file("/var/log/example.log");
};
};
```



## Example: Using the osquery() driver with custom configuration

The following {{% param "product.abbrev" %}} configuration sample reads osquery result logs from the `/tmp/osquery_input.log` file, and writes the log messages generated from the traps into a file. Using the `format-json` template, the outgoing message will be a well-formed JSON message.


## Input message:
### Input message

```shell
{"name":"pack_osquery-monitoring_osquery_info","hostIdentifier":"testhost","calendarTime":"Fri Jul 21 10:04:41 2017 UTC","unixTime":"1500631481","decorations":{"host_uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332","username":"myuser"},"columns":{"build_distro":"xenial","build_platform":"ubuntu","config_hash":"43cd1c6a7d0c283e21e026a53e619b2e582e94ee","config_valid":"1","counter":"4","extensions":"active","instance_id":"d0c3eb0d-f8e0-4bea-868b-18a2c61b438d","pid":"19764","resident_size":"26416000","start_time":"1500629552","system_time":"223","user_time":"476","uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332","version":"2.5.0","watcher":"19762"},"action":"added"}
```



## {{% param "product.abbrev" %}} configuration:
### {{% param "product.abbrev" %}} configuration

```shell
@version: 3.10
@include "scl.conf"
source s_osquery {
osquery(
file(/tmp/osquery_input.log)
prefix(.osquery.)
);
};
destination d_file {
file(
"/tmp/output.txt"
template("$(format_json --key .osquery.*)\n")
);
};
log {
source(s_osquery);
destination(d_file);
flags(flow-control);
};
@version: 3.10
@include "scl.conf"
source s_osquery {
osquery(
file(/tmp/osquery_input.log)
prefix(.osquery.)
);
};
destination d_file {
file(
"/tmp/output.txt"
template("$(format_json --key .osquery.*)\n")
);
};
log {
source(s_osquery);
destination(d_file);
flags(flow-control);
};
```

## Outgoing message:
### Outgoing message

```shell
Outgoing message; message='{"_osquery":{"unixTime":"1500631481","name":"pack_osquery-monitoring_osquery_info","hostIdentifier":"testhost","decorations":{"username":"myuser","host_uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332"},"columns":{"watcher":"19762","version":"2.5.0","uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332","user_time":"476","system_time":"223","start_time":"1500629552","resident_size":"26416000","pid":"19764","instance_id":"d0c3eb0d-f8e0-4bea-868b-18a2c61b438d","extensions":"active","counter":"4","config_valid":"1","config_hash":"43cd1c6a7d0c283e21e026a53e619b2e582e94ee","build_platform":"ubuntu","build_distro":"xenial"},"calendarTime":"Fri Jul 21 10:04:41 2017 UTC","action":"added"}}\x0a'
Outgoing message; message='{"_osquery":{"unixTime":"1500631481","name":"pack_osquery-monitoring_osquery_info","hostIdentifier":"testhost","decorations":{"username":"myuser","host_uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332"},"columns":{"watcher":"19762","version":"2.5.0","uuid":"4C4C4544-004D-3610-8043-C2C04F4D3332","user_time":"476","system_time":"223","start_time":"1500629552","resident_size":"26416000","pid":"19764","instance_id":"d0c3eb0d-f8e0-4bea-868b-18a2c61b438d","extensions":"active","counter":"4","config_valid":"1","config_hash":"43cd1c6a7d0c283e21e026a53e619b2e582e94ee","build_platform":"ubuntu","build_distro":"xenial"},"calendarTime":"Fri Jul 21 10:04:41 2017 UTC","action":"added"}}\x0a'
```



To configure a destination to send the log messages to Elasticsearch, see {{% xref "/chapter-destinations/configuring-destinations-elasticsearch2/_index.md" %}}. For other destinations, see {{% xref "/chapter-destinations/_index.md" %}}.

This file was deleted.

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,3 @@ The following action can be used to log the length of an SSH session (the time d
value('value name="MESSAGE" An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to ${DATE}')
)
```

For another example, see [The grouping-by() parser in syslog-ng blog post](https://www.syslog-ng.com/community/b/blog/posts/the-grouping-by-parser-in-syslog-ng-3-8)

10 changes: 0 additions & 10 deletions content/headless/chunk/topic-troubleshooting-unusual-directory.md

This file was deleted.

This file was deleted.

Loading