Ansible Role: td-agent-bit

Description:

Installs and configures td-agent-bit on Debian/RedHat based systems. Defaults to installing via binary packages, but falls back to building from source where no binary is available, such as for Ubuntu 14.x. Includes extremely simple interfaces for Tail and SystemD inputs, and the capability to automatically install and configure logrotate for Tail inputs.

Installation:

ansible-galaxy install jlamendo.td_agent_bit

Example Playbook:

---
- hosts: all
  vars:
    SPLUNK_HOST: "splunk.example.com"
    SPLUNK_PORT: "8088"
    SPLUNK_TOKEN: ""
    flb_outputs:
      - Name: splunk
        Match: "*"
        Host: "{{ SPLUNK_HOST }}"
        Port: "{{ SPLUNK_PORT }}"
        Splunk_Token: "{{ SPLUNK_TOKEN }}"
        TLS: "On"
        TLS.Verify: "On"
        Message_Key: "log"
    flb_inputs:
      files:
        - path: "/var/log/syslog"
          parser: "syslog-rfc3164-local"
      services:
        - sshd
  gather_facts: true
  roles:
    - jlamendo.td_agent_bit

Role Variables:

td-agent-bit Service Variables

flb_svc_flush

• Description: Time in seconds.nanoseconds that the engine loop waits before flushing data ingested by input plugins over to output plugins.
• Default: flb_svc_flush: 5

flb_svc_log_level

• Description: td-agent-bit logs internally to the journal via stdout. This sets the internal log level.
• Default: flb_svc_log_level: info

flb_metrics_enable

• Description: Whether to enable the internal HTTP Metrics server, used for gathering metrics about td-agent-bit itself.
• Default: flb_metrics_enable: false

flb_metrics_bind_cidr

• Description:This role attempts to find the server's internal IP by comparing instance interfaces to this CIDR range. If you want to select a specific IP, use a /32. Once discovered, the IP is used to bind the internal metrics server to.
• Default: flb_metrics_bind_cidr: 172.16.0.0/12

flb_metrics_bind_port

• Description: Which port to bind the metrics server to.
• Default: flb_metrics_bind_port: 9000

flb_upstart_script

• Description: This is only relevant on systems with upstart-managed init. In some cases, the td-agent-bit installer will fail to appropriately create an upstart script for the service. This role will detect that scenario, and generate an upstart script for td-agent-bit. This variable defines where the upstart script should be installed to.
• Default: flb_upstart_script: "/etc/init/td-agent-bit.conf"

flb_logrotate_conf_dir

• Description: The directory where logrotate scripts should be installed.
• Default: flb_logrotate_conf_dir: "/etc/logrotate.d/"

flb_logrotate_prefix

• Description: A prefix that is prepended to the filename for all logrotate scripts generated by this role. The prefix helps the role differentiate its own logrotate scripts from those created by other systems, and therefore to manage them without interference. It is also used to help ensure that there are not duplicate logrotate scripts which rotate the same file.
• Default: flb_logrotate_prefix: "td-agent-autorotate__"

flb_custom_parsers

• Description: A list of custom parser files that will be templated out and injected into the td-agent-bit configuration.
• Default: none
• Example:

flb_custom_parsers:
 - ./templates/parsers/json_parsers.conf.j2

Note: You can see the built-in parsers here. Be sure to look at this before writing your own custom parser, as it is likely one already exists that will fit your need.

flb_includes

• Description: A list of files that will be included into the main td-agent-bit config file via @INCLUDE statements.
• Default: none
• Example:

flb_includes:
 - ./templates/includes/custom_dynamic_output.conf.j2

---

Compile Options

Note: With the one exception of flb_build_force, these options are only relevant when td-agent-bit is being built from source

flb_build_force

• Description: A boolean flag controlling whether to force compiling from source instead of using pre-built distribution binaries
• Default: flb_build_force: false

flb_build_vers

• Description: This option defines what version of td-agent-bit will be downloaded and compiled should the install from packages fail.
• Default: flb_build_vers: "1.4.6"

flb_build_dir

• Description: This option defines where the td-agent-bit sources will be downloaded to and compiled should the install from packages fail.
• Default: flb_build_dir: "/usr/src/td-agent-bit"

flb_sys_conf_dir

• Description: The root directory where system config files are stored. Changing this value may render the td-agent-bit service defunct. Leave it set to the default unless your environment mandates that you change it.
• Default: flb_sys_conf_dir: "/etc"

flb_build_install_prefix:

• Description: Used to derive the location where the td-agent-bit binary will be installed to. Changing this value may render the td-agent-bit service defunct. Leave it set to the default unless your environment mandates that you change it.
• Default: flb_build_install_prefix: "/usr/local"

flb_out_name

• Description: The name of the binary generated by compiling. Changing this value may render the td-agent-bit service defunct. Leave it set to the default unless your environment mandates that you change it.
• Default: flb_out_name: "td-agent-bit"

flb_conf_dir

• Description: The location where td-agent-bit configuration files are stored. Changing this value may render the td-agent-bit service defunct. Leave it set to the default unless your environment mandates that you change it.
• Default: flb_conf_dir: "{{ flb_sys_conf_dir }}/{{ flb_out_name }}/"

flb_conf_file

• Description: The location where the main td-agent-bit configuration file is stored. Changing this value may render the td-agent-bit service defunct. Leave it set to the default unless your environment mandates that you change it.
• Default: flb_conf_file: "{{ flb_out_name }}.conf"

---

Tail Default Variables

Note: All of the tail plugin defaults can also be set on a per-input basis, using the same syntax.

tail_input.buffer_chunk_sz

• Description: This sets the default for the maximum chunk size that the tail plugin will read from a file. This effectively limits the maximum size of a single log line as well.
• Default: tail_input.buffer_chunk_sz: 512k

tail_input.buffer_max_sz

• Description: This sets the default for the maximum amount of data the tail plugin will buffer before forcing a flush. This limit is permissive, and may be exceeded by the value of tail_input.buffer_chunk_sz - 1.
• Default: tail_input.buffer_max_sz: 5M

tail_input.skip_long_lines

• Description: When td-agent-bit encounters a log line that is larger than buffer_chunk_sz, it is not possible for it to ingest this line. Ordinarily, td-agent-bit will indefinitely halt processing on that file when this occurs. This setting directs td-agent-bit to instead skip the problematic line and carry on. This may result in data loss, so in certain situations where that is absolutely unacceptable setting this to Off and having rigorous monitoring to know when td-agent-bit has halted may be preferable.
• Default: tail_input.skip_long_lines: On

tail_input.state_db

• Description: This sets the default path for the internal state databases that the tail plugin will create. One database will be created for each monitored file. The database helps td-agent-bit retain state in the event of a system failure, and also helps the agent track with logrotate more effectively.
• Default: tail_input.state_db: /var/log/.flb-buffer

tail_input.state_db_sync

• Description: Sets the default mode that the tail plugin's internal state database will use to persist data to disk. More information about the available options and what they do can be found here.
• Default: tail_input.state_db_sync: Normal

tail_input.mem_max_sz

• Description: The maximum memory that a single instance of the tail plugin can consume. The default is set quite permissively - if this limit is hit, the plugin will pause reading log lines from disk until the data is flushed to the outputs.
• Default: tail_input.mem_max_sz: 56M

tail_input.path_key

• Description: If supplied, td-agent-bit will automatically add the path of the file that this log event was collected from to the key specified here. Caution is advised here, as it may clobber existing data if the key conflicts with a key that already exists in the event.
• Default: none
• Example: tail_input.path_key: event_source

tail_input.tag_regex

• Description: A regular expression used to extract parts of the filename log events are collected from, in order to include them in the tag. Regular Expressions are in Ruby syntax, and must used named capture groups. These capture groups can then be referenced in Tag, and will be interpolated in to generate a pseudo-dynamic tag.
• Default: none
• Example:

tail_input.tag_regex: "[^-]+-(?<fork_id>[\\d]+)-.*"
inputs:
  files:
    - name: example
      path: /var/log/example*.log
      tag: example_log.<fork_id>

---

Auto-Logrotate Variables

This role automatically generates logrotate configs for each input defined via flb_inputs.files. These variables define various defaults and options for the logrotate configs that will be generated. All of the options here can additionally be set as options on individual flb_inputs.files entries.

tail_input.rotate_enabled

• Description: Enables the generation of logrotate configs for each entry in flb_inputs.files
• Default: tail_input.rotate_enabled: true

tail_input.rotate_interval

• Description: How often the logrotate script will run.
• Default: tail_input.rotate_interval: "hourly"

tail_input.rotate_missingok

• Description: Whether to generate an error when a log file is missing or not. Changing this is not recommended.
• Default: tail_input.rotate_missingok: true

tail_input.rotate_retention

• Description: How many rotated log files to retain. Be aware that setting this to a value above 0 may result in td-agent-bit ingesting your archives.
• Default: tail_input.rotate_retention: 0

tail_input.rotate_maxsize

• Description: How large a log file can be before logrotate forces a rotation regardless of interval. This can be very useful in tandem with adding a cron job to run logrotate more frequently, but the default logrotate cronjob on most systems will not run more frequently than once per hour.
• Default: tail_input.rotate_maxsize: 5M

tail_input.rotate_strategy

• Description: The strategy used to rotate logs. td-agent-bit supports create and copytruncate modes.
• Default: tail_input.rotate_strategy: copytruncate

tail_input.rotate_owner

• Description: The owner for the new log files that logrotate creates
• Default: none

tail_input.rotate_group

• Description: The group for the new log files that logrotate creates
• Default: none

tail_input.rotate_allow_duplicates

• Description: Whether or not a logrotate script should still be created when it is determined that another logrotate script, not managed by this role, is already monitoring the target file/path.
• Default: tail_input.rotate_allow_duplicates: true

tail_input.rotate_scripts

• Description: Custom scripts for logrotate to run at defined phases of the log rotation lifecycle.
• Default: none
• Example:

 tail_input.rotate_scripts:
   - name: postrotate
     script: "/usr/bin/systemctl restart apache"

---

Host/Group Variables:

These options primarily define the per-host/group settings you will use to monitor specific log files, services, and other input sources supported by td-agent-bit.

flb_inputs.files

The files key is a list of entries that will be templated into individual [INPUT] sections using the tail plugin in the td-agent-bit configuration. This option is primarily a convenience that is intended to significantly reduce the effort required to monitor lots of files on lots of hosts, and won't be suitable for every need. If more custom configuration is necessary, or if you want to be closer to the config and have less configured for you automatically, a custom input entry is a better choice. In addition to the options listed here, each entry can also be configured with all of the options listed in the Tail Defaults and Auto-Logrotate Variables sections.

flb_inputs.files[].path

• Description: Path to an individual log file which td-agent-bit should tail and ingest. Supports * wildcards.
• Default: none
• Example: flb_inputs.files[].path: "/var/log/syslog"

flb_inputs.files[].exclude_path

• Description: Pattern for log files which match the path pattern, but should be excluded anyways. Supports * wildcards.
• Default: none
• Example: flb_inputs.files[].exclude_path: "/var/log/app.log*"

flb_inputs.files[].parser

• Description: Select a parser to parse the input file into individual keys before sending it to the output buffer. More information about available built-in parsers can be found here.
• Default: none
• Example: flb_inputs.files[].parser: "syslog-rfc3164-local"

flb_inputs.files[].tag

• Description: Append a tag to each event created by this entry. Primarily useful for routing specific inputs to specific outputs via the Match key on outputs, but can also be used to enrich data with variables available from ansible.
• Default: none
• Example: flb_inputs.files[].tag: syslog::{{ ansible_hostname }}

flb_inputs.services

• Description: A very simple input format that accepts a list of strings that represent the names of services running on the target host. td-agent-bit will then tail the journald entries for these services. No logrotate configuration is generated, because journald handles this and none is needed. Tags will automatically be added in the format service.{{ systemd service name }}.
• Footguns: This input is exclusive to targets that use SystemD as their init system. If the target system is using an alternative init, the role will detect this. If SystemD is not detected, it will refuse to install/compile the related td-agent-bit extensions, skip injecting the service configuration into the td-agent-bit config, and emit a warning when the role is run, but everything else will continue to function as expected.
• Default: none
• Example:

 flb_inputs.services:
   - sshd

flb_inputs.custom:

• Description: Custom inputs for td-agent-bit. No changes are made to the config, instead this is a 1:1 map from YAML to the td-agent-bit config. Keys are case-sensitive.
• Default: none
• Example:

This example shows a block of td-agent-bit config, and an flb_inputs.custom value that would result in that config being created.

[INPUT]
  Name                  systemd
  Read_From_Tail        true
  Strip_Underscores     true
  Systemd_Filter        _SYSTEMD_UNIT=sshd.service
  Tag                   example-host::*

 flb_inputs.custom:
   - Name: systemd
     Read_From_Tail: true
     Strip_Underscores: true
     Systemd_Filter: _SYSTEMD_UNIT=sshd.service
     Tag: "{{ ansible_hostname }}::*"

flb_outputs:

• Description: Configuration of outputs for td-agent-bit. Similar to custom inputs, this is a 1:1 YAML:config mapping, and no changes are made.
• Default: none
• Example:

This example shows a block of td-agent-bit config, and an flb_outputs value that would result in that config being created.

[OUTPUT]
  Name                  splunk
  Match                 *
  Host                  splunk.example.com
  Port                  8088
  Splunk_Token          xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
  TLS                   On
  TLS.Verify            Off

 flb_outputs:
   - Name: splunk
     Match: "*"
     Host: "{{ SPLUNK_HOST }}"
     Port: "{{ SPLUNK_PORT }}"
     Splunk_Token: "{{ SPLUNK_TOKEN }}"
     TLS: "On"
     TLS.Verify: "Off"

flb_filters

• Description: A list of dicts containing filters to be applied to data passing through td-agent-bit. There is one special key, filter_priority that is used to set the order that the filters are templated out in, where 1 will be first. Filters without a priority set will be treated as priority 100, where no intentional changes are made to the order for filters of equal priority, but no guarantees of ordering are made other. Other than this special key, no changes are made to the config and this is a 1:1 map from YAML to the td-agent-bit config. Keys are case-sensitive. A key can be repeated with multiple values by setting the value to a list of desired values rather than a string.
• Default: none
• Example:

This example shows a block of td-agent-bit config, and an flb_filters value that would result in that config being created.

[FILTER]
   Name                 nest
   Match                *
   Operation            nest
   Wildcard             *
   Nest_under           event

flb_filters:
 - Name: nest
   Match: '*'
   Operation: nest
   Wildcard: '*'
   Nest_under: event

flb_lua_filters

• Description: A dict containing custom lua filters to be applied to data passing through td-agent-bit. Expects a dict where each key is the source for a .lua.j2 jinja2 template containing the source code for your filter, and the value of that key is a list of single-element dicts where the key is the lua function to be called, and the value is the Match pattern to call it on.
• Default: none
• Example:

In this example, the custom_filter.lua.j2 file present on the originating host will be templated out to the target. The inject_host lua function inside that script will be run on all inputs, and the inject_index function will be run on all inputs with a Tag matching syslog.* This example shows a block of td-agent-bit config, and an flb_lua_filters value that would result in that config being created.

[FILTER]
  Name                  lua
  Match                 *
  script                /etc/td-agent-bit/filters/custom_filter.lua
  call                  inject_host

[FILTER]
  Name                  lua
  Match                 syslog.*
  script                /etc/td-agent-bit/filters/custom_filter.lua
  call                  inject_index

flb_lua_filters:
 ./templates/lua_filters/custom_filter.lua.j2:
   - inject_host: '*'
   - inject_index: 'syslog.*'

Dependencies:

None.

Author:

• Jon Lamendola

License:

Licensed under the MIT license. Full license text available in the LICENSE file.