Monitoring Stack

Service Name	Description
Grafana	Display Statistics & Metrics from database
Alertmanager	Query db of metrics and send alerts based on user defined rules
Prometheus	Collect and store metrics scraped from exporters in database
Node Exporter	Export hardware and OS metrics via http endpoint
ZnapZend Exporter	Export state information on zfs snapshots and replication tasks

The services outlined above are deployed as containers using either podman or docker depending on Host OS. Containers are managed via systemd services and/or cockpit-podman module

Supported OS

• Rocky Linux 8.X
• Rocky Linux 9.X
• Ubuntu 20.04
• Ubuntu 22.04

Installation

• Clone git repo to "/usr/share"

cd /usr/share/
git clone https://github.com/45drives/monitoring-stack.git

• Included inventory file "hosts" has two groups "metrics" and "exporters"

  • All hosts in the "metrics" group will have prometheus, alertmanager and grafana installed
  • All hosts in the "exporters" group will have node_exporter and znapzend_exporter installed
  • By default "metrics" and "exporters" is populated by localhost. This is sufficient for a single server deployment.
    • To add multiple servers add new hosts in the "exporters" group
    • It is possible to have the metric stack not run on the same server as the exporter services.

• Configure email send/recieve setting for alertmanager in "group_vars/metrics.yml"

• Default ports are defined in the table below, they can be changed in metrics.yml or exporters.yml

Default Setting	Value
Prometheus Port	9091
Alertmanager Port	9093
Grafana Port	3000
Grafana Default User	admin
Grafana Default Password	admin
Node Exporter Port	9100
Znapzend Port	9101

• Run metrics playbook

cd /usr/share/monitoring-stack
ansible-playbook -i hosts deploy-monitoring.yml

• To uninstall monitoring stack

ansible-playbook -i hosts purge-monitoring.yml

Verification

To ensure monitoring stack is working as expected, simulate failure condition and you will recieve an email notification

• Offline a disk in your zpool
  • Set disk as "Offline" in Houston UI, "ZFS + File Sharing"
  • Or in cli: zpool offline tank 1-1
• After ~30 seconds you should see email with subject line "[FIRING:1] ZpoolDegradedState ($HOSTNAME node warning degraded $POOL_NAME)"