Monitoring

Set of scripts for evaluating various monitoring stacks (log aggregation, metrics collection, metrics correlation, etc.)

Currently, the following stacks can be installed.

• Sensu (including Uchiwa)
• Flapjack
• Logstash (with Elasticsearch/Kibana and logstash-forwarder)
• Heka
• Sentry
• Tessera
• Graylog
• Statsd
• Graphite (with Grafana)
• InfluxDB
• Flume
• Fluentd
• OpenTSDB and TCollector
• Cachet

This repository started off from the ideas on my Monitoring Everything blog series.

Goals

Scripts included in this project must:

• Follow best practices for Ubuntu and the applications being installed
• Document service boundaries (ports, sockets, etc.) and default credentials
• Be as minimal as possible
• Be as reusable as possible (middleware and persistence, for example, can be reused between applications)
• Try to stay close to each application's defaults (specifically port numbers and file locations)

This makes it easier for people to solve issues during evaluation, as well as prepare for deploying the applications into staging and production environments.

I won't accept scripts for the following applications. These applications are either too old or too bloated for cloud monitoring.

• Nagios
• Zabbix

Contributing

Want to add additional scripts? Improve existing scenarios? Just fork it, and submit a pull request. It's that simple.

Usage

Two Vagrant boxes are provided with this script.

• monitoring on 192.168.12.10
• app1 on 192.168.12.11

The monitoring box is intended for the different monitoring stacks, while app1 is intended to try out clients.

Sensu/Logstash

The monitoring box provides the following:

• Kibana
• Tessera
• Grafana
• Flapjack
• Uchiwa
• ElasticHQ
• Cachet

For client nodes, it provides:

• AMQP (TCP/5672)
• AMQP over SSL (TCP/5671)
• Redis (TCP/6379)
• Statsd (UDP/8125)
• Lumberjack receiver (TCP/5043)
• Graphite line receiver (TCP/2013)
• Graphite Pickle receiver (TCP/2014)

OpenTSDB

This package provides scripts to install OpenTSDB and TCollector. OpenTSDB depends on HBase, which is installed in pseudo-distributed mode alongside a standalone ZooKeeper.

vagrant up --no-provision monitoring
vagrant ssh monitoring

sudo -i
cd /vagrant && ./install-all-opentsdb.sh

This script also installs Grafana with the OpenTSDB backend configured. Elasticsearch is installed as a dashboard store for Grafana.

• Grafana
• OpenTSDB dashboard

TCollector is installed on the monitoring host to provide some sample metrics. Note that this script can take more than 10 minutes to install, depending on your bandwidth.

To start collecting metrics from app1, simply SSH to it and run the following.

sudo -i
cd /vagrant && ./install-tcollector.sh

Sentry

This package provides scripts to install Sentry with the following configuration.

• memcached
• supervisord
• Redis as work queue, update buffer and TSDB
• PostgreSQL as node store

You can use the following to perform the installation.

vagrant up --no-provision monitoring
vagrant ssh monitoring

sudo -i
cd /vagrant && ./install-all-sentry.sh

After installation, the Sentry web interface should be available. Login with the username admin and the password secret.

Graylog

To install Graylog with the web interface, simply use the following.

vagrant up --no-provision monitoring
vagrant ssh monitoring

sudo -i
cd /vagrant && ./install-all-graylog.sh

After installation, you can access one the web interface. Use the username admin and the password password.

Heka

Heka was created by Mozilla as a lighter alternative to Logstash. This package provides scripts for installing a Heka router that outputs to Elasticsearch.

vagrant up --no-provision monitoring
vagrant ssh monitoring

sudo -i
cd /vagrant && ./install-all-heka.sh

This package provides the following:

• Heka dashboard
• Kibana
• ElasticHQ
• Heka Protobuf input on 192.168.12.10 at TCP port 5565

InfluxDB

InfluxDB is an open-source distributed time series database with no external dependencies. This package provides scripts for using InfluxDB as a general replacement for Graphite. It will install Sensu and InfluxDB, configuring Sensu to relay metrics to InfluxDB.

vagrant up --no-provision monitoring
vagrant ssh monitoring

sudo -i
cd /vagrant && ./install-all-influxdb.sh

This package provides the following:

• Grafana configured for InfluxDB
• InfluxDB admin interface with default credentials
• Uchiwa dashboard for Sensu

Use vagrant up app1 to start collecting metrics

Flume

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. This package provides scripts for using Flume with Elasticsearch and receivers for Avro and Syslog protocols.

vagrant up --no-provision monitoring
vagrant ssh monitoring

sudo -i
cd /vagrant && ./install-all-flume.sh

This package provides the following:

• Kibana
• ElasticHQ
• Avro input on 192.168.12.10 at TCP port 41414
• Syslog input on 192.168.12.10 at TCP port 1514

When using Kibana, you will need to change the index pattern to [flume-]YYYY-MM-DD.

Fluentd

Fluentd is an open source data collector to unify log management. This package provides scripts for using Fluentd with Elasticsearch and receivers for Syslog and HTTP protocols.

vagrant up --no-provision monitoring
vagrant ssh monitoring

sudo -i
cd /vagrant && ./install-all-fluentd.sh

This package provides the following:

• Kibana
• ElasticHQ
• HTTP input on 192.168.12.10 at TCP port 9880
• Syslog input on 192.168.12.10 at TCP port 1514

Testing Syslog Receivers

To quickly test the functionality of a syslog-compatible receiver, you can use the logger command on Ubuntu.

message="hello world"

# With TCP syslog receiver
logger -n localhost -P 1514 $message

# With UDP syslog receiver
logger -n localhost -P 1514 -d $message