CloudWatch collectd plugin

The CloudWatch collectd plugin is a publishing extension for collectd, an open source statistic gathering daemon. With our extension all configured collectd metrics are automatically published to CloudWatch. This plugin allows monitoring of servers and applications within and outside of EC2 instances. The plugin offers additional functionality for EC2 customers such as automatic discovery of Instance ID and AWS region.

Attention: collectd python plugin is required for some collectd distribution.

• For example: for redhat distribution, run "yum install -y collectd-python".

Installation

• Download installation script, place it on the instance and execute it:

chmod u+x setup.py
sudo ./setup.py

• Follow on screen instructions

Configuration

Plugin specific configuration

The default location of the configuration file used by collectd-cloudwatch plugin is: /opt/collectd-plugins/cloudwatch/config/plugin.conf. The parameters in this file are optional when plugin is executed on EC2 instance. This file allows modification of the following parameters:

• credentials_path - Used to point to AWS account configuration file
• region - Manual override for region used to publish metrics
• host - Manual override for EC2 Instance ID and Host information propagated by collectd
• proxy_server_name - Manual override for proxy server name, used by plugin to connect aws cloudwatch at *.amazonaws.com.
• proxy_server_port - Manual override for proxy server port, used by plugin to connect aws cloudwatch at *.amazonaws.com.
• enable_high_resolution_metrics - The storage resolution is for high resolution support
• flush_interval_in_seconds - The flush_interval_in_seconds is used for flush interval, it means how long plugin should flush the metrics to Cloudwatch
• whitelist_pass_through - Used to enable potentially unsafe regular expressions. By default regex such as a line containing .* or .+ only is automatically disabled in the whitelist configuration. Setting this value to True may result in a large number of metrics being published. Before changing this parameter, read pricing information to understand how to estimate your bill.
• dimensions_path - Path to file that contains custom dimension definition. (see Adding EC2 metadata dimensions to metrics below for details)
• push_asg - Used to include the Auto-Scaling Group as a dimension for all metrics (see Adding simple dimensions to metrics below for details)
• push_constant - Used to include a Fixed dimension (see constant_dimension_value below) on all metrics. Useful for collating all metrics of a certain type (see Adding simple dimensions to metrics below for details)
• constant_dimension_value - Used to specify the value for the Fixed dimension (see Adding simple dimensions to metrics below for details)
• debug - Provides verbose logging of metrics emitted to CloudWatch

Example configuration file

credentials_path = "/home/user/.aws/credentials"
region = "us-west-1"
host = "Server1"
proxy_server_name = "http://myproxyserver.com"
proxy_server_port = "8080"
whitelist_pass_through = False
dimensions_path = "/opt/collectd-plugins/cloudwatch/config/dimensions"
push_asg = False
push_constant = True
constant_dimension_value = "ALL"
debug = False
enable_high_resolution_metrics = False
flush_interval_in_seconds = 60

Adding simple dimensions to metrics

We support adding both the ASG name to dimensions, as well as a "fixed dimension". Fixed dimensions are an additional value that will be added all metrics.

Example configuration file

push_constant = True
constant_dimension_value = "MyConstantValueHere"

The above configuration will result in all metrics being pushed with "FixedDimension" : "MyConstantValueHere"

Example configuration file

push_constant = True
constant_dimension_value = "MyConstantValueHere"

The above configuration will result in all metrics being pushed with "FixedDimension" : "MyConstantValueHere"

Example configuration file

push_asg = False

The above configuration will push the AutoScaling Group name for metrics as well

Adding EC2 metadata dimensions to metrics

User can specify in a dimension file the instance metadata that he/she wants to be pushed to aws cloudwatch along with the metric information. For example, a user can specify region, availability-zone, private-ip, instanceid, and more in the dimension file. In effect those attributes will be pushed along with the metric data to aws cloudwatch giving more clarity and information to the user about the particular metric(s).

Usage: Create dimensions file in whatever location you want. Recommended Path: /opt/collectd-plugins/cloudwatch/config/dimensions

Sample Dimensions File:

Host
PrivateIp
InstanceId
Region

After setup.py is ran there will be a dimensions file located at the dimensions_path file location. By default, dimensions will be Host and PluginInstance. Any key within the instance dimension document is supported however the first letter of the key will have to be captilized and there should be no spaces at the end of the key.

For example, the key in the identity document the key imageId would become ImageId in the dimensions file.

If you update a dimension in the dimensions file collectd will need to be restarted. Setup.py must be ran in interactive mode.

AWS account configuration

The account configuration is optional for EC2 instances with IAM Role attached. By default the AWS account configuration file is expected to be stored in: /opt/collectd-plugins/cloudwatch/config/.aws/credentials. The following parameters can be configured in the above file:

• aws_access_key - Access Key ID for account with permissions to write to CloudWatch
• aws_secret_key - Secret Access Key for the above account

Example configuration file

aws_access_key = valid_access_key
aws_secret_key = valid_secret_key

Whitelist configuration

The CloudWatch collectd plugin allows users to select metrics to be published. This is done by adding metric names or regular expressions written in python regex syntax to the whitelist config file. The default location of this configuration is: /opt/collectd-plugins/cloudwatch/config/whitelist.conf.

Warning:

1. In order to send all metrics from collectd directly to CloudWatch you can add the following rule to the whitelist.conf: .*
2. The metric names in CloudWatch are slightly different from the metric key used in the white list.

Example configuration:

swap--swap-free
memory--memory-.*
df-.*-percent_bytes-used

Effect:

1. Only the swap.swap.free metric will be published from all swap metrics
2. All memory metrics will be published
3. The df.percent_bytes.used metric will be published for every file system reported by df plugin

Usage

Once the plugin is configured correctly, restart collectd to load new configuration.

sudo /etc/init.d/collectd restart

From now on your collectd metrics will be published to CloudWatch.

Troubleshooting

Our plugin uses collectd logfile plugin. In order to enable logging in collectd, modify the collectd.conf to contain the following section:

LoadPlugin logfile

<Plugin logfile>
       LogLevel info
       File "/var/log/collectd.log"
       Timestamp true
       PrintSeverity false
</Plugin>

The collectd log can be filtered for CloudWatch plugin events using grep:

grep "[AmazonCloudWatchPlugin]" /var/log/collectd.log

Contributing

1. Create your fork by clicking Fork button on top of the page.
2. Download your repository: git clone https://github.com/USER/cloudwatch-collectd-plugin.git
3. Create your feature branch: git checkout -b my-new-feature
4. Commit your changes: git commit -am 'My new feature description'
5. Push to the branch: git push origin my-new-feature
6. Submit a pull request

License

The MIT License (MIT)

Copyright 2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.