Elastic Metrics

Collect performance metrics from ElasticSearch.

elasticmetrics is a Python library, designed to be used in different contexts and easy to integrate with other tools. Each step of data collection, transformation and reporting is defined in a separate reusable module:

• collectors: abstract logic of collecting data.
• metrics: abstract selecting and/or aggregating measurments (metrics)
• formatters: transform metrics into other formats.
• tool: combine the functionality of other modules to form a CLI application

Collectors

collectors.ElasticSearchCollector collects cluster and node stats by calling ElasticSearch APIs.

from elasticmetrics.collectors import ElasticSearchCollector

collector = ElasticSearchCollector('es.example.org')
collector.cluster_health()  # call _cluster/health, get ES cluster high level stats
collector.cluster_stats()  # call _cluster/stats, get ES cluster detailed stats
collector.node_stats()  # call _node/_local/stats, get ES node detailed stats


# collector supports detailed configurations like
# SSL, basic HTTP auth with UTF-8 credentials, and control over SSL context
insecure_ssl_collector = ElasticSearchCollector(
                            'localhost',
                            port=9200,
                            scheme='https',
                            user=u't€stuser',
                            password=u't€stpássword',
                            ssl_context={'no_cert_verify': True}
                         )

The returned values are exactly what's returned from the Elastic APIs.

Composing Features

Features from different modules can be composed together to achieve expected behavior.

from elasticmetrics.collectors import ElasticSearchCollector
from elasticmetrics.metrics import node_performance_metrics
from elasticmetrics.formatters import flatten_metrics

collector = ElasticSearchCollector(
                'es.example.org',
                scheme='https',
                user='testuser',
                password='testpassword'
            )
metrics_as_dotted_paths = flatten_metrics(
    node_performance_metrics(collector.node_stats()),
    prefix='example_es_server'
)
# metrics_as_dotted_paths can be pushed to a time series backend, like Graphite

Installation

$ pip install elasticmetrics

The only dependencies are Python 2.7+/3.4+ and setuptools.

However on development (and test) environment pytest, mock and pycodestyle are required.

# on dev/test env
$ pip install -r requirements/dev.txt

CLI Tool

elasticmetrics.tool is a CLI program that exposes some of the functionlaty of the library. It'll execute when imported:

$ python -m elasticmetrics.tool --help

Elastic credentials can be passed as arguments, or set as environment variables. The example below will connect to ElasticSearch listening on the default port on localhost over HTTPS, and only collect node stats, and reads access credentials from environment variables.

$ export ELASTICSEARCH_USER="someuser"
$ export ELASTICSEARCH_PASSWORD="somepassword"
$ python -m elasticmetrics.tool --ssl --quiet --collect node_stats

Development

• Code is on GitHub

Tests

Tox is most convenient to run tests with, since it handles creation of virtualenvs

$ tox

Or when development dependencies are installed (preferably with a virtual environment), tests can be run by directly calling pytest.

$ pytest

License

elasticmetrics is released under the terms of the MIT license.

The MIT License (MIT)

Copyright (c) 2019 Byte B.V.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.