Merged matsgoran's Elasticsearch 7 branch

.editorconfig

@@ -0,0 +1,17 @@
+root = true
+
+[*]
+end_of_line = lf
+insert_final_newline = true
+charset = utf-8
+
+[*.py]
+indent_style = space
+indent_size = 4
+
+[Makefile]
+indent_style = tab
+
+[{*.json,*.yml,*.yaml}]
+indent_style = space
+indent_size = 2

.travis.yml

@@ -6,7 +6,30 @@ env:
 - TOXENV=py27
 install:
 - pip install tox
-script: make test
+- >
+  if [[ -n "${ES_VERSION}" ]] ; then
+    wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-${ES_VERSION}.tar.gz
+    mkdir elasticsearch-${ES_VERSION} && tar -xzf elasticsearch-${ES_VERSION}.tar.gz -C elasticsearch-${ES_VERSION} --strip-components=1
+    ./elasticsearch-${ES_VERSION}/bin/elasticsearch &
+  fi
+script:
+- >
+  if [[ -n "${ES_VERSION}" ]] ; then
+    wget -q --waitretry=1 --retry-connrefused --tries=30 -O - http://127.0.0.1:9200
+    make test-elasticsearch
+  else
+    make test
+  fi  
+jobs:
+  include:
+    - stage: 'Elasticsearch test'
+      env: TOXENV=py27 ES_VERSION=7.0.0-linux-x86_64
+    - env: TOXENV=py27 ES_VERSION=6.6.2
+    - env: TOXENV=py27 ES_VERSION=6.3.2
+    - env: TOXENV=py27 ES_VERSION=6.2.4
+    - env: TOXENV=py27 ES_VERSION=6.0.1
+    - env: TOXENV=py27 ES_VERSION=5.6.16
+
 deploy:
   provider: pypi
   user: yelplabs

Makefile

@@ -16,6 +16,9 @@ install-hooks:
 test:
 	tox
 
+test-elasticsearch:
+	tox -- --runelasticsearch
+
 test-docker:
 	docker-compose --project-name elastalert build tox
 	docker-compose --project-name elastalert run tox

README.md

@@ -51,9 +51,11 @@ Currently, we have built-in support for the following alert types:
 - AWS SNS
 - VictorOps
 - PagerDuty
+- PagerTree
 - Exotel
 - Twilio
 - Gitter
+- Line Notify
 
 Additional rule types and alerts can be easily imported or written.
 
@@ -93,7 +95,7 @@ Eg: ``--rule this_rule.yaml``
 Available at the [ElastAlert Kibana plugin repository](https://github.com/bitsensor/elastalert-kibana-plugin).
 
 ### Docker
-A [Dockerized version](https://github.com/bitsensor/elastalert) of ElastAlert including a REST api is build from `master` to `bitsensor/elastalert:latest`. 
+A [Dockerized version](https://github.com/bitsensor/elastalert) of ElastAlert including a REST api is build from `master` to `bitsensor/elastalert:latest`.
 
 ```bash
 git clone https://github.com/bitsensor/elastalert.git; cd elastalert

changelog.md

@@ -1,14 +1,35 @@
 # Change Log
 
+# v0.1.38
+
+### Added
+- Added PagerTree alerter
+- Added Line alerter
+- Added more customizable logging
+- Added new logic in test-rule to detemine the default timeframe
+
+### Fixed
+- Fixed an issue causing buffer_time to sometimes be ignored
+
+# v0.1.37
+
+### Added
+- Added more options for Opsgenie alerter
+- Added more pagerduty options
+- Added ability to add metadata to elastalert logs
+
+### Fixed
+- Fixed some documentation to be more clear
+- Stop requiring doc_type for metric aggregations
+- No longer puts quotes around regex terms in blacklists or whitelists
+
 # v0.1.36
 
 ### Added
 - Added a prefix "metric_" to the key used for metric aggregations to avoid possible conflicts
 - Added option to skip Alerta certificate validation
-- Added option to gracefully fail when loading rules
 
 ### Fixed
-- No longer puts quotes around regex terms in blacklists or whitelists
 - Fixed a typo in the documentation for spike rule
 
 # v0.1.35

config.yaml.example

@@ -62,3 +62,53 @@ writeback_index: elastalert_status
 # sending the alert until this time period has elapsed
 alert_time_limit:
   days: 2
+
+# Custom logging configuration
+# If you want to setup your own logging configuration to log into
+# files as well or to Logstash and/or modify log levels, use
+# the configuration below and adjust to your needs.
+# Note: if you run ElastAlert with --verbose/--debug, the log level of
+# the "elastalert" logger is changed to INFO, if not already INFO/DEBUG.
+#logging:
+#  version: 1
+#  incremental: false
+#  disable_existing_loggers: false
+#  formatters:
+#    logline:
+#      format: '%(asctime)s %(levelname)+8s %(name)+20s %(message)s'
+#
+#    handlers:
+#      console:
+#        class: logging.StreamHandler
+#        formatter: logline
+#        level: DEBUG
+#        stream: ext://sys.stderr
+#
+#      file:
+#        class : logging.FileHandler
+#        formatter: logline
+#        level: DEBUG
+#        filename: elastalert.log
+#
+#    loggers:
+#      elastalert:
+#        level: WARN
+#        handlers: []
+#        propagate: true
+#
+#      elasticsearch:
+#        level: WARN
+#        handlers: []
+#        propagate: true
+#
+#      elasticsearch.trace:
+#        level: WARN
+#        handlers: []
+#        propagate: true
+#
+#      '':  # root logger
+#        level: WARN
+#          handlers:
+#            - console
+#            - file
+#        propagate: false

docs/source/elastalert.rst

@@ -194,6 +194,20 @@ The default value is ``.raw`` for Elasticsearch 2 and ``.keyword`` for Elasticse
 
 ``skip_invalid``: If ``True``, skip invalid files instead of exiting.
 
+=======
+Logging
+-------
+
+By default, ElastAlert uses a simple basic logging configuration to print log messages to standard error.
+You can change the log level to ``INFO`` messages by using the ``--verbose`` or ``--debug`` command line options.
+
+If you need a more sophisticated logging configuration, you can provide a full logging configuration
+in the config file. This way you can also configure logging to a file, to Logstash and
+adjust the logging format.
+
+For details, see the end of ``config.yaml.example`` where you can find an example logging
+configuration.
+
 
 .. _runningelastalert:
 

docs/source/recipes/writing_filters.rst

@@ -97,7 +97,7 @@ For ranges on fields::
 Negation, and, or
 *****************
 
-Any of the filters can be embedded in ``not``, ``and``, and ``or``::
+For Elasticsearch 2.X, any of the filters can be embedded in ``not``, ``and``, and ``or``::
 
     filter:
     - or:
@@ -113,6 +113,13 @@ Any of the filters can be embedded in ``not``, ``and``, and ``or``::
                 term:
                   _type: "something"
 
+For Elasticsearch 5.x, this will not work and to implement boolean logic use query strings::
+
+    filter:
+     - query:
+          query_string:
+            query: "somefield: somevalue OR foo: bar"
+            
 
 Loading Filters Directly From Kibana 3
 --------------------------------------

docs/source/ruletypes.rst

@@ -1078,6 +1078,9 @@ Optional:
 ``query_key``: Group metric calculations by this field. For each unique value of the ``query_key`` field, the metric will be calculated and
 evaluated separately against the threshold(s).
 
+``min_doc_count``: The minimum number of events in the current window needed for an alert to trigger.  Used in conjunction with ``query_key``,
+this will only consider terms which in their last ``buffer_time`` had at least ``min_doc_count`` records.  Default 1.
+
 ``use_run_every_query_size``: By default the metric value is calculated over a ``buffer_time`` sized window. If this parameter is true
 the rule will use ``run_every`` as the calculation window.
 
@@ -1097,6 +1100,54 @@ allign with the time elastalert runs, (This both avoid calculations on partial d
 See: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-datehistogram-aggregation.html#_offset for a
 more comprehensive explaination.
 
+Spike Aggregation
+~~~~~~~~~~~~~~~~~~
+
+``spike_aggregation``: This rule matches when the value of a metric within the calculation window is ``spike_height`` times larger or smaller
+than during the previous time period. It uses two sliding windows to compare the current and reference metric values.
+We will call these two windows "reference" and "current".
+
+This rule requires:
+
+``metric_agg_key``: This is the name of the field over which the metric value will be calculated. The underlying type of this field must be
+supported by the specified aggregation type.  If using a scripted field via ``metric_agg_script``, this is the name for your scripted field
+
+``metric_agg_type``: The type of metric aggregation to perform on the ``metric_agg_key`` field. This must be one of 'min', 'max', 'avg',
+'sum', 'cardinality', 'value_count'.
+
+``spike_height``: The ratio of the metric value in the last ``timeframe`` to the previous ``timeframe`` that when hit
+will trigger an alert.
+
+``spike_type``: Either 'up', 'down' or 'both'. 'Up' meaning the rule will only match when the metric value is ``spike_height`` times
+higher. 'Down' meaning the reference metric value is ``spike_height`` higher than the current metric value. 'Both' will match either.
+
+``buffer_time``: The rule will average out the rate of events over this time period. For example, ``hours: 1`` means that the 'current'
+window will span from present to one hour ago, and the 'reference' window will span from one hour ago to two hours ago. The rule
+will not be active until the time elapsed from the first event is at least two timeframes. This is to prevent an alert being triggered
+before a baseline rate has been established. This can be overridden using ``alert_on_new_data``.
+
+Optional:
+
+``query_key``: Group metric calculations by this field. For each unique value of the ``query_key`` field, the metric will be calculated and
+evaluated separately against the 'reference'/'current' metric value and ``spike height``.
+
+``metric_agg_script``: A `Painless` formatted script describing how to calculate your metric on-the-fly::
+
+    metric_agg_key: myScriptedMetric
+    metric_agg_script:
+        script: doc['field1'].value * doc['field2'].value
+
+``threshold_ref``: The minimum value of the metric in the reference window for an alert to trigger. For example, if
+``spike_height: 3`` and ``threshold_ref: 10``, then the 'reference' window must have a metric value of 10 and the 'current' window at
+least three times that for an alert to be triggered.
+
+``threshold_cur``: The minimum value of the metric in the current window for an alert to trigger. For example, if
+``spike_height: 3`` and ``threshold_cur: 60``, then an alert will occur if the current window has a metric value greater than 60 and
+the reference window is less than a third of that value.
+
+``min_doc_count``: The minimum number of events in the current window needed for an alert to trigger.  Used in conjunction with ``query_key``,
+this will only consider terms which in their last ``buffer_time`` had at least ``min_doc_count`` records.  Default 1.
+
 Percentage Match
 ~~~~~~~~~~~~~~~~
 
@@ -1629,7 +1680,9 @@ Provide absolute address of the pciture, for example: http://some.address.com/im
 
 ``slack_alert_fields``: You can add additional fields to your slack alerts using this field. Specify the title using `title` and a value for the field using `value`. Additionally you can specify whether or not this field should be a `short` field using `short: true`.
 
-``slack_title_link``: You can add a link in your Slack notification by setting this to a valid URL.
+``slack_title``: Sets a title for the message, this shows up as a blue text at the start of the message
+
+``slack_title_link``: You can add a link in your Slack notification by setting this to a valid URL. Requires slack_title to be set.
 
 ``slack_timeout``: You can specify a timeout value, in seconds, for making communicating with Slac. The default is 10. If a timeout occurs, the alert will be retried next time elastalert cycles.
 
@@ -1752,6 +1805,15 @@ See https://v2.developer.pagerduty.com/docs/send-an-event-events-api-v2
 
 ``pagerduty_v2_payload_source_args``: If set, and ``pagerduty_v2_payload_source`` is a formattable string, Elastalert will format the source based on the provided array of fields from the rule or match.
 
+PagerTree
+~~~~~~~~~
+
+PagerTree alerter will trigger an incident to a predefined PagerTree integration url.
+
+The alerter requires the following options:
+
+``pagertree_integration_url``: URL generated by PagerTree for the integration.
+
 Exotel
 ~~~~~~
 
@@ -2000,6 +2062,16 @@ Example usage::
     jira_alert_owner: $owner$
 
 
+
+Line Notify
+~~~~~~~~~~~
+
+Line Notify will send notification to a Line application. The body of the notification is formatted the same as with other alerters.
+
+Required:
+
+``linenotify_access_token``: The access token that you got from https://notify-bot.line.me/my/
+
 theHive
 ~~~~~~~
 
@@ -2045,3 +2117,4 @@ Example usage::
         - domain: "{match[field1]}_{rule[name]}"
         - domain: "{match[field]}"
         - ip: "{match[ip_field]}"
+