changelog

logdata-anomaly-miner (2.7.0) unstable; urgency=low
  Bugfixes:
  * Fix broken links to python-modules
  * Fix initalization time index in EFD
  * Fix correct time index when learn mode is off
  * Fix edge case where confidence is divided by 0 in EventFrequencyDetector
  * Fix time until learn_mode is switched automatically
  * Fix issue where numerical values could not be used in CharsetDetector and EntropyDetector
  * Fix feature list order in PCA detector
  * Fix time index persistence in EFD
  * Fix last seen log in EFD
  * Fix newline parsing in json string values.
  * Fix issue of nested optional json dictionaries not parsing.

  Changes:
  * Implemented mechanism to allow granular configuration of LogResources.
  * Add seasonality to EFD
  * Add atom time in reset counter method
  * Add output_event_handlers parameter to the unparsed event handlers.
  * Add stop_learning_time and stop_learning_no_anomaly_time to schemas
  * Change id_path_list parameter to be optional in ValueRangeDetector.
  * Imporove JsonModelElement parser
  * Add learn mode and detection thresholds in MatchValueAverageChangeDetector
  * Implement AminerId config property
  * Add LogResource to the json output of anomalies
  * Add use_real_time parameter to yaml config
  * Move manpages to Debian independent directory
  * Add ignore_log_resource parameter to analysis components.
  * Improved performance of JsonModelElement
  * Add XmlModelElement
  * Implement stop_learning_time parameters in MatchValueAverageChangeDetector.
  * Add support for Debian Bookworm
  * Updated to urllib 1.26.19
  * Updated scipy to 1.10.0
  * Extend missing timestamp warning in ByteStreamLineAtomizer.
  * Enhance timezones in DateTimeModelElement
  * Increase default persistence time
  * Add support for Redhat based Linux
  * Improved documenation
  

 -- Markus Wurzenberger <markus.wurzenberger@ait.ac.at> Thu,  11 Jul 2024 12:00:00 +0000


logdata-anomaly-miner (2.6.1) unstable; urgency=low
  Changes:
  * minor refactoring
  * fixed code styling issues

 -- Markus Wurzenberger <markus.wurzenberger@ait.ac.at> Tue,  21 Feb 2023 12:00:00 +0000

logdata-anomaly-miner (2.6.0) unstable; urgency=low
  Bugfixes:
  * fixed bug in JsonModelElement where the aminer gets stuck in an endless loop searching for \x.
  * added input file path sanitization and fixed exception handling.
  * fixed a test for the remote control save config method.
  * fixed bug, that occured when starting one of the detectors VTD, VCD and TSA with an already existing persistency of the ETD, but not of the detectors.
  * fixed the MissingMatchPathValueDetector by comparing the detector_info[0] instead of the old_last_seen_timestamp.
  * ParserCount: Fixed timestamp in output
  * implemented the output_logline parameter in the NewMatchPathValueDetector.
  * fixed bug where the MissingMatchPathListValueDetector could not be used in yaml, because the ConfigValidator could not load the module.
  * runHowToEntropyDetector had missing permissions on CFG_PATH in some lines.
  * fixed bug with closing the streams.
  Changes:
  * renamed schemas to python files.
  * enabled systemd autorestart
  * improved documentation
  * added SlidingEventFrequencyDetector
  * added timestamp_scale parameter to the DateTimeModelElement.
  * added unique path param for EFD
  * added check so EXP_TYPE_MANDATORY is enforced.
  * replace raw data output with last log of event type rather than end of time window
  * added event count cluster detector
  * added experimental jsonstringparser
  * improved parameter consistency
  * added ScoringEventHandler
  * EFD: Added the functionality to analze the scoring_path_list with the ScoringEventHandler
  * ETD/TSA: Moved the initialization part of the TSA from the ETD to the TSA
  * support for ZeroMQ-Eventhandler
  * added support for named-pipes

 -- Markus Wurzenberger <markus.wurzenberger@ait.ac.at> Fri,  20 Jan 2023 12:00:00 +0000

logdata-anomaly-miner (2.5.1) unstable; urgency=low
  Bugfixes:
  * EFD: Fixed problem that appears with empty windows
  * Fixed index out of range if matches are empty in JsonModelElement array.
  * EFD: Fixed problem that appears with empty windows
  * EFD: Enabled immediate detection without training, if both limits are set
  * EFD: Fixed bug related to auto_include_flag
  * Remove spaces in aminer logo
  * ParserCounter: Fixed do_timer
  * Fixed code to allow the usage of AtomFilterMatchAction in yaml configs
  * Fixed JsonModelElement when json object is null
  * Fix incorrect message of charset detector
  * Fix match list handling for json objects
  * Fix incorrect message of charset detector
  Changes:
  * Added nullable functionality to JsonModelElements
  * Added include-directive to supervisord.conf
  * ETD: Output warning when count first exceeds range
  * EFD: Added option to output anomaly when the count first exceeds the range
  * VTD: Added variable type 'range'
  * EFD: Added the function reset_counter
  * EFD: Added option to set the lower and upper limit of the range interval
  * Enhance EFD to consider multiple time windows
  * VTD: Changed the value of parameter num_updates_until_var_reduction to track all variables from False to 0.
  * PAD: Used the binom_test of the scipy package as test if the model should be reinitialized if too few anomalies occur than are expected
  * Add ParsedLogAtom to aminer parser to ensure compatibility with lower versions
  * Added script to add build-id to the version-string
  * Support for installations from source in install-script
  * Fixed and stadardize the persistence time of various detectors
  * Refactoring
  * Improve performance
  * Improve output handling
  * Improved testing

 -- Markus Wurzenberger <markus.wurzenberger@ait.ac.at> Mon,  09 May 2022 12:00:00 +0000

logdata-anomaly-miner (2.5.0) unstable; urgency=low
  Bugfixes:
  * Fixed bug in YamlConfig
  Changes:
  * Added supervisord to docker
  * Moved unparsed atom handlers to analysis(yamlconfig)
  * Moved new_match_path_detector to analysis(yamlconfig)
  * Refactor: merged all UnparsedHandlers into one python-file
  * Added remotecontrol-command for reopening eventhandlers
  * Added config-parameters for logrotation
  * Improved testing

 -- Markus Wurzenberger <markus.wurzenberger@ait.ac.at>  Fri,  03 Dec 2021 12:00:00 +0000

logdata-anomaly-miner (2.4.2) unstable; urgency=low
  Bugfixes:
  * PVTID: Fixed output format of previously appeared times
  * VTD: Fixed bugs (static -> discrete)
  * VTD: Fixed persistency-bugs
  * Fixed %z performance issues
  * Fixed error where optional keys with an array type are not parsed when being null
  * Fixed issues with JasonModelElement
  * Fixed persistence handling for ValueRangeDetector
  * PTSAD: Fixed a bug, which occurs, when the ETD stops saving the values of one analyzed path
  * ETD: Fixed the problem when entries of the match_dictionary are not of type MatchElement
  * Fixed error where json data instead of array was parsed successfully.
  Changes:
  * Added multiple parameters to VariableCorrelationDetector
  * Improved VTD
  * PVTID: Renamed parameter time_window_length to time_period_length
  * PVTID: Added check if atom time is None
  * Enhanced output of MTTD and PVTID
  * Improved docker-compose-configuration
  * Improved testing
  * Enhanced PathArimaDetector
  * Improved documentation
  * Improved KernelMsgParsingModel
  * Added pretty print for json output
  * Added the PathArimaDetector
  * TSA: Added functionality to discard arima models with too few log lines per time step
  * TSA: improved confidence calculation
  * TSA: Added the option to force the period length
  * TSA: Automatic selection of the pause area of the ACF
  * Extended EximGenericParsingModel
  * Extended AudispdParsingModel

 -- Markus Wurzenberger <markus.wurzenberger@ait.ac.at>  Tue,  23 Nov 2021 12:00:00 +0000

logdata-anomaly-miner (2.4.1) unstable; urgency=low
  Bugfixes:
  * Fixed issues with array of arrays in JsonParser
  * Fixed problems with invalid json-output
  * Fixed ValueError in DTME
  * Fixed error with parsing floats in scientific notation with the JsonModelElement.
  * Fixed issue with paths in JsonModelElement
  * Fixed error with \x encoded json
  * Fixed error where EMPTY_ARRAY and EMPTY_OBJECT could not be parsed from the yaml config
  * Fixed a bug in the TSA when encountering a new event type
  * Fixed systemd script
  * Fixed encoding errors when reading yaml configs
  Changes:
  * Add entropy detector
  * Add charset detector
  * Add value range detector
  * Improved ApacheAccessModel, AudispdParsingModel
  * Refactoring
  * Improved documentation
  * Improved testing
  * Improved schema for yaml-config
  * Added EMPTY_STRING option to the JsonModelElement
  * Implemented check to report unparsed atom if ALLOW_ALL is used with data
    with a type other than list or dict

 -- Markus Wurzenberger <markus.wurzenberger@ait.ac.at>  Fri,  23 Jul 2021 12:00:00 +0000

logdata-anomaly-miner (2.4.0) unstable; urgency=low
  Bugfixes:
  * Fixed error in JsonModelElement
  * Fixed problems with umlauts in JsonParser
  * Fixed problems with the start element of the ElementValueBranchModelElement
  * Fixed issues with the stat and debug command line parameters
  * Fixed issues if posix acl are not supported by the filesystem
  * Fixed issues with output for non ascii characters
  * Modified kafka-version
  Changes:
  * Improved command-line-options install-script
  * Added documentation
  * Improved VTD CM-Test
  * Improved unit-tests
  * Refactoring
  * Added TSAArimaDetector
  * Improved ParserCount
  * Added the PathValueTimeIntervalDetector
  * Implemented offline mode
  * Added PCA detector
  * Added timeout-paramter to ESD

 -- Markus Wurzenberger <markus.wurzenberger@ait.ac.at>  Fri,  04 Jun 2021 12:00:00 +0000

logdata-anomaly-miner (2.3.1) unstable; urgency=low
  Bugfixes:
  * Replaced username and groupname with uid and gid for chown()
  * Removed hardcoded username and groupname

 -- Markus Wurzenberger <markus.wurzenberger@ait.ac.at>  Thu,  08 Apr 2021 12:00:00 +0000

logdata-anomaly-miner (2.3.0) unstable; urgency=low
  Bugfixes:
  * Changed pyyaml-version to 5.4
  * NewMatchIdValueComboDetector: Fix allow multiple values per id path
  * ByteStreamLineAtomizer: fixed encoding error
  * Fixed too many open directory-handles
  * Added close() function to LogStream
  Changes:
  * Added EventFrequencyDetector
  * Added EventSequenceDetector
  * Added JsonModelElement
  * Added tests for Json-Handling
  * Added command line parameter for update checks
  * Improved testing
  * Splitted yaml-schemas into multiple files
  * Improved support for yaml-config
  * YamlConfig: set verbose default to true
  * Various refactoring

 -- Markus Wurzenberger <markus.wurzenberger@ait.ac.at>  Mon,  29 Mar 2021 12:00:00 +0000

logdata-anomaly-miner (2.2.1) unstable; urgency=low
  Bugfixes:
  * Fixed warnigs due to files in Persistency-Directory
  * Fixed ACL-problems in dockerfile and autocreate /var/lib/aminer/log
  Changes:
  * added simple test for dockercontainer
  * negate result of the timeout-command. 1 is okay. 0 must be an error
  * added bullseye-tests
  * make tmp-dir in debian-bullseye-test and debian-buster-test unique

 -- Markus Wurzenberger <markus.wurzenberger@ait.ac.at>  Mon,  25 Jan 2021 12:00:00 +0000

logdata-anomaly-miner (2.2.0) unstable; urgency=low

  Changes:
  * Added Dockerfile
  * Addes checks for acl of persistency directory
  * Added VariableCorrelationDetector
  * Added tool for managing multiple persistency files
  * Added supress-list for output
  * Added suspend-mode to remote-control
  * Added requirements.txt
  * Extended documentation
  * Extended yaml-configuration-support
  * Standardize command line parameters
  * Removed --Forground cli parameter
  * Fixed Security warnings by removing functions that allow race-condition
  * Refactoring
  * Ethical correct naming of variables
  * Enhanced testing
  * Added statistic outputs
  * Enhanced status info output
  * Changed global learn_mode behavior
  * Added RemoteControlSocket to yaml-config
  * Reimplemented the default mailnotificationhandler
  Bugfixes:
  * Fixed typos in documentation
  * Fixed issue with the AtomFilter in the yaml-config
  * Fixed order of ETD in yaml-config
  * Fixed various issues in persistency

 -- Markus Wurzenberger <markus.wurzenberger@ait.ac.at>  Fri,  18 Dec 2020 17:00:00 +0000

logdata-anomaly-miner (2.1.0) unstable; urgency=low

  Changes:
  * Added VariableTypeDetector,EventTypeDetector and EventCorrelationDetector
  * Added support for unclean format strings in the DateTimeModelElement
  * Added timezones to the DateTimeModelElement
  * Enhanced ApacheAccessModel
  * Yamlconfig: added support for kafka stream
  * Removed cpu limit configuration
  * Various refactoring
  * Yamlconfig: added support for more detectors
  * Added new command-line-parameters
  * Renamed executables to aminer.py and aminerremotecontroly.py
  * Run Aminer in forgroundd-mode per default
  * Added various unit-tests
  * Improved yamlconfig and checks
  * Added start-config for parser to yamlconfig
  * Renamed config templates
  * Removed imports from init.py for better modularity
  * Created AnalysisComponentsPerformanceTests for the EventTypeDetector
  * Extended demo-config
  * Renamed whitelist to allowlist
  * Added warnings for non-existent resources
  * Changed default of auto_include_flag to false
  Bugfixes:
  * Fixed some exit() in forks
  * Fixed debian files
  * Fixed JSON output of the AffectedLogAtomValues in all detectors
  * Fixed normal output of the NewMatchPathValueDetector
  * Fixed reoccuring alerting in MissingMatchPathValueDetector

 -- Markus Wurzenberger <markus.wurzenberger@ait.ac.at>  Thu,  05 Nov 2020 17:00:00 +0000

logdata-anomaly-miner (2.0.2) unstable; urgency=low

  Changes:
  * Added help parameters
  * Added help-screen
  * Added version parameter
  * Adden path and value filter
  * Change time model of ApacheAccessModel for arbitrary time zones
  * Update link to documentation
  * Added SECURITY.md
  * Refactoring
  * Updated man-page
  * Added unit-tests for loadYamlconfig
  Bugfixes:
  * Fixed header comment type in schema file
  * Fix debian files

 -- Markus Wurzenberger <markus.wurzenberger@ait.ac.at>  Wed,  17 Jul 2020 17:00:00 +0000

logdata-anomaly-miner (2.0.1) unstable; urgency=low

  Changes:
  * Updated documentation
  * Updated testcases
  * Updated demos
  * Updated debian files
  * Added copyright headers
  * Added executable bit to AMiner

 -- Markus Wurzenberger <markus.wurzenberger@ait.ac.at>  Wed,  24 Jun 2020 17:00:00 +0000

logdata-anomaly-miner (2.0.0) bionic; urgency=low

  Changes:
  * Updated documentation
  * Added functions getNameByComponent and getIdByComponent to
    AnalysisChild.py
  * Update DefaultMailNotificationEventHandler.py to python3
  * Extended AMinerRemoteControl
  * Added support for configuration in yaml format
  * Refactoring
  * Added KafkaEventHandler
  * Added JsonConverterHandler
  * Added NewMatchIdValueComboDetector
  * Enabled multiple default timestamp paths
  * Added debug feature ParserCount
  * Added unit and integration tests
  * Added installer script
  * Added VerboseUnparsedHandler
  Bugfixes including:
  * Fixed dependencies in Debian packaging
  * Fixed typo in various analysis components
  * Fixed import of ModelElementInterface in various parsing components
  * Fixed issues with byte/string comparison
  * Fixed issue in DecimalIntegerValueModelElement, when parsing
    integer including sign and padding character
  * Fixed unnecessary long blocking time in SimpleMultisourceAtomSync
  * Changed minum matchLen in DelimitedDataModelElement to 1 byte
  * Fixed timezone offset in ModuloTimeMatchRule
  * Minor bugfixes

 -- Markus Wurzenberger <markus.wurzenberger@ait.ac.at>  Fri,  29 May 2020 17:00:00 +0000

logdata-anomaly-miner (1.0.0) bionic; urgency=low

  Changes:
  * Ported code to Python 3
  * Code cleanup using pylint
  * Added util/JsonUtil.py to encode byte strings for storing them as json objects
  * Added docs/development-procedures.txt which documents development procedures
  Features:
  * New MissingMatchPathListValueDetector to detect stream interuption
  * Added parsing support for kernel IP layer martian package messages
  * Systemd parsing of apt invocation messages.
  Bugfixes:
  * AnalysisChild: handle remote control client connection errors correctly
  * Various bugfixes

 -- Markus Wurzenberger <markus.wurzenberger@ait.ac.at>  Tue,  2 Oct 2018 17:00:00 +0000

logdata-anomaly-miner (0.0.8) xenial; urgency=low

  Apart from bugfixes, new parsing and analysis components were added:
  * Base64StringModelElement
  * DecimalFloatValueModelElement
  * StringRegexMatchRule
  * EnhancedNewMatchPathValueComboDetector

 -- Roman Fiedler <roman.fiedler@ait.ac.at>  Tue, 30 May 2017 17:00:00 +0000

logdata-anomaly-miner (0.0.7) xenial; urgency=low

  The datetime parsing DateTimeModelElement was reimplemented
  to fix various shortcomings of strptime in Python and libc.
  This will require changes in configuration due to API changes,
  e.g.:

  -time_model=DateTimeModelElement('time', '%b %d %H:%M:%S', 15, False)
  +time_model=DateTimeModelElement('time', '%b %d %H:%M:%S')

  See /usr/lib/logdata-anomaly-miner/aminer/parsing/DateTimeModelElement.py
  source code documentation for currently supported datetime format
  options.

  The code for reading log input was improved to allow also input
  from UNIX sockets. Thus the configuration was changed to support
  those modes:

  -config_properties['LogFileList']=['/var/log/auth.log', ...
  +config_properties['LogResourceList'] = ['file:///var/log/auth.log', ...

 -- Roman Fiedler <roman.fiedler@ait.ac.at>  Mon, 9 Jan 2017 18:00:00 +0000

logdata-anomaly-miner (0.0.6) xenial; urgency=low

  The input IO-handling was redesigned, thus introducing following
  API changes. The changes are flaged with (D)eveloper and (U)ser
  to indicate if only developers of own AMiner addons are affected
  or also users may need to migrate their configuration.

  * Upper layers receive LogAtom objects instead of log lines,
    parsing data as separate parameters. Thus also separate paths
    for forwarding of parsed and unparsed atoms are not required
    any more. See below for details (D, U):

    * Update any own UnparsedAtomHandler/ParsedAtomHandlerInterface
      implementations to use new interface "input.AtomHandlerInterface"
      and access to additional information to new methods and
      fields (D):

  -from aminer.parsing import ParsedAtomHandlerInterface
  +from aminer.input import AtomHandlerInterface
  -class YourHandler(ParsedAtomHandlerInterface, ...
  +class YourHandler(AtomHandlerInterface,
  -  def receiveParsedAtom(self, atom_data, parser_match):
  +  def receive_atom(self, log_atom):
  -    timestamp=parser_match.get_default_timestamp()
  +    timestamp=log_atom.get_timestamp()
  +    parser_match=log_atom.parser_match
  -    print '%s' % atom_data
  +    print '%s' % log_atom.rawData

    * With parsed/unparsed atom processing path convergence, naming
      of other classes does not make sense any more (U):

  -from aminer.analysis import VolatileLogarithmicBackoffParsedAtomHistory
  +from aminer.util import VolatileLogarithmicBackoffAtomHistory
  - from aminer.analysis import ParsedAtomFilters
  + from aminer.analysis import AtomFilters
  - match_action=Rules.ParsedAtomFilterMatchAction(...
  + match_action=Rules.AtomFilterMatchAction(...

  - parsed_atom_handlers=[]
  - unparsed_atom_handlers=[]
  - analysis_context.atomizer_factory=SimpleByteStreamLineAtomizerFactory(
  -     parsing_model, parsed_atom_handlers, unparsed_atom_handlers, ...
  + atom_filter=AtomFilters.SubhandlerFilter(None)
  + analysis_context.atomizer_factory=SimpleByteStreamLineAtomizerFactory(
  +     parsing_model, [atom_filter], ...

  For handling of unparsed atoms:

  - unparsed_atom_handlers.append(SimpleUnparsedAtomHandler(anomaly_event_handlers))
  + atom_filter.add_handler(SimpleUnparsedAtomHandler(anomaly_event_handlers),
  +     stop_when_handled_flag=True)

  For handling of parsed atoms:

  - parsed_atom_handlers.append(...
  + atom_filter.add_handler(...

 -- Roman Fiedler <roman.fiedler@ait.ac.at>  Fri, 4 Nov 2016 18:00:00 +0000

logdata-anomaly-miner (0.0.5) xenial; urgency=low

  Following API changes were introduced:

  * Lower input layers dealing with binary data stream reading,
    splitting into atoms and forwarding data to the parsing model
    were redesigned. Following configuration changes are required
    to adapt "config.py" and probably "analysis.py" to the new
    API:

    * analysis_context.register_component(): register_as_raw_atom_handler
      parameter not needed any more, can be removed.

    * SimpleParsingModelRawAtomHandler is not needed any more,
      that part can be replaced by configuration:

  # Now define the AtomizerFactory using the model. A simple line
  # based one is usually sufficient.
    from aminer.input import SimpleByteStreamLineAtomizerFactory
    analysis_context.atomizer_factory=SimpleByteStreamLineAtomizerFactory(
        parsing_model, parsed_atom_handlers, unparsed_atom_handlers,
        anomaly_event_handlers, default_timestamp_paths=['/model/syslog/time'])

    * SimpleUnparsedAtomHandler was moved from "aminer.events"
      to "aminer.input".

 -- Roman Fiedler <roman.fiedler@ait.ac.at>  Mon, 11 Oct 2016 18:00:00 +0000

logdata-anomaly-miner (0.0.4) xenial; urgency=low

  Following API changes were introduced:

  * Event handling (general): Change of EventHandlerInterface
    to include also event_source as last parameter. See
    /usr/lib/logdata-anomaly-miner/aminer/events/__init__.py

  * VolatileLogarithmicBackoffEventHistory: Added event ID and
    source to stored tuple to allow unique identification of events.
    Split result of "getHistory()" to include "eventId, eventType,
    event_message, sorted_log_lines, event_data, event_source".

 -- Roman Fiedler <roman.fiedler@ait.ac.at>  Fri, 26 Aug 2016 15:15:00 +0000

logdata-anomaly-miner (0.0.3) xenial; urgency=low

  Following API changes were introduced:

  * To improve readability of configuration files, main parser,
    analysis and event classes were added to the submodule namespaces.
    After imports directly from the submodule, e.g.
    "from aminer.parsing import FixedDataModelElement",
    the name duplication "FixedDataModelElement.FixedDataModelElement"
    is not required any more, "FixedDataModelElement" is sufficient.
    Use "sed -i -e 's/Name.Name/Name/g' [files]" to adapt.

  * Component timing was restructured to allow forensic/realtime
    triggering. Therefore also clean interface was added, which
    is now also used to reduce redundant code in component registration.
    Old way:

    analysis_context.register_component(new_match_path_detector,
      component_name=None, register_as_raw_atom_handler=False,
      register_as_time_triggered_handler=True)

    New way:

    analysis_context.register_component(new_match_path_detector,
      register_as_raw_atom_handler=False)

    For own custom time-triggered components, make sure to implement
    the "aminer.util.TimeTriggeredComponentInterface". Use any standard
    component, e.g. "/usr/lib/logdata-anomaly-miner/aminer/analysis/NewMatchPathDetector.py"
    as example.

  * Introduction of "AnalysisContext" to have common handle for
    all data required to perform the analysis. Therefore also
    the signature of "build_analysis_pipeline" in "config.py/analysis.py"
    has changed from

    def build_analysis_pipeline(aminer_config):

    to

    def build_analysis_pipeline(analysis_context):

    Old references to "aminer_config" within the configuration
    script have to be replaced by "analysis_context.aminer_config".

 -- Roman Fiedler <roman.fiedler@ait.ac.at>  Thu, 21 Jul 2016 19:00:00 +0000