V2.7.0

What's Changed

• Implemented mechanism to allow granular configuration of LogResources. by @ernstleierzopf in #1206
• rewrote unittests for the NewMatchPathDetector by @ernstleierzopf in #1210
• rewrote unittests for ParserCount by @ernstleierzopf in #1215
• rewrote unittests for ValueRangeDetector by @ernstleierzopf in #1216
• Fix broken links by @landauermax in #1220
• 203 rewrite unittests unparsed atom handlers by @ernstleierzopf in #1218
• 202 rewrite unittests timestamps unsorted detector by @ernstleierzopf in #1217
• rewrite unittests. by @ernstleierzopf in #1221
• fix JsonModelElement parser by @ernstleierzopf in #1248
• Bump scipy from 1.5.4 to 1.10.0 by @dependabot in #1247
• Add seasonality to EFD by @landauermax in #1240
• Add learn mode and detection thresholds in MatchValueAverageChangeDet… by @landauermax in #1245
• extract data initialization from VTD to another file. by @ernstleierzopf in #1251
• suppress deprecation warning for pkg_resources. by @ernstleierzopf in #1253
• add Documentation for NewMatchPathDetector and order other analysis c… by @ernstleierzopf in #1255
• implement AminerId config property. by @ernstleierzopf in #1254
• Remove setting atom_time in analysis components and implement use_real_time parameter in the yaml config. by @ernstleierzopf in #1257
• add LogResource to the json output of anomalies. by @ernstleierzopf in #1256
• 213 timestamp error handling by @ernstleierzopf in #1259
• 207 tsa arima detector missing module by @ernstleierzopf in #1250
• Bump urllib3 from 1.26.5 to 1.26.17 by @dependabot in #1265
• Move manpages to Debian independent directory by @sebastinas in #1266
• implement ignore_log_resource parameter in analysis components. by @ernstleierzopf in #1262
• 218 json model element performance by @ernstleierzopf in #1267
• Bump urllib3 from 1.26.17 to 1.26.18 by @dependabot in #1270
• implement stop_learning_time parameters in MatchValueAverageChangeDet… by @ernstleierzopf in #1263
• 214 xml log parser by @ernstleierzopf in #1261
• HOTFIX: fix testingwrapper - development branch failing by @ernstleierzopf in #1271
• add Debian Bookworm and Ubuntu 22.4 Tests. by @ernstleierzopf in #1269
• HOTFIX: fix defusedxml installation. by @ernstleierzopf in #1273
• Fix Dockerfiles for Debian/Ubuntu.\nAdd support for Fedora and RedHat OS. by @ernstleierzopf in #1278
• create different versions of the SyslogParsingModel for AIT-LDS versi… by @ernstleierzopf in #1264
• implement PersistableComponentInterface. by @ernstleierzopf in #1272
• try fixing jenkinsfile. by @ernstleierzopf in #1280
• remove interactive from Jenkinsfile. by @ernstleierzopf in #1282
• HOTFIX Jenkinsfile by @ernstleierzopf in #1285
• try fixing Jenkinsfile. by @ernstleierzopf in #1286
• made cd own sh command, otherwise $PWD is not updated in the same line. by @ernstleierzopf in #1287
• fix MultiLocaleDTME unittests for 29th Feb. by @ernstleierzopf in #1289
• add some test headers for rewriting PersistenceUtilTest. by @ernstleierzopf in #1279
• 233 fix PCADetector Documentation. by @ernstleierzopf in #1294
• 234 improve README and aminer_install to work with all supported OSes. by @ernstleierzopf in #1295
• update Jenkinsfile to run all tests in parallel. by @ernstleierzopf in #1296
• make jenkins tests independent from computing capacities. by @ernstleierzopf in #1304
• 236 charset detector fix integer handling by @ernstleierzopf in #1299
• increase kafka timeout. by @ernstleierzopf in #1308
• Create Russellmitchell reference configuration by @ernstleierzopf in #1288
• 241 dtme timestamps by @ernstleierzopf in #1316
• Fix feature list order in PCA detector by @landauermax in #1310
• extend documentation for important aminer files. by @ernstleierzopf in #1317
• rewrite unittests for JsonUtil. by @ernstleierzopf in #1318
• rewrite unittests for SecureOSFunctions. by @ernstleierzopf in #1320
• Fix time index persistence in EFD by @landauermax in #1313
• fix newline parsing in json string values. by @ernstleierzopf in #1325
• extend missing timestamp warning in ByteStreamLineAtomizer. by @ernstleierzopf in #1326
• Bump urllib3 from 1.26.18 to 1.26.19 by @dependabot in #1330
• Release 2.7.0 by @whotwagner in #1331

Full Changelog: V2.6.1...V2.7.0

Release of V2.6.1

What's Changed

• fixed test26 so no fix definition number has to be added. by @ernstleierzopf in #1182
• fixed error with string indices in YamlConfig. by @ernstleierzopf in #1200
• fixed deepsource issues. by @ernstleierzopf in #1205
• extended FrequencyDetector wiki tests. by @ernstleierzopf in #1192
• Implemented ElasticSearchWikiTest. by @ernstleierzopf in #1201
• Added AIT-LDS2 Parsers by @ernstleierzopf in #1203

Full Changelog: V2.6.0...V2.6.1

Release of V2.6.0

Bugfixes:

• fixed bug in JsonModelElement where the aminer gets stuck in an endless loop searching for \x.
• added input file path sanitization and fixed exception handling.
• fixed a test for the remote control save config method.
• fixed bug, that occured when starting one of the detectors VTD, VCD and TSA with an already existing persistency of the ETD, but not of the detectors.
• fixed the MissingMatchPathValueDetector by comparing the detector_info[0] instead of the old_last_seen_timestamp.
• ParserCount: Fixed timestamp in output
• implemented the output_logline parameter in the NewMatchPathValueDetector.
• fixed bug where the MissingMatchPathListValueDetector could not be used in yaml, because the ConfigValidator could not load the module.
• runHowToEntropyDetector had missing permissions on CFG_PATH in some lines.
• fixed bug with closing the streams.

Changes:

• renamed schemas to python files.
• enabled systemd autorestart
• improved documentation
• added SlidingEventFrequencyDetector
• added timestamp_scale parameter to the DateTimeModelElement.
• added unique path param for EFD
• added check so EXP_TYPE_MANDATORY is enforced.
• replace raw data output with last log of event type rather than end of time window
• added event count cluster detector
• added experimental jsonstringparser
• improved parameter consistency
• added ScoringEventHandler
• EFD: Added the functionality to analze the scoring_path_list with the ScoringEventHandler
• ETD/TSA: Moved the initialization part of the TSA from the ETD to the TSA
• support for ZeroMQ-Eventhandler
• added support for named-pipes

Release of V2.5.1

Bugfixes:

• EFD: Fixed problem that appears with empty windows
• Fixed index out of range if matches are empty in JsonModelElement array.
• EFD: Fixed problem that appears with empty windows
• EFD: Enabled immediate detection without training, if both limits are set
• EFD: Fixed bug related to auto_include_flag
• Remove spaces in aminer logo
• ParserCounter: Fixed do_timer
• Fixed code to allow the usage of AtomFilterMatchAction in yaml configs
• Fixed JsonModelElement when json object is null
• Fix incorrect message of charset detector
• Fix match list handling for json objects
• Fix incorrect message of charset detector

Changes:

• Added nullable functionality to JsonModelElements
• Added include-directive to supervisord.conf
• ETD: Output warning when count first exceeds range
• EFD: Added option to output anomaly when the count first exceeds the range
• VTD: Added variable type 'range'
• EFD: Added the function reset_counter
• EFD: Added option to set the lower and upper limit of the range interval
• Enhance EFD to consider multiple time windows
• VTD: Changed the value of parameter num_updates_until_var_reduction to track all variables from False to 0.
• PAD: Used the binom_test of the scipy package as test if the model should be reinitialized if too few anomalies occur than are expected
• Add ParsedLogAtom to aminer parser to ensure compatibility with lower versions
• Added script to add build-id to the version-string
• Support for installations from source in install-script
• Fixed and stadardize the persistence time of various detectors
• Refactoring
• Improve performance
• Improve output handling
• Improved testing

Release of V2.5.0

Bugfixes:

• Fixed bug in YamlConfig

Changes:

• Added supervisord to docker
• Moved unparsed atom handlers to analysis(yamlconfig)
• Moved new_match_path_detector to analysis(yamlconfig)
• Refactor: merged all UnparsedHandlers into one python-file
• Added remotecontrol-command for reopening eventhandlers
• Added config-parameters for logrotation
• Improved testing

Release of V2.4.2

Bugfixes:

• PVTID: Fixed output format of previously appeared times
• VTD: Fixed bugs (static -> discrete)
• VTD: Fixed persistency-bugs
• Fixed %z performance issues
• Fixed error where optional keys with an array type are not parsed when being null
• Fixed issues with JasonModelElement
• Fixed persistence handling for ValueRangeDetector
• PTSAD: Fixed a bug, which occurs, when the ETD stops saving the values of one analyzed path
• ETD: Fixed the problem when entries of the match_dictionary are not of type MatchElement
• Fixed error where json data instead of array was parsed successfully.

Changes:

• Added multiple parameters to VariableCorrelationDetector
• Improved VTD
• PVTID: Renamed parameter time_window_length to time_period_length
• PVTID: Added check if atom time is None
• Enhanced output of MTTD and PVTID
• Improved docker-compose-configuration
• Improved testing
• Enhanced PathArimaDetector
• Improved documentation
• Improved KernelMsgParsingModel
• Added pretty print for json output
• Added the PathArimaDetector
• TSA: Added functionality to discard arima models with too few log lines per time step
• TSA: improved confidence calculation
• TSA: Added the option to force the period length
• TSA: Automatic selection of the pause area of the ACF
• Extended EximGenericParsingModel
• Extended AudispdParsingModel

Release of V2.4.1

Bugfixes:

• Fixed issues with array of arrays in JsonParser
• Fixed problems with invalid json-output
• Fixed ValueError in DTME
• Fixed error with parsing floats in scientific notation with the JsonModelElement.
• Fixed issue with paths in JsonModelElement
• Fixed error with \x encoded json
• Fixed error where EMPTY_ARRAY and EMPTY_OBJECT could not be parsed from the yaml config
• Fixed a bug in the TSA when encountering a new event type
• Fixed systemd script
• Fixed encoding errors when reading yaml configs

Changes:

• Add entropy detector
• Add charset detector
• Add value range detector
• Improved ApacheAccessModel, AudispdParsingModel
• Refactoring
• Improved documentation
• Improved testing
• Improved schema for yaml-config
• Added EMPTY_STRING option to the JsonModelElement
• Implemented check to report unparsed atom if ALLOW_ALL is used with data
  with a type other than list or dict

Release of V2.4.0

Bugfixes:

• Fixed error in JsonModelElement
• Fixed problems with umlauts in JsonParser
• Fixed problems with the start element of the ElementValueBranchModelElement
• Fixed issues with the stat and debug command line parameters
• Fixed issues if posix acl are not supported by the filesystem
• Fixed issues with output for non ascii characters
• Modified kafka-version

Changes:

• Improved command-line-options install-script
• Added documentation
• Improved VTD CM-Test
• Improved unit-tests
• Refactoring
• Added TSAArimaDetector
• Improved ParserCount
• Added the PathValueTimeIntervalDetector
• Implemented offline mode
• Added PCA detector
• Added timeout-paramter to ESD

Release of V2.3.1

Bugfixes:

• Replaced username and groupname with uid and gid for chown()
• Removed hardcoded username and groupname

Release of V2.3.0

Bugfixes:

• Changed pyyaml-version to 5.4
• NewMatchIdValueComboDetector: Fix allow multiple values per id path
• ByteStreamLineAtomizer: fixed encoding error
• Fixed too many open directory-handles
• Added close() function to LogStream

Changes:

• Added EventFrequencyDetector
• Added EventSequenceDetector
• Added JsonModelElement
• Added tests for Json-Handling
• Added command line parameter for update checks
• Improved testing
• Splitted yaml-schemas into multiple files
• Improved support for yaml-config
• YamlConfig: set verbose default to true
• Various refactoring