-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
output/eve: reduce fflush call count #12135
base: master
Are you sure you want to change the base?
Conversation
Issue: 3449 Add a flush function to packet logger registration and collapse the parameter count for registration functions.
This commit adds 2 EVE output buffering settings - buffer-size value which specifies the amount of buffering, if any, for regular/file output types. - flush-interval Specifies the cadence at which Suricata will direct detect threads to flush EVE output. Issue: 3449
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #12135 +/- ##
===========================================
+ Coverage 62.68% 83.07% +20.39%
===========================================
Files 840 910 +70
Lines 153669 258147 +104478
===========================================
+ Hits 96323 214462 +118139
+ Misses 57346 43685 -13661
Flags with carried forward coverage won't be shown. Click here to find out more.
|
Issue: 3449
Issue: 3449 Add flushing functions and infrastructure. This includes: - Flushing functions for packet loggers - Log file flushing support
Issue: 3449 Add a flush directive to the packet that is distinct from the existing "log flush" flag as the new flag is to distinguish between the 2 use cases.
Issue: 3449
Information: QA ran without warnings. Pipeline 23450 |
@@ -280,3 +284,14 @@ outputs: | |||
# event-set: false # log packets that have a decoder/stream event | |||
# state-update: false # log packets triggering a TCP state update | |||
# spurious-retransmission: false # log spurious retransmission packets | |||
# | |||
heartbeat: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should this not belong in outputs section ?
Dummy question : why do we need the heartbeat flushing ? |
Continuation of #11883
Reduce fflush calls on output streams (regular files only).
Output can be buffered, specify the buffer-size with
outputs.<type>.buffer-size
. A value of 0 selects no buffering; otherwise, up to the buffer-size value can be buffered. Note that this buffering is part of the stdio library.Since output can be buffered, a mechanism that periodically flushes the output streams has been added. The
heartbeat.output-flush-interval
configuration setting specifies at what interval the output should be flushed. A value of 0 means never flush.Link to redmine ticket: 3449
Describe changes:
buffer-size
. When 0, unbuffered I/O is used; other values are used to set the stdio buffer size. The value isoutputs.eve-log.buffer-size
heartbeat.output-flush-interval
-- to set cadence for Suricata periodically directing detect threads to flush EVE output. To be used in conjunction withbuffer-size
. Setheartbeat.output-flush-interval
to the number of seconds Suricata should periodically cause the EVE output to be flushed. The default value is0
which instructs Suricata never to cause the EVE output to be flushed.heartbeat.output-flush-interval
heartbeat.output-flush-interval
is between 1 and 60 in seconds).Updates:
Suricata build information
Benchmarks/Measurements
Hyperfine was used to measure results with my pcap collection and ET Pro
Summary: Buffering had the biggest impact; using the flushing mechanism had little impact but is necessary for integrity.
Recommendation:
eve-log.buffer-size
: TBDheartbeat.output-flush-interval
TBDPermutations benchmarked for
buffer-size
andoutput-flush-interval
Hyperfine output