-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[libbeat]: Add lowercase processor #41424
Conversation
This pull request is now in conflicts. Could you fix it? 🙏
|
This pull request does not have a backport label.
To fixup this pull request, you need to add the backport labels for the needed
|
|
Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's a good start, mostly naming requests.
I think we could replace some existing processors like https://www.elastic.co/guide/en/beats/filebeat/current/rename-fields.html with the new centralized alterFieldNameProcessor
. This would simplify things and would re-use most of the code.
let's add a config option
Let's focus on keys only this time. The original feature request mentioned only keys and customers can use ingest pipelines to process values or the script processor https://www.elastic.co/guide/en/beats/filebeat/current/processor-script.html If we need this in the future, we can add a new configuration parameter |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some naming left-overs and typos.
Main points:
-
If we want to support
full_path: true
there is no way around traversing the graph of the entire event. -
Also, the field name match has to be case-insensitive which also requires traversing the graph and listing all keys at each level. The match should be
strings.ToLower("existing_key") == strings.ToLower("key_from_processor_config")
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it makes sense to move some functionality from this PR to the actual mapstr.M
implementation. Namely, we can slightly modify mapFind
https://github.com/elastic/elastic-agent-libs/blob/4babafd5ed1e5079acf74212ed3da01740b22de7/mapstr/mapstr.go#L473-L485 to support case-insensitive search. Then we can expose it by GetValueFold
function.
b733741
to
010ba77
Compare
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few improvements.
Co-authored-by: Denis <denis@rdner.de>
Co-authored-by: Denis <denis@rdner.de>
Co-authored-by: Denis <denis@rdner.de>
Co-authored-by: Denis <denis@rdner.de>
Co-authored-by: Denis <denis@rdner.de>
bb3ee1b
to
93ca71f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thank you for addressing the comments!
* [libbeat] Add lowercase_fields and uppercase_fields processors * [libbeat]: Add lowercase_key processor * only loweercase * changelog * addressed comments minus documentation * addressed comments & documentation * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/alterFieldProcessor.go Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/alterFieldProcessor.go Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/alterFieldProcessor.go Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/lowercase.go Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/alterFieldProcessor.go Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/alterFieldProcessor.go Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/alterFieldProcessor.go Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/lowercase.go Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * case insensitive search * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * case insensitive search * using flattenkeys * changelog * cosmetic changes * lowercase with flattenKeys * lowercase with foldvalue * Update CHANGELOG.next.asciidoc Co-authored-by: Denis <denis@rdner.de> * resolved conflicts * added license * full_path search * cosmetic changes * using alterpath * update version * refactor * linter issues * fix * fix * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/alterFieldProcessor.go Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * skip mandatory fields with test * skip mandatory fields * additonal logic to skipping mandatory field * optimization * fix test case * fix flaky test --------- Co-authored-by: davidifr <davidfr.mail@gmail.com> Co-authored-by: Denis <denis@rdner.de> (cherry picked from commit bfde79f)
* [libbeat] Add lowercase_fields and uppercase_fields processors * [libbeat]: Add lowercase_key processor * only loweercase * changelog * addressed comments minus documentation * addressed comments & documentation * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/alterFieldProcessor.go Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/alterFieldProcessor.go Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/alterFieldProcessor.go Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/lowercase.go Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/alterFieldProcessor.go Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/alterFieldProcessor.go Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/alterFieldProcessor.go Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/lowercase.go Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * case insensitive search * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * case insensitive search * using flattenkeys * changelog * cosmetic changes * lowercase with flattenKeys * lowercase with foldvalue * Update CHANGELOG.next.asciidoc Co-authored-by: Denis <denis@rdner.de> * resolved conflicts * added license * full_path search * cosmetic changes * using alterpath * update version * refactor * linter issues * fix * fix * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/alterFieldProcessor.go Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * Update libbeat/processors/actions/docs/lowercase.asciidoc Co-authored-by: Denis <denis@rdner.de> * skip mandatory fields with test * skip mandatory fields * additonal logic to skipping mandatory field * optimization * fix test case * fix flaky test --------- Co-authored-by: davidifr <davidfr.mail@gmail.com> Co-authored-by: Denis <denis@rdner.de> (cherry picked from commit bfde79f) Co-authored-by: Khushi Jain <khushi.jain@elastic.co>
@@ -245,7 +245,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff] | |||
- Add process.entity_id, process.group.name and process.group.id in add_process_metadata processor. Make fim module with kprobes backend to always add an appropriately configured add_process_metadata processor to enrich file events {pull}38776[38776] | |||
|
|||
*Auditbeat* | |||
|
|||
- Add `lowercase` processor. {issue}22254[22254] {pull}41424[41424] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This needs to go in the *Affecting all Beats*
section :)
Proposed commit message
Part of #22254
This PR implements the
lowercase
processor. It continues the implementation done here #34022. It converts event keys that match the provided fields to lowercase. It performs case insensitive matching by defaultExample:
NewAlterFieldProcessor
can be used to extenduppercase
processor.