Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[libbeat]: Add lowercase processor #41424

Merged
merged 54 commits into from
Nov 5, 2024
Merged

Conversation

khushijain21
Copy link
Contributor

@khushijain21 khushijain21 commented Oct 24, 2024

Proposed commit message

Part of #22254

This PR implements the lowercase processor. It continues the implementation done here #34022. It converts event keys that match the provided fields to lowercase. It performs case insensitive matching by default

Example:

Fields:  []string{"Field1"}   // list of provided fields

Event:{
          "Field1": "Field2",
          "Field3": "Value",
},

Output:{
        "field1": "Field2",
        "Field3": "Value",
},

NewAlterFieldProcessor can be used to extend uppercase processor.

@khushijain21 khushijain21 requested a review from a team as a code owner October 24, 2024 10:11
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 24, 2024
@khushijain21 khushijain21 changed the title [libbeat]: Add lowercase_key processor [WIP] [libbeat]: Add lowercase_key processor Oct 24, 2024
Copy link
Contributor

mergify bot commented Oct 24, 2024

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b issue-22254 upstream/issue-22254
git merge upstream/main
git push upstream issue-22254

Copy link
Contributor

mergify bot commented Oct 24, 2024

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @khushijain21? 🙏.
For such, you'll need to label your PR with:

  • The upcoming major version of the Elastic Stack
  • The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

Copy link
Contributor

mergify bot commented Oct 24, 2024

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

@mergify mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Oct 24, 2024
@pierrehilbert pierrehilbert added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Oct 24, 2024
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 24, 2024
Copy link
Member

@rdner rdner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a good start, mostly naming requests.

I think we could replace some existing processors like https://www.elastic.co/guide/en/beats/filebeat/current/rename-fields.html with the new centralized alterFieldNameProcessor. This would simplify things and would re-use most of the code.

libbeat/processors/actions/changeFieldProcessor.go Outdated Show resolved Hide resolved
libbeat/processors/actions/changeFieldProcessor.go Outdated Show resolved Hide resolved
libbeat/processors/actions/changeFieldProcessor.go Outdated Show resolved Hide resolved
libbeat/processors/actions/changeFieldProcessor.go Outdated Show resolved Hide resolved
libbeat/processors/actions/lowercase_fields.go Outdated Show resolved Hide resolved
libbeat/processors/actions/lowercase_fields.go Outdated Show resolved Hide resolved
libbeat/processors/actions/lowercase_fields.go Outdated Show resolved Hide resolved
@rdner
Copy link
Member

rdner commented Oct 24, 2024

How should nested keys be handled https://github.com/elastic/beats/pull/34022/files#r1050555165

let's add a config option full_path: false (default value is false). When false we only modify the last segment, otherwise the entire path. Needs to be documented with examples.

should we add one more config option for user to specify if "key", "value" or "both" should be converted to lowercase instead having a different processor for each of them?

Let's focus on keys only this time. The original feature request mentioned only keys and customers can use ingest pipelines to process values or the script processor https://www.elastic.co/guide/en/beats/filebeat/current/processor-script.html

If we need this in the future, we can add a new configuration parameter values: []string that would lowercase string values for the given list of keys.

Copy link
Member

@rdner rdner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some naming left-overs and typos.

Main points:

  • If we want to support full_path: true there is no way around traversing the graph of the entire event.

  • Also, the field name match has to be case-insensitive which also requires traversing the graph and listing all keys at each level. The match should be strings.ToLower("existing_key") == strings.ToLower("key_from_processor_config").

libbeat/processors/actions/alterFieldProcessor.go Outdated Show resolved Hide resolved
libbeat/processors/actions/alterFieldProcessor.go Outdated Show resolved Hide resolved
libbeat/processors/actions/alterFieldProcessor.go Outdated Show resolved Hide resolved
libbeat/processors/actions/alterFieldProcessor.go Outdated Show resolved Hide resolved
libbeat/processors/actions/alterFieldProcessor.go Outdated Show resolved Hide resolved
libbeat/processors/actions/docs/lowercase.asciidoc Outdated Show resolved Hide resolved
libbeat/processors/actions/docs/lowercase.asciidoc Outdated Show resolved Hide resolved
libbeat/processors/actions/docs/lowercase.asciidoc Outdated Show resolved Hide resolved
libbeat/processors/actions/lowercase.go Outdated Show resolved Hide resolved
libbeat/processors/actions/lowercase.go Outdated Show resolved Hide resolved
Copy link
Member

@rdner rdner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it makes sense to move some functionality from this PR to the actual mapstr.M implementation. Namely, we can slightly modify mapFind https://github.com/elastic/elastic-agent-libs/blob/4babafd5ed1e5079acf74212ed3da01740b22de7/mapstr/mapstr.go#L473-L485 to support case-insensitive search. Then we can expose it by GetValueFold function.

libbeat/processors/actions/alterFieldProcessor.go Outdated Show resolved Hide resolved
libbeat/processors/actions/alterFieldProcessor.go Outdated Show resolved Hide resolved
libbeat/processors/actions/alterFieldProcessor.go Outdated Show resolved Hide resolved
libbeat/processors/actions/alterFieldProcessor.go Outdated Show resolved Hide resolved
libbeat/processors/actions/alterFieldProcessor.go Outdated Show resolved Hide resolved
libbeat/processors/actions/alterFieldProcessor.go Outdated Show resolved Hide resolved
libbeat/processors/actions/alterFieldProcessor.go Outdated Show resolved Hide resolved
libbeat/processors/actions/alterFieldProcessor.go Outdated Show resolved Hide resolved
libbeat/processors/actions/docs/lowercase.asciidoc Outdated Show resolved Hide resolved
libbeat/processors/actions/docs/lowercase.asciidoc Outdated Show resolved Hide resolved
libbeat/processors/actions/alterFieldProcessor.go Outdated Show resolved Hide resolved
libbeat/processors/actions/alterFieldProcessor.go Outdated Show resolved Hide resolved
libbeat/processors/actions/alterFieldProcessor.go Outdated Show resolved Hide resolved
libbeat/processors/actions/alterFieldProcessor.go Outdated Show resolved Hide resolved
libbeat/processors/actions/alterFieldProcessor.go Outdated Show resolved Hide resolved
libbeat/processors/actions/docs/lowercase.asciidoc Outdated Show resolved Hide resolved
CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved
libbeat/processors/actions/lowercase_test.go Show resolved Hide resolved
@pierrehilbert
Copy link
Collaborator

Error: some files are not up-to-date. Run 'make update' then review and commit the changes. Modified: [libbeat/processors/actions/lowercase_test.go]
make[1]: *** [scripts/Makefile:153: check] Error 1
make[1]: Leaving directory '/home/runner/work/beats/beats/libbeat'

Copy link
Member

@rdner rdner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few improvements.

libbeat/processors/actions/alterFieldProcessor.go Outdated Show resolved Hide resolved
libbeat/processors/actions/alterFieldProcessor.go Outdated Show resolved Hide resolved
libbeat/processors/actions/docs/lowercase.asciidoc Outdated Show resolved Hide resolved
libbeat/processors/actions/docs/lowercase.asciidoc Outdated Show resolved Hide resolved
libbeat/processors/actions/docs/lowercase.asciidoc Outdated Show resolved Hide resolved
libbeat/processors/actions/docs/lowercase.asciidoc Outdated Show resolved Hide resolved
libbeat/processors/actions/lowercase_test.go Show resolved Hide resolved
Copy link
Member

@rdner rdner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you for addressing the comments!

@khushijain21 khushijain21 merged commit bfde79f into elastic:main Nov 5, 2024
142 checks passed
mergify bot pushed a commit that referenced this pull request Nov 5, 2024
* [libbeat] Add lowercase_fields and uppercase_fields processors

* [libbeat]: Add lowercase_key processor

* only loweercase

* changelog

* addressed comments minus documentation

* addressed comments & documentation

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/alterFieldProcessor.go

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/alterFieldProcessor.go

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/alterFieldProcessor.go

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/lowercase.go

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/alterFieldProcessor.go

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/alterFieldProcessor.go

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/alterFieldProcessor.go

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/lowercase.go

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* case insensitive search

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* case insensitive search

* using flattenkeys

* changelog

* cosmetic changes

* lowercase with flattenKeys

* lowercase with foldvalue

* Update CHANGELOG.next.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* resolved conflicts

* added license

* full_path search

* cosmetic changes

* using alterpath

* update version

* refactor

* linter issues

* fix

* fix

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/alterFieldProcessor.go

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* skip mandatory fields with test

* skip mandatory fields

* additonal logic to skipping mandatory field

* optimization

* fix test case

* fix flaky test

---------

Co-authored-by: davidifr <davidfr.mail@gmail.com>
Co-authored-by: Denis <denis@rdner.de>
(cherry picked from commit bfde79f)
rdner pushed a commit that referenced this pull request Nov 5, 2024
* [libbeat] Add lowercase_fields and uppercase_fields processors

* [libbeat]: Add lowercase_key processor

* only loweercase

* changelog

* addressed comments minus documentation

* addressed comments & documentation

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/alterFieldProcessor.go

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/alterFieldProcessor.go

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/alterFieldProcessor.go

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/lowercase.go

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/alterFieldProcessor.go

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/alterFieldProcessor.go

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/alterFieldProcessor.go

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/lowercase.go

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* case insensitive search

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* case insensitive search

* using flattenkeys

* changelog

* cosmetic changes

* lowercase with flattenKeys

* lowercase with foldvalue

* Update CHANGELOG.next.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* resolved conflicts

* added license

* full_path search

* cosmetic changes

* using alterpath

* update version

* refactor

* linter issues

* fix

* fix

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/alterFieldProcessor.go

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* Update libbeat/processors/actions/docs/lowercase.asciidoc

Co-authored-by: Denis <denis@rdner.de>

* skip mandatory fields with test

* skip mandatory fields

* additonal logic to skipping mandatory field

* optimization

* fix test case

* fix flaky test

---------

Co-authored-by: davidifr <davidfr.mail@gmail.com>
Co-authored-by: Denis <denis@rdner.de>
(cherry picked from commit bfde79f)

Co-authored-by: Khushi Jain <khushi.jain@elastic.co>
@@ -245,7 +245,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
- Add process.entity_id, process.group.name and process.group.id in add_process_metadata processor. Make fim module with kprobes backend to always add an appropriately configured add_process_metadata processor to enrich file events {pull}38776[38776]

*Auditbeat*

- Add `lowercase` processor. {issue}22254[22254] {pull}41424[41424]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to go in the *Affecting all Beats* section :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-8.x Automated backport to the 8.x branch with mergify enhancement Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants