[filbeat][azure-blob-storage] - Adding support for Microsoft Entra ID RBAC authentication

[ShourieG]

Type of change

• Enhancement
• Docs

Proposed commit message

Added support for Microsoft Entra ID RBAC authentication.
Added mock tests by injecting the gock transport layer in the azure client.
Added some config tests with the new config options.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
  - [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

No Impact

Live Testing

Live testing was performed using our internal Azure dev environment.
Process followed:

1. Fresh storage account and container created and then uploaded with dummy files.
2. New service principle app created.
3. blobOwner role granted to service app via Entra ID IAM configs.
4. client_id, tenant_id and client_secret of the service app used to poll poll the blob storage.
5. Records were output to a local file with success.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Closes [filebeat][azure-blob-storage] - Add support for authorization via Microsoft Entra ID / RBAC  #40434

Testing

We performed live testing internally using an active azure account and the new authentication system is working properly after assigning the service principal app with the blobOwner permission level.

Use cases

Screenshots

Logs

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @ShourieG? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b abs/rbac upstream/abs/rbac
git merge upstream/main
git push upstream abs/rbac

[efd6]

Do we need this? From what I can see, an implementation of http.RoundTripper would satisfy the testing requirements that tests here add, arguably more cleanly.

[andrewkroh]

I had the same thought when I gave this a quick scan yesterday. The declarative setup is nice, but I'm not sure it's worth another dependency.

[ShourieG]

If we go that route then we would lose the context to the serv.URL thats local to each test. serv.URL would be required to set & override the urls in the response JSON without which the sdk will always try to call
https://login.microsoftonline.com for the token validation check, as this is hardwired into the sdk and cannot be overridden by conventional means, hence the only option is to intercept the call from within the test and override at that point, which is why I was relying on gock here. If we chose not to use gock then the option remains to set serv.URL globally which I'm not sure would be right as future test expansion could lead to test concurrency issues.

[efd6]

If we go that route then we would lose the context to the serv.URL thats local to each test.

Why? There is nothing in the gock code that could not be done easily and more transparently with local code. AFAICS the mock code provides author convenience at the cost of transparency for the maintainer.

[ShourieG]

@efd6, have removed gock and simplified the test.

[efd6]

nits only

[efd6]

Suggested change

	1. `client_id` : The client ID of the Azure Entra ID application.
	2. `client_secret` : The client secret of the Azure Entra ID application.
	3. `tenant_id` : The tenant ID of the Azure Entra ID application.
	1. `client_id`: The client ID of the Azure Entra ID application.
	2. `client_secret`: The client secret of the Azure Entra ID application.
	3. `tenant_id`: The tenant ID of the Azure Entra ID application.

[efd6]

Suggested change

	NOTE: According to our internal testing it seems that we require at least an access level of **blobOwner** for the service principle to be able to read the blobs. If you are facing any issues with the access level, please ensure that the access level is set to **blobOwner**.
	NOTE: According to our internal testing it seems that we require at least an access level of **blobOwner** for the service principle to be able to read the blobs. If you are facing any issues with the access level, ensure that the access level is set to **blobOwner**.