The secret zero problem

[ManolisKoufidakis]

Hi,
I'm using azure kubernetes service and I'm implementing the Azure AD workload identity at the moment. We need to access other azure resources (like keyvault) from inside the pods and this is the solution to go (even if it is in preview).

However, as I worked on this task the following scenario came in mind and I'd like to share it here.

In order for the Azure AD Workload Identity to work, it uses a mutating admission webhook to project a signed service account token to the workload’s volume and inject some properties as environmental variables to pods.

If I get it right then the following scenario is possible: An attacker attacks the pod and gains access to the environmental variables. Then, they can use AZURE_CLIENT_ID, AZURE_TENANT_ID and azure-identity-token in order to gain an Azure AD access token. This access token can be used to gain access to whatever resources the access policies of the underlying managed identity allow.
That means that the secret zero issue is not addressed with this solution. Am I correct?

If yes, then could this be solved by not persisting the service account token in a file? Or maybe solving this is out of scope for this project?

[9Rune5]

Where would the access token be located? If the barbarians reach inside your pod they can access everything your process can. And your process needs that token one way or another (several times in fact, as the token has a tendency to expire).

Given a solution that consists of many micro services, you can assign each service a unique service account (mapped to a distinct managed id). Keep your backend services off limits to outside actors and limit each service access to an individual database. Maybe assign each service an individual key vault? How many globally accessed secrets do you really need? Does your connection string to Azure SQL contain a username/password? (it too can use a token)

When the layer reachable from the outside is limited in scope (can only access your backend stuff through simple REST calls) then you have greatly reduced the attack surface.

[ManolisKoufidakis]

What is actually located in a workload’s volume is a signed service account token. This token can be used to get an access token to access the resources that the underlying managed identity has access to.

Your recommendations are great and in fact we do all of them (different service account per service, different keyvaults, backend resources accesible via REST api etc). However, an attacker will always try to find some hole in all of these. Also, vulnerabilities always exist and they might open an undesired port in any system. Thus, maximizing the security posture is always a never ending struggle.

My original point was that we did all the trouble and removed the secrets from env variables and in general from inside the container, connected k8s OIDC server with azure active directory, assigned proper access rights to managed identities etc etc and at the end of the day we didn't solve the issue of the security of our secrets. We just made it a bit harder for an attacker to get something valuable. Of course, this is a great gain as security is additive and a combination of actions and practices. However, it's good to be concious about the end result that we achieved. This will help us improve even more and continue pushing towards more secure implementations.