[AWS] Test EKS Pod Identity support

[zmoog]

A recent AWS SDK update brought support for EKS Pod Identity.

We need to test if and which AWS-related Filebeat inputs Metricbeat metricset versions work with this authentication mechanism.

Related:

• Upgrade AWS SDK to support EKS Pod Identity  #38174

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[zmoog]

Set up an EKS cluster with EKS Pod Identity

Create an EKS cluster named mbranca-dev using eksctl:

eksctl create cluster mbranca-dev --region eu-north-1

Get the kubectl config:

aws eks update-kubeconfig --name mbranca-dev --region eu-north-1    

Set up an EKS Pod Identity Agent using the EKS' Addons. See https://docs.aws.amazon.com/eks/latest/userguide/pod-id-agent-setup.html for additional info.

• pick the correct addon version according to the cluster version; see https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html#pod-id-cluster-versions
• install the Addon named "Amazon EKS Pod Identity Agent"

Check that the is running EKS Pod Identity Agent:

$ kubectl get pods -n kube-system | grep 'eks-pod-identity-agent'
eks-pod-identity-agent-8dcbn   1/1     Running   0          19s
eks-pod-identity-agent-8hxnw   1/1     Running   0          19s

Create an IAM role with the permissions needed to collect logs and metrics (TODO: apply the least-privilege principle to the policy).

cat >my-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances",
        "ec2:DescribeRegions",
        "cloudwatch:GetMetricData",
        "cloudwatch:ListMetrics",
        "iam:ListAccountAliases",
        "rds:DescribeDBInstances",
        "rds:ListTagsForResource",
        "s3:*",
        "sns:ListTopics",
        "sqs:ChangeMessageVisibility",
        "sqs:DeleteMessage",
        "sqs:ListQueues",
        "sqs:ReceiveMessage",
        "sts:AssumeRole",
        "sts:GetCallerIdentity",
        "tag:GetResources",
        "logs:*"
      ],
      "Resource": "*"
    }
  ]
}
EOF

Create the IAM policy:

aws iam create-policy --policy-name my-policy --policy-document file://my-policy.json

Define a service account for the Elastic Agent (it's the standard service account from the quick start)

cat >my-service-account.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: elastic-agent
  namespace: kube-system
  labels:
    k8s-app: elastic-agent
EOF

kubectl apply -f my-service-account.yaml

Run the following command to create a trust policy file for the IAM role.

cat >trust-relationship.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
            "Effect": "Allow",
            "Principal": {
                "Service": "pods.eks.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession"
            ]
        }
    ]
}
EOF

aws iam create-role --role-name my-role --assume-role-policy-document file://trust-relationship.json --description "my-role-description"

aws iam attach-role-policy --role-name my-role --policy-arn=arn:aws:iam::111222333444:policy/my-policy

Finally, we assign an IAM role to a Kubernetes service account.

aws eks create-pod-identity-association --cluster-name mbranca-dev \
  --role-arn arn:aws:iam::111222333444:role/my-role \
  --namespace kube-system \
  --service-account elastic-agent \
  --region eu-north-1

Inspect the configuration we created to double-check that we're getting everything.

aws iam get-role --role-name my-role --query Role.AssumeRolePolicyDocument
aws iam list-attached-role-policies --role-name my-role --query AttachedPolicies --output text

export policy_arn=arn:aws:iam::111222333444:policy/my-policy
aws iam get-policy --policy-arn $policy_arn

# There could be multiple versions if you tweaked the policy from the AWS console.
aws iam get-policy-version --policy-arn $policy_arn --version-id v1
aws iam get-policy-version --policy-arn $policy_arn --version-id v2
aws iam get-policy-version --policy-arn $policy_arn --version-id v3

Deploy the Elastic Agent:

# From the enroll a new Elastic Agent in Fleet
kubectl apply -f elastic-agent-managed-kubernetes.yml  

$ kubectl get pods -n kube-system | grep elastic-agent
elastic-agent-5vmlf            1/1     Running   0          93s
elastic-agent-6t2zd            1/1     Running   0          93s

Double-check that AWS injected additional env vars:

kubectl describe pod elastic-agent-5vmlf -n kube-system | grep AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE:
      AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE:  /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token

kubectl describe deployment elastic-agent -n kube-system | grep "Service Account"

Enter the pod and install the AWS CLI to verify the EKS Pod Identity config is in place and I can use AWS services .

kubectl exec -it elastic-agent-5vmlf  -n kube-system -- /bin/bash

apt update
apt install python3-pip
pip3 install awscli

$ aws sts get-caller-identity
{
    "UserId": "AROAWC3A256YBIEFUZMHG:eks-mbranca-de-elastic-ag-bd932bdf-40f5-42e3-b15e-0b0437f6f3e3",
    "Account": "111222333444",
    "Arn": "arn:aws:sts::111222333444:assumed-role/my-role/eks-mbranca-de-elastic-ag-bd932bdf-40f5-42e3-b15e-0b0437f6f3e3"
}

[zmoog]

Logs

awss3 input (S3 polling mode)

TBA

awss3 input (SQS mode)

TBA

cloudwatch-logs input

# create a log group
aws logs create-log-group --log-group-name /mbranca/dev/eks-pod-identity-test --region eu-north-1

# create a log stream
aws logs create-log-stream --log-group-name /mbranca/dev/eks-pod-identity-test --log-stream-name 2024-11-12 --region eu-north-1

# create some log events
cat > events.json <<EOF
[
 {
    "timestamp": $(date +%s)000,
    "message": "Example Event 3"
  }
]
EOF

# put logs
aws logs put-log-events --log-group-name /aws/eks/mbranca-dev/cluster --log-stream-name elastic-agent --log-events file://events.json --region eu-north-1

gent policy:

inputs:
  - id: aws-cloudwatch-aws_logs-5f7eb11d-81f9-4957-8838-d694c7ac5c3e
    name: aws_logs-3
    revision: 18
    type: aws-cloudwatch
    use_output: default
    meta:
      package:
        name: aws_logs
        version: 1.4.0
    data_stream:
      namespace: ekspodidentity
    package_policy_id: 5f7eb11d-81f9-4957-8838-d694c7ac5c3e
    streams:
      - id: aws-cloudwatch-aws_logs.generic-5f7eb11d-81f9-4957-8838-d694c7ac5c3e
        data_stream:
          dataset: aws_logs.generic
        log_group_arn: >-
          arn:aws:logs:eu-north-1:111222333444:log-group:/mbranca/dev/eks-pod-identity-test:*
        start_position: beginning
        scan_frequency: 1m
        api_sleep: 200ms
        default_region: eu-north-1
        tags:
          - forwarded
        publisher_pipeline.disable_host: true

[zmoog]

Metrics

cloudwatch

Agent policy:

inputs:
  - id: aws/metrics-ec2-765d9a4f-760c-4616-823f-b5b9c60a82dd
    name: aws-2
    revision: 7
    type: aws/metrics
    use_output: default
    meta:
      package:
        name: aws
        version: 2.14.2
    data_stream:
      namespace: ekspodidentity
    package_policy_id: 765d9a4f-760c-4616-823f-b5b9c60a82dd
    streams:
      - id: aws/metrics-aws.ec2_metrics-765d9a4f-760c-4616-823f-b5b9c60a82dd
        data_stream:
          dataset: aws.ec2_metrics
          type: metrics
        metricsets:
          - cloudwatch
        period: 5m
        default_region: eu-north-1
        tags_filter: null
        metrics:
          - name:
              - CPUUtilization
              - CPUCreditUsage
              - CPUCreditBalance
              - CPUSurplusCreditBalance
              - CPUSurplusCreditsCharged
              - StatusCheckFailed
              - StatusCheckFailed_Instance
              - StatusCheckFailed_System
            namespace: AWS/EC2
            resource_type: 'ec2:instance'
            statistic:
              - Average
          - name:
              - DiskReadBytes
              - DiskReadOps
              - DiskWriteBytes
              - DiskWriteOps
              - NetworkIn
              - NetworkPacketsIn
              - NetworkOut
              - NetworkPacketsOut
            namespace: AWS/EC2
            resource_type: 'ec2:instance'
            statistic:
              - Sum

Dashboard

[gizas]

apt update
apt install python3-pip
pip3 install awscli

The step where you install awscli inside agent is only for testing purposes right?

Also in metrics how is the authentication happening?
(In logs eg. I see the log_group_arn: >-)

[zmoog]

The step where you install awscli inside agent is only for testing purposes right?

Yep. I installed the CLI only to test the EKS Pod Identity setup. I'm circling back and adding more comments for each step to clarify the step and my intent.

[zmoog]

Also in metrics how is the authentication happening?

The EKS Pod Identity (possibly eks-pod-identity-agent) and the AWS CLI handle all the heavy lifting for the app, including getting and refreshing credentials.

At a high level, EKS pod identity allows you to associate an IAM role with a service account in K8s. When a pod uses that service account, the EKS pod identity assumes the roles and passes the credentials on to the pod.

EKS pod identity is similar to the EC2 instance profile, which does the same for the whole EC2 instance. However, EKS pod identity is much more granular and allows admins to assign a dedicated role to each service account. This is pretty cool.

[gizas]

Thank you !!!! I guess this is a great candidate for an external doc.

One more qq: Why then you add the log_group_arn in logs?

[zmoog]

Thank you !!!! This is a great candidate for an external doc.

One more qq: Why then you add the log_group_arn in logs?

I was targeting log events from the log group /mbranca/dev/eks-pod-identity-test only, so I used the ARN. log_group_arn is a good option when you are targeting one log group only.