AWS Cloud Showcases

The goal of this repository is to demonstrate and evaluate AWS services and to provide an infrastructure for additional showcases.

Overview

Infrastructure

Edit

Examples

• Kubernetes

  • Demonstrate DNS capability of the cluster

  • Examples which demonstrate persistence

  • Examples which demonstrate how to provide external access

• Confluent Enterprise

  • Confluent AuthN with LDAP and AuthZ with RBAC

  • Confluent AuthN with two LDAPs and AuthZ with RBAC

  • Confluent AuthN with Kerberos and AuthZ with RBAC

• Confluent Community

  • Simple Kafka Deployment for Docker Compose and Kubernetes

• LDAP

  • OpenLDAP Proxy for two ADs

Prerequisites

Tools

Download and install the following tools:

• Terraform

• AWS CLI

• Kubectl

• Helm

Configure the tools:

Create AWS Cli profile

export SHOWCASE_AWS_PROFILE=tu-dev-ueisele
aws configure --profile ${SHOWCASE_AWS_PROFILE}

IAM Permissions

The required IAM permissions can be found in the individual directories.

• S3 Terraform State Backend Setup

• VPC IAM Policy

• EKS IAM Policy

• Kubernetes IAM Policy

To capture the required policy, the tool iamlive was used. How to use it together with terraform is describe in the blog article Determining AWS IAM Policies According To Terraform And AWS CLI.

Roughly summarized the following steps are required:

Install iamlive

go install github.com/iann0036/iamlive@latest

Start iamlive in proxy mode

iamlive \
    --profile ${SHOWCASE_AWS_PROFILE} \
    --mode proxy --bind-addr 0.0.0.0:10080 \
    --force-wildcard-resource \
    --output-file required-iam-policy.json

Navigate to the terminal in which you want to run terraform and set the proxy env variables

export HTTP_PROXY=http://127.0.0.1:10080 \
       HTTPS_PROXY=http://127.0.0.1:10080 \
       AWS_CA_BUNDLE="${HOME}/.iamlive/ca.pem"

If you now run terraform apply, the required policy is recorded by iamlive

terraform apply --var "profile=${SHOWCASE_AWS_PROFILE}"

Next Steps

Kubernetes Authentication and Authorization:

• https://aws.amazon.com/blogs/containers/introducing-oidc-identity-provider-authentication-amazon-eks/

• https://aws.amazon.com/blogs/containers/kubernetes-rbac-and-iam-integration-in-amazon-eks-using-a-java-based-kubernetes-operator/

• https://github.com/rustrial/aws-eks-iam-auth-controller

• https://github.com/aws-samples/k8s-rbac-iam-java-operator/tree/master/java-operator

Security

• Security Groups for Pods: https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html

• Network Isolation with Calico: https://docs.aws.amazon.com/eks/latest/userguide/calico.html

Access

• VPN Server for secure access

• Public access for Confluent Platform

Resource Usage

• Evaluate Cluster Autoscaler and alternatives (see https://towardsdev.com/karpenter-vs-cluster-autoscaler-dd877b91629b). At the moment for example, even if Node Affinity perfers an ARM instance, an x64 instance is started.

• Autoscaling should remove replace an instance with a smaler type if possible.