A Simple Anthos Installer

A GitOps based Anthos Multi Cloud installer framework.

Note: This is not an officially supported Google product.

🥅 Goals

• Provide starter scripts to install Anthos components with minimal manual steps.
• Use CFT Terraform modules that follow GCP best practices.
• Adding/modifying/removing Anthos/GCP components should be painless.
• Use of small modules so each one can be deployed and debugged independently
• GitOps enabled.

❓ What can it Install?

• A Regional GKE Cluster on GCP in a dedicated VPC with Workload Identity, GKE Connect, Anthos Config Management (ACM) and Anthos Service Mesh (ACM) enabled.

• An EKS Cluster on AWS in a dedicated VPC with GKE Connect and Anthos Config Management (ACM) enabled. Also creates a Kubernetes Service Account to use to login to the GCP console.

🚀 Quickest Quickstart

The quickest way to get an environment without installing any tools except git and gcloud is to use CloudBuild. See README-CloudBuild.md for details.

🤓 Quickstart (tested on Linux)

Install gcloud, Terraform, Terragrunt, awscli (if EKS required). Check the pre-requisites

🖥️ Prepare

# Clone the repo
git clone https://github.com/GoogleCloudPlatform/simple-anthos-installer
cd simple-anthos-installer

# Make sure authenticate with Application default login as this required for the google provider. See https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/provider_reference#authentication
gcloud auth application-default login # Mandatory

# Setup Project Env variables
export PROJECT_ID="<GCP_PROJECTID>" # Mandatory
gcloud config set core/project ${PROJECT_ID}  # Mandatory

export GCP_REGION="gcp-region"  # Optional, Defaults to us-east1
export GCP_AZS="az-1,az-2" # Optional. Defaults to us-east1-b,us-east1-c,us-east1-d

# Install kpt tool required for ASM install
sudo apt-get install google-cloud-sdk-kpt
# Specify the ACM repo to use. You can clone this one https://github.com/GoogleCloudPlatform/csp-config-management
export ACM_REPO="git@github.com:your-git-repo/csp-config-management.git" # Mandatory if using ACM

Create the GKE Resources

# Create the GKE Cluster with Workload Identity, GKE Connect(Hub), ACM and ASM enabled.
cd terragrunt/gke-gcp
terragrunt run-all apply --terragrunt-non-interactive

Create EKS Resources

# Setup AWS credentials 
export AWS_ACCESS_KEY_ID="aws-secret-key-id" # Mandatory
export AWS_SECRET_ACCESS_KEY="aws-secret-key" # Mandatory
export AWS_REGION="aws-region" # Optional. Defaults to us-east-1


# Create the EKS Cluster connected with GKE Connect(Hub) and ACM enabled.
cd terragrunt/eks-aws
terragrunt run-all apply --terragrunt-non-interactive

This will create 2 clusters named gke-dev-01 and eks-dev-01 in GKE and EKS respectively connected to GKE Hub.

Detailed Instructions

• A Simple Anthos Installer
  • 🥅 Goals
• ❓ What can it Install?
• 🚀 Quickest Quickstart
• 🤓 Quickstart (tested on Linux)
  • 🖥️ Prepare
  • Create the GKE Resources
  • Create EKS Resources
• Detailed Instructions
• Pre-requisites
  • Local Machine
  • GCP Requirements
  • Clone (or create) a git repo you want to use for ACM
• Detailed Usage
  • 1. Create GKE Cluster on GCP
    • ACM
    • ASM
  • 2. Create EKS Cluster on AWS
    • Login to the Cluster in GCP Console
    • ACM
  • Enjoy!
  • Cleanup
• Customization and Extending
  • GKE Directory Structure
    • To change the region:
    • Modify/Add a New Environment:
  • AWS Directory Structure
  • Terraform State
• Development and Testing
  • Validating the scripts
  • Incrementally building the infrastructure
• Known Issues
• Contributing
• References
• Related Projects
• Licence

Pre-requisites

Local Machine

• Terraform 0.13.x
• Terragrunt 0.28.x
• gcloud
• awscli
• gcloud installed and configured with a GCP project.

export PROJECT_ID="<GCP_PROJECTID>"
gcloud config set core/project ${PROJECT_ID}  

• kpt installed using sudo apt-get install google-cloud-sdk-kpt see this for more details. You may need to install the other tools manually depending on how gcloud was installed.

GCP Requirements

• Following APIs are Enabled:
  • Compute
  • Cloud Build
  • Kubernetes
  • Anthos
  • Secrets Manager (to store AWS credentials)

Clone (or create) a git repo you want to use for ACM

By default it uses the reference repo git@github.com:GoogleCloudPlatform/csp-config-management.git

To change this to use your own repo, clone the above repo and modify the sync_repo variable in the files gke-gcp/us-central1/dev/5_acm/terragrunt.hcl and eks-aws/us-east-1/dev/5_acm/terragrunt.hcl to point to your repo.

Detailed Usage

1. Create GKE Cluster on GCP

From the root git folder

cd terragrunt/gke-gcp
terragrunt run-all apply --terragrunt-non-interactive

Go get some ☕ and if all goes well, in about 20 minutes, you should see this on the Anthos console in the Clusters view:

ACM

You will need to configure the cluster's ACM SSH public key on your git config management repo you created earlier. The install script outputs a value git_creds_public which is the public key to use with you git provider.

Once you have updated the SSH public key successfully, in the Anthos Config Management screen, you should see the following:

ASM

You will have to enable sidecar injection into the namespaces you want by following the directions here: https://cloud.google.com/service-mesh/docs/scripted-install/gke-install#deploying_and_redeploying_workloads

2. Create EKS Cluster on AWS

# Setup AWS credentials 
export AWS_ACCESS_KEY_ID="aws-secret-key-id"
export AWS_SECRET_ACCESS_KEY="aws-secret-key"


# Create the EKS Cluster connected with GKE Connect(Hub) and ACM enabled.
cd terragrunt/eks-aws
terragrunt run-all apply --terragrunt-non-interactive

Login to the Cluster in GCP Console

In order to see the EKS cluster details in the Anthos Dashbaord, you have to Login to the Cluster using a KSA token.

Go to the Cloud Build output for the EKS Hub module and look for the output value for ksa_token. Use this token to Login to the GCP console from the Kubernetes Clusters page.

ACM

Follow the steps from GKE above to enable ACM

Enjoy!

Now you have a 2 clusters connected to an envrion (your GCP project) with ACM enabled.

Cleanup

# From terragrunt/eks-aws or terragrunt/gke-gcp directory
cd terragrunt/eks-aws
terragrunt run-all destroy --terragrunt-non-interactive --terragrunt-ignore-dependency-errors

The above cleanup will fail deleting the vpc if your project adds additional firewall rules that this script did not create which will prevent the VPC from being deleted. Easier way would be to use a dedicated project and delete the project when you are finished with it.

Customization and Extending

Terragrunt is used to call the CFT Terraform modules as it simplifies dependency management and makes use of DRY pattern.

This project creates a dev environment

GKE Directory Structure

The terragrunt/gke-gcp directory is structured as follows:

├── account.hcl
└── us-east1
    ├── dev
    │   ├── 0_activate-apis
    │   │   └── terragrunt.hcl
    │   ├── 1_vpc
    │   │   └── terragrunt.hcl
    │   ├── 2_gke
    │   │   └── terragrunt.hcl
    │   ├── 3_hub
    │   │   └── terragrunt.hcl
    │   ├── 4_acm
    │   │   └── terragrunt.hcl
    │   ├── 5_asm
    │   │   └── terragrunt.hcl
    │   └── env.hcl
    └── region.hcl

It is organized by region (us-east1 in this case) and under that an environment (dev).

• account.hcl : contains GCP project details
• region.hcl : contains GCP region and AZ details
• env.hcl : environment name set to dev by default

These .hcl provide sensible defaults which can be overriden using environment variables.

The numbering scheme for the directories is a best practice to document the order of deployment.

To change the region:

• Change region variable in the region.hcl
• Rename the directory us-central1 to your desired region. This has no effect on the code. This is for purely documenting your code to reflect the region.

Modify/Add a New Environment:

• Change environment_name variable in the env.hcl
• Copy the directory dev and rename it to your desired environment name. Reanming the directory has no effect on the code. This is for purely documenting your code to reflect the environment. It is best practice to name the directories (region and environment) to the same as the one in the .hcl files.

AWS Directory Structure

The terragrunt/aws-eks directory is structured as similarly:

├── account.hcl
└── us-east-1
    ├── dev
    │   ├── 1_vpc
    │   │   └── terragrunt.hcl
    │   ├── 2_eks
    │   │   └── terragrunt.hcl
    │   ├── 3_hub
    │   │   └── terragrunt.hcl
    │   ├── 4_hub_login
    │   │   └── terragrunt.hcl
    │   ├── 5_acm
    │   │   └── terragrunt.hcl
    │   └── env.hcl
    └── region.hcl

Terraform State

The Terraform state is stored in GCS bucket with the naming convention terraform-state-dev-$PROJECT_ID-$REGION. The directory structure will match the directory structure above.

Development and Testing

Validating the scripts

From the gke-gcp or eks-aws directory run:

terragrunt run-all validate

A terragrunt plan can be done but it needs to be done separately in each numbered folder since there are dependencies in the some of the modules that needs to exist for a plan command to work.

Incrementally building the infrastructure

You can deploy all the infrastructure for a specific cloud provider or one module at a time.

From a numbered directory (1_xx) or its parent, run:

terragrunt run-all apply --terragrunt-non-interactive

Known Issues

• Clean Upgrade and Uninstall of ASM is not supported as the asm install scripts do not support this yet. See GoogleCloudPlatform/anthos-service-mesh-packages#480
• Install of ASM is not supported for non GKE clusters.
• There are some Cloud Build scripts, you read the instructions here

Contributing

Bug reports and feedback welcome. Please see CONTRIBUTING.md

References

Videos:

• Automated, modularized and versioned infrastructure with Terraform and Terragrunt
• 5 Lessons Learned From Writing Over 300,000 Lines of Infrastructure Code

Related Projects

• anthos-multicloud-workshop

Licence

Apache 2.0