Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: Adding terraform bootstrap for fully private deployment #200

Merged
merged 45 commits into from
Jul 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
fe41c87
adding the e2e test for bootstrap with terraform
edgarsilva948 Jun 4, 2024
4e3654e
fixing versions comment
edgarsilva948 Jun 4, 2024
9f0484c
adding the fully private bootstrap
edgarsilva948 Jun 5, 2024
9e3cdbd
adding image push script
edgarsilva948 Jun 7, 2024
cb7adc3
fixing private images
edgarsilva948 Jun 8, 2024
cb68236
adding eks fully private sg rules
edgarsilva948 Jun 8, 2024
f695de2
fixing sg locals loop
edgarsilva948 Jun 9, 2024
7c7f796
adjusting security group rules
edgarsilva948 Jun 9, 2024
c9763ab
adjusting security group rules
edgarsilva948 Jun 9, 2024
067fe7c
adding package pull policy
edgarsilva948 Jun 9, 2024
b3a67a5
adding sleep after the family provider
edgarsilva948 Jun 9, 2024
6ea8619
upgrading upjet version
edgarsilva948 Jun 9, 2024
48ac2e6
adding the skip dependency comment
edgarsilva948 Jun 10, 2024
bc1a81e
downgrading crossplane to work fully private
edgarsilva948 Jun 10, 2024
8368352
fixing timeouts and sleep
edgarsilva948 Jun 10, 2024
2a46d4c
fixing the sg rules
edgarsilva948 Jun 10, 2024
fa9eeaf
adjusting script lua from eksctl
edgarsilva948 Jun 10, 2024
f9f442c
adding sg rules comment
edgarsilva948 Jun 10, 2024
371258d
fixing terraform bootstrap
edgarsilva948 Jun 18, 2024
0bef156
breaking the small changes into new branches
edgarsilva948 Jun 18, 2024
e26ae3a
reverting runtime config fix
edgarsilva948 Jun 18, 2024
84ce195
reverting the e2e files
edgarsilva948 Jun 18, 2024
933442b
reverting docs fix
edgarsilva948 Jun 18, 2024
9b739b0
reverting terraform main.tf fix
edgarsilva948 Jun 18, 2024
f88bf32
adding default for ecr_region and ecr_account_id, shortening the name…
candonov Jun 25, 2024
6a7aa10
ecr repos
candonov Jun 27, 2024
84ac0e7
ecr pull through cache updates
candonov Jul 3, 2024
05a18e6
removing unnessesary script
candonov Jul 3, 2024
fcde89e
typo
candonov Jul 3, 2024
3ce4351
typo
candonov Jul 3, 2024
4af7bf2
updating gif
candonov Jul 3, 2024
9686189
fixing readme and adding cidr comments
edgarsilva948 Jul 5, 2024
8d9fea1
fixing readme to include the ecr secret
edgarsilva948 Jul 5, 2024
d94f0af
adjusting scripts and readme
edgarsilva948 Jul 5, 2024
578dfe6
adding the cleanup script and adjusting readme
edgarsilva948 Jul 5, 2024
d16482f
adding the faq.md in the clean up section
edgarsilva948 Jul 5, 2024
0f3da0d
adding crane auth login
edgarsilva948 Jul 5, 2024
647e272
adding upbound/provider-aws-cloudfront repo creation
edgarsilva948 Jul 5, 2024
5807632
adding proxy comments
edgarsilva948 Jul 9, 2024
e847b21
fixing conflicts
edgarsilva948 Jul 9, 2024
1518fc1
ecr updates
candonov Jul 23, 2024
ea892fd
Update bootstrap/terraform-fully-private/README.md
candonov Jul 23, 2024
d7756df
Update bootstrap/terraform-fully-private/providers/upjet-aws/provider…
candonov Jul 23, 2024
645f7a7
Update bootstrap/terraform-fully-private/README.md
candonov Jul 23, 2024
94a59e3
adding docs to explain the org/repo requirement
edgarsilva948 Jul 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
198 changes: 198 additions & 0 deletions bootstrap/terraform-fully-private/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,198 @@
# EKS Cluster bootstrap with Terraform for Crossplane

This example deploys the following components
- Creates a new sample VPC with Two Private Subnets and the required VPC Endpoints
- Creates EKS Cluster Control plane with one managed node group
- Crossplane Add-on to EKS Cluster
- Upbound AWS Provider for Crossplane
- AWS Provider for Crossplane
- Kubernetes Provider for Crossplane
- Helm Provider for Crossplane

> [!IMPORTANT]
> Some AWS services, such as IAM and WAFv2, do not have VPC endpoints. To ensure that these services work correctly with Crossplane providers, you need to add a proxy. See an example [here](https://github.com/awslabs/crossplane-on-eks/blob/main/bootstrap/terraform-fully-private/providers/upjet-aws/runtime-config.yaml).

## Crossplane Deployment Design

```mermaid
graph TD;
subgraph AWS Cloud
id1(VPC)-->Private-Subnet1;
id1(VPC)-->Private-Subnet2;
candonov marked this conversation as resolved.
Show resolved Hide resolved
id1(VPC)-->Private-Subnet3;
Private-Subnet1-->EKS{{"EKS #9829;"}}
Private-Subnet2-->EKS{{"EKS #9829;"}}
Private-Subnet3-->EKS{{"EKS #9829;"}}
EKS==>ManagedNodeGroup;
ManagedNodeGroup-->|enable_crossplane=true|id2([Crossplane]);
subgraph Kubernetes Add-ons
id2([Crossplane])-.->|enable_aws_provider=false|id3([AWS-Provider]);
id2([Crossplane])-.->|enable_upjet_aws_provider=true|id4([Upbound-AWS-Provider]);
id2([Crossplane])-.->|enable_kubernetes_provider=true|id5([Kubernetes-Provider]);
id2([Crossplane])-.->|enable_helm_provider=true|id6([Helm-Provider]);
end
end
```

## How to Deploy

### Prerequisites:
Ensure that you have installed the following tools on your laptop before starting to work with this module and run Terraform Plan and Apply:

1. [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)
2. [Kubectl](https://kubernetes.io/docs/tasks/tools/)
3. [Terraform >= v1.0.0](https://learn.hashicorp.com/tutorials/terraform/install-cli)
4. [jq](https://stedolan.github.io/jq/download/) - Command-line JSON processor
5. [crane](https://github.com/google/go-containerregistry/blob/main/cmd/crane/README.md) - Tool for interacting with container registries

These tools are necessary to execute the scripts and manage your crossplane images efficiently. Make sure you follow the installation instructions provided in the links for each tool.


### Troubleshooting
1. If `terraform apply` errors out after creating the cluster when trying to apply the helm charts, try running the command:
```shell
aws eks --region <enter-your-region> update-kubeconfig --name <cluster-name> --alias <cluster-name>
```
and executing terraform apply again.

1. Make sure you have upgraded to the latest version of AWS CLI. Make sure your AWS credentials are properly configured as well.

### Deployment Steps
candonov marked this conversation as resolved.
Show resolved Hide resolved
#### Step 0: Clone the repo using the command below

```shell script
git clone https://github.com/awslabs/crossplane-on-eks.git
```

> [!IMPORTANT]
> The examples in this repository make use of one of the Crossplane AWS providers.
For that reason `upbound_aws_provider.enable` is set to `true` and `aws_provider.enable` is set to `false`. If you use the examples for `aws_provider`, adjust the terraform [main.tf](https://github.com/awslabs/crossplane-on-eks/blob/main/bootstrap/terraform/main.tf) in order install only the necessary CRDs to the Kubernetes cluster.

#### Step 1: ECR settings

> [!IMPORTANT]
> You need to use the same org/repository for the Crossplane images. Crossplane only allows packages in the same OCI registry and org to be part of the same family. This prevents a malicious provider from declaring itself part of a family and thus being granted RBAC access to its types. [Source Code Reference.](https://github.com/crossplane/crossplane/blob/master/internal/controller/rbac/provider/roles/reconciler.go#L300-L307)

Note: You can change the default `us-east-1` region in the following scripts before executing them.

To Create Crossplane private ECR repos, run the following script:

```
./scripts/create-crossplane-ecr-repos.sh
```

> [!IMPORTANT]
> There is currently a bug when using `docker pull`, `docker tag`, and `docker push` where the annotated layer information may be dropped. To avoid this issue, we need to use `crane` instead. For more details, you can refer to this issue: [crossplane/crossplane#5785](https://github.com/crossplane/crossplane/issues/5785).

To copy Crossplane images to a private ECR, run the following script:

```shell script
./scripts/copy-images-to-ecr.sh
```

#### Step 2: Run Terraform INIT
Initialize a working directory with configuration files

```shell script
cd bootstrap/terraform-fully-private/
terraform init
```

#### Step 3: Run Terraform PLAN
If your ECR repo is in different account or region than where the Terraform is pointing to, you can adjust the variables.tf file:

```
variable "ecr_account_id" {
type = string
description = "ECR repository AWS Account ID"
default = "" #defaults to var.region
}

variable "ecr_region" {
type = string
description = "ECR repository AWS Region"
default = "" #defaults to current account
}
```

Run Terraform plan:
```shell script
export TF_VAR_region=<ENTER YOUR REGION> # if ommited, defaults to var.region
terraform plan
```

#### Step 4: Finally, Terraform APPLY
To create resources:

```shell script
terraform apply -var='docker_secret={"username":"your-docker-username", "accessToken":"your-docker-password"}' --auto-approve
```

### Configure `kubectl` and test cluster
EKS Cluster details can be extracted from terraform output or from AWS Console to get the name of cluster.
This following command used to update the `kubeconfig` in your local machine where you run kubectl commands to interact with your EKS Cluster.

#### Step 5: Run `update-kubeconfig` command

`~/.kube/config` file gets updated with cluster details and certificate from the below command
```shell script
aws eks --region <enter-your-region> update-kubeconfig --name <cluster-name> --alias <cluster-name>
```
#### Step 6: List all the worker nodes by running the command below
```shell script
kubectl get nodes
```
#### Step 7: Verify the pods running in `crossplane-system` namespace
```shell script
kubectl get pods -n crossplane-system
```
#### Step 8: Verify the names provider and provider configs
Run the following command to get the list of providers:
```shell script
kubectl get providers
```
The expected output looks like this:
```
NAME INSTALLED HEALTHY PACKAGE AGE
aws-provider True True xpkg.upbound.io/crossplane-contrib/provider-aws:v0.36.0 36m
kubernetes-provider True True xpkg.upbound.io/crossplane-contrib/provider-kubernetes:v0.6.0 36m
provider-helm True True xpkg.upbound.io/crossplane-contrib/provider-helm:v0.13.0 36m
upbound-aws-provider True True xpkg.upbound.io/upbound/provider-aws:v0.27.0 36m
```
Run the following commands to get the list of provider configs:
```shell script
kubectl get provider
```
The expected output looks like this:
```
NAME AGE
providerconfig.aws.crossplane.io/aws-provider-config 36m

NAME AGE
providerconfig.helm.crossplane.io/default 36m

NAME AGE
providerconfig.kubernetes.crossplane.io/kubernetes-provider-config 36m
```

#### Step 9: Access the ArgoCD UI
Get the load balancer url:
```
kubectl -n argocd get service argo-cd-argocd-server -o jsonpath="{.status.loadBalancer.ingress[*].hostname}{'\n'}"
```
Copy and paste the result in your browser.<br>
The initial username is `admin`. The password is autogenerated and you can get it by running the following command:
```
echo "$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d)"
```

## Clean up
1. Delete resources created by Crossplane such as first Claims, then XRDs and Compositions.

2. See the faq.md for more details on cleaning up the resources created.

3. Delete the EKS cluster resources and ECR repositories with the following script:

```bash
./scripts/cleanup.sh
```
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
apiVersion: apiextensions.crossplane.io/v1alpha1
kind: EnvironmentConfig
metadata:
name: cluster
data:
awsAccountID: "${awsAccountID}"
eksOIDC: ${eksOIDC}
vpcID: ${vpcID}
53 changes: 53 additions & 0 deletions bootstrap/terraform-fully-private/ecr.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
locals {
ecr_account_id = var.ecr_account_id != "" ? var.ecr_account_id : data.aws_caller_identity.current.account_id
ecr_region = var.ecr_region != "" ? var.ecr_region : local.region
}

module "secrets-manager" {
source = "terraform-aws-modules/secrets-manager/aws"
version = "1.1.2"

name = "ecr-pullthroughcache/docker"
secret_string = jsonencode(var.docker_secret)
}

module "ecr" {
source = "terraform-aws-modules/ecr/aws"
version = "2.2.1"

create_repository = false

registry_pull_through_cache_rules = {
ecr = {
ecr_repository_prefix = "ecr"
upstream_registry_url = "public.ecr.aws"
}
k8s = {
ecr_repository_prefix = "k8s"
upstream_registry_url = "registry.k8s.io"
}
quay = {
ecr_repository_prefix = "quay"
upstream_registry_url = "quay.io"
}
dockerhub = {
ecr_repository_prefix = "docker-hub"
upstream_registry_url = "registry-1.docker.io"
credential_arn = module.secrets-manager.secret_arn
}
}

manage_registry_scanning_configuration = true
registry_scan_type = "BASIC"
registry_scan_rules = [
{
scan_frequency = "SCAN_ON_PUSH"
filter = [
{
filter = "*"
filter_type = "WILDCARD"
},
]
}
]
}
Loading
Loading