[IBCDPE-1095] Secure cluster ingress for telemetry data #40

BryanFauble · 2024-10-22T17:10:19Z

Problem:

Previously in order to ingest data into the otel collector running in the EKS cluster you either needed to be running in the cluster, or, port-forward to the service running the otel collectors.

Solution:

Adopting the usage of envoy gateway (https://gateway.envoyproxy.io/docs/concepts/concepts_overview/) to create an ingress for the otel collector.
Using Auth0 for JWT verification and requirement to submit data to the endpoint.
Adopting the usage of the Auth0 terraform provider to handle automatic configuration of Auth0 resources: https://registry.terraform.io/providers/auth0/auth0/latest/docs

Testing:

I verified that by setting a few environment variables in SchematicAPI that I could export telemetry data with an auth header, over HTTPS/TLS 1.3:
I also verified that sending data without a JWT returns an expected HTTP 401 error
I verified that sending data with a valid JWT, but an invalid audience returns an expected HTTP 403 error.

TODO:

Determine if we can use the Auth0 terraform provider to handle setting up Auth0 for us: https://registry.terraform.io/providers/auth0/auth0/latest/docs
Set up urlrewrite so we can place these endpoints specifically behind /telemetry/v1/...

… are not shared with control plane

BryanFauble · 2024-10-23T19:41:44Z

deployments/stacks/dpe-auth0/main.tf

+# Used to create the Auth0 resources for the DPE stack
+resource "auth0_resource_server" "k8s-cluster-telemetry" {
+  name        = "${var.cluster_name}-telemetry"
+  identifier  = "${var.cluster_name}-telemetry"
+  signing_alg = "RS256"
+
+  allow_offline_access                            = false
+  token_lifetime                                  = 86400
+  skip_consent_for_verifiable_first_party_clients = true
+  # https://registry.terraform.io/providers/auth0/auth0/latest/docs/resources/resource_server_scopes
+  # Says to use the following, however it errors out:
+  # This object has no argument, nested block, or exported attribute named "scopes".
+  # lifecycle {
+  #   ignore_changes = [scopes]
+  # }
+}
+
+resource "auth0_client" "oauth2_clients" {
+  for_each = { for client in var.auth0_clients : client.name => client }
+
+  name        = each.value.name
+  description = each.value.description
+  app_type    = each.value.app_type
+
+  jwt_configuration {
+    alg = "RS256"
+  }
+}
+
+resource "auth0_client_credentials" "client_secrets" {
+  for_each = { for client in auth0_client.oauth2_clients : client.name => client }
+
+  client_id             = auth0_client.oauth2_clients[each.key].id
+  authentication_method = "client_secret_post"
+}
+
+resource "auth0_client_grant" "access_to_k8s_cluster" {
+  for_each = { for client in var.auth0_clients : client.name => client }
+
+  client_id = auth0_client.oauth2_clients[each.key].id
+  audience  = auth0_resource_server.k8s-cluster-telemetry.identifier
+  scopes    = []
+}


This creates a few items in Auth0.

an API that each client is granted access to:

The applications where each has a unique client ID and client secret. These will be added to the machines where data is going to be exported out of in order to implement the client credential exchange:

BryanFauble · 2024-10-23T19:42:24Z

deployments/stacks/dpe-k8s-deployments/main.tf

+  cluster_issuer_name  = "selfsigned"
+  # To determine more elegant ways to fill in these values, for example, if we have
+  # a pre-defined DNS name for the cluster (https://sagebionetworks.jira.com/browse/IT-3931)
+  ssl_hostname   = var.ssl_hostname
+  auth0_jwks_uri = var.auth0_jwks_uri


Depends on https://sagebionetworks.jira.com/browse/IT-3931 to move away from selfsigned certs

BryanFauble · 2024-10-28T19:43:52Z

deployments/main.tf

-  vpc_cidr_block                         = "10.52.16.0/20"
-  public_subnet_cidrs                    = ["10.52.16.0/24", "10.52.17.0/24"]
+  vpc_cidr_block = "10.52.16.0/20"
+  # A public subnet is required for each AZ in which the worker nodes are deployed
+  public_subnet_cidrs                    = ["10.52.16.0/24", "10.52.17.0/24", "10.52.19.0/24"]
  private_subnet_cidrs_eks_control_plane = ["10.52.18.0/28", "10.52.18.16/28"]
  azs_eks_control_plane                  = ["us-east-1a", "us-east-1b"]

-  private_subnet_cidrs_eks_worker_nodes = ["10.52.20.0/22", "10.52.24.0/22", "10.52.28.0/22"]
-  azs_eks_worker_nodes                  = ["us-east-1a", "us-east-1b", "us-east-1c"]
+  private_subnet_cidrs_eks_worker_nodes = ["10.52.28.0/22", "10.52.24.0/22", "10.52.20.0/22"]
+  azs_eks_worker_nodes                  = ["us-east-1c", "us-east-1b", "us-east-1a"]
+
+  enable_cluster_ingress = true
+  enable_otel_ingress    = true
+  ssl_hostname           = "a09a38cc5a8d6497ea69c6bf6318701b-1974793757.us-east-1.elb.amazonaws.com"
+  auth0_jwks_uri         = "https://dev-57n3awu5je6q653y.us.auth0.com/.well-known/jwks.json"
 }


The juggling of this is an unfortunate side affect of how the VPC module creates subnets in specific AZs. By doing this it allows us to:

Create 3 public subnets in each AZ

Create 2 private subnets in a, and b for the eks control plane

Create 3 private subnets in each AZ for the private worker nodes.

Why this is needed:

In order for the AWS load balancer to forward traffic to a private subnet, there needs to be a public subnet in that AZ.

Worker nodes can be deployed to all 3 AZs, and because of this when a pod was scheduled into the c subnet, the load balancer could not forward traffic to that instance.

BryanFauble · 2024-10-30T17:18:23Z

deployments/main.tf

  aws_account_id = "631692904429"
  region         = "us-east-1"

  cluster_name = "dpe-k8-sandbox"
  vpc_name     = "dpe-sandbox"

-  vpc_cidr_block                         = "10.52.16.0/20"


My comment keeps getting deleted, here is a link explaining why this change was made: #40 (comment)

* Upgrade airflow to latest helm chart and airflow:2.9.3

* Set up SMTP settings with test email & Move to lets encrypt

BryanFauble · 2024-11-05T20:38:29Z

Merging this PR to collapse this mess of PR dependencies

BryanFauble added 25 commits October 17, 2024 15:14

Shrink VPC size and create subnets specifically for worker nodes that…

b141b73

… are not shared with control plane

Add back var

948004d

Correct cidr block

5449ea5

Update cidr blocks

750d79f

Correct node lengths

dd796a1

Correct array slicing

6f7ab29

Correct indexing

9e1c36f

Update default eks cluster version

ac3303e

Shrink EKS control plane subnet range

55344c3

Set range back

a38e7bc

Enable otel collector ingress with Auth0

877e506

Set ssl hostname

8da96b6

Try a reference grant with http route

da9b28a

Try without url re-write

4a365ad

Try exact url matching

8da3838

Correct service name in reference grant

41488c7

Set up merged gateway deployment

370a4b7

Correct vars

c2a4bf8

Remove namespaced deployment of http-route

7d32e8f

Move ingress to signoz specific deployment

5f77316

Reference paramref by ns

b4b9814

Point to updated hostname

ef64277

Test deploying a temp merged gateway

6807d1c

Try with url rewrite for /telemetry

cc3006d

Go back to referenceGrant approach

f83e9c7

spacelift-int-sagebionetworks bot temporarily deployed to spacelift/dpe-dev-kubernetes-deployments October 22, 2024 17:32 Inactive

BryanFauble added 2 commits October 22, 2024 10:35

Leave filter off for now

8c0c3a4

Deploy public subnet to all AZs

42689d4

spacelift-int-sagebionetworks bot had a problem deploying to spacelift/dpe-dev-kubernetes-deployments October 22, 2024 17:52 Failure

juggle cidrs

2d90747

BryanFauble commented Oct 23, 2024

View reviewed changes

BryanFauble requested a review from zaro0508 October 23, 2024 19:42

BryanFauble marked this pull request as ready for review October 23, 2024 19:44

BryanFauble requested a review from a team as a code owner October 23, 2024 19:44

Update policy name

a083e0f

Base automatically changed from ibcdpe-1097-shrink-vpc-subnets to signoz-testing October 25, 2024 16:26

BryanFauble added 2 commits October 25, 2024 10:37

Correct node lengths

7133f17

Correct indexing

3011207

spacelift-int-sagebionetworks bot temporarily deployed to spacelift/root-spacelift-administrative-stack October 25, 2024 17:41 Inactive

spacelift-int-sagebionetworks bot had a problem deploying to spacelift/dpe-dev-kubernetes-deployments October 25, 2024 17:57 Failure

Merge branch 'signoz-testing' into ibcdpe-1095-cluster-ingress-signoz

ff180d6

BryanFauble commented Oct 28, 2024

View reviewed changes

spacelift-int-sagebionetworks bot temporarily deployed to spacelift/dpe-dev-auth0 October 29, 2024 18:16 Inactive

Point to updated auth0 domain

64035b4

spacelift-int-sagebionetworks bot temporarily deployed to spacelift/dpe-dev-auth0 October 29, 2024 18:21 Inactive

Updated auth0 jwks uri

547bc9e

spacelift-int-sagebionetworks bot temporarily deployed to spacelift/root-spacelift-administrative-stack October 29, 2024 18:25 Inactive

spacelift-int-sagebionetworks bot had a problem deploying to spacelift/dpe-dev-kubernetes-infrastructure October 29, 2024 18:29 Failure

BryanFauble added 2 commits October 29, 2024 11:32

Add notes to readme

0cfe041

Remove auth0 from versions

3e31d87

spacelift-int-sagebionetworks bot had a problem deploying to spacelift/dpe-dev-kubernetes-deployments October 29, 2024 18:44 Failure

BryanFauble commented Oct 30, 2024

View reviewed changes

BryanFauble added 4 commits November 5, 2024 10:23

Move token lifecycle to 1.25 days from 1 day

957109e

[IBCDPE-1111] upgrade airflow (#42)

dfc12c0

* Upgrade airflow to latest helm chart and airflow:2.9.3

Point to updated config value for airflow regex deserialization (#43)

4161baa

Signoz alert manager setup for SMTP (#41)

903bc50

* Set up SMTP settings with test email & Move to lets encrypt

BryanFauble merged commit 5c5654f into signoz-testing Nov 5, 2024
2 of 10 checks passed

BryanFauble deleted the ibcdpe-1095-cluster-ingress-signoz branch November 5, 2024 20:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[IBCDPE-1095] Secure cluster ingress for telemetry data #40

[IBCDPE-1095] Secure cluster ingress for telemetry data #40

BryanFauble commented Oct 22, 2024 •

edited

Loading

BryanFauble Oct 23, 2024

BryanFauble Oct 23, 2024

BryanFauble Oct 28, 2024

BryanFauble Oct 30, 2024

BryanFauble commented Nov 5, 2024

[IBCDPE-1095] Secure cluster ingress for telemetry data #40

[IBCDPE-1095] Secure cluster ingress for telemetry data #40

Conversation

BryanFauble commented Oct 22, 2024 • edited Loading

BryanFauble Oct 23, 2024

Choose a reason for hiding this comment

BryanFauble Oct 23, 2024

Choose a reason for hiding this comment

BryanFauble Oct 28, 2024

Choose a reason for hiding this comment

BryanFauble Oct 30, 2024

Choose a reason for hiding this comment

BryanFauble commented Nov 5, 2024

BryanFauble commented Oct 22, 2024 •

edited

Loading