[IBCDPE-938] Deploy Signoz (OTEL Visualization) to kubernetes cluster

[BryanFauble]

Document describing the overall design of the solution and some included network diagrams at the bottom:
https://sagebionetworks.jira.com/wiki/spaces/DPE/pages/3351773274/IBCDPE-938+OpenTelemetry+Data+collection+Visualization

Problem:

1. We did not have a way to view telemetry data produced by services like schematic
2. We were deploying EC2 instances to a single AZ, public subnets were not deployed to all AZs.
3. We did not have a north-south gateway to handle traffic into the cluster (From internet into cluster)
4. We did not have AWS SES to send out emails
5. We did not have any TLS set up for secure messaging into the cluster
6. We did not have authentication mechanisms set up to prevent unauthenticated traffic from getting ingested into the cluster

Solution:

1. Deploy the signoz helm chart: https://github.com/SigNoz/charts/tree/main/charts/signoz - Which include the clickhouse operator for data storage
2. Availability zone: Updating AZ deployment to deploy worker nodes on 3 private subnets, create 3 public subnets, and deploy EKS control plane on 2 AZs
3. envoy-gateway is added to the cluster to handle north-south traffic: https://gateway.envoyproxy.io/docs/ - This also gives us an easy way to secure the endpoints with auth filters/expose future services to the internet.
4. cert-manager is added to the cluster to handle TLS/cert generation
5. AWS SES module created to configure SES settings to allow SigNoz alert manager component to send out emails.
6. Setting up Auth0 to allow clients to request an access token from Auth0 that can be verified at the gateway within the k8s cluster.

Testing:

1. I have tested the deployment of signoz to the sandbox cluster in the DNT dev account
2. I have tested the north-south traffic provided through envoy-gateway
3. I have tested with auth0 that JWT verification works through the envoy-gateway
4. I tested sending the data to a backend service in the k8s cluster to verify traffic makes it to the pod
5. I tested and provisioned a cert from let's encrypt with cert-manager for HTTPS/TLS
6. I tested that pods are started in on-demand instances when they have the nodeSelector set
7. I tested AWS SES and have successfully requested production level access for the SES account in our sandbox cluster

Follow up tickets:

• Set up Clickhouse Cold Storage + Backups (Does not block this review): https://sagebionetworks.jira.com/browse/IBCDPE-1094
• Create cluster ingress for otel data: https://sagebionetworks.jira.com/browse/IBCDPE-1095
• Depending on response from Create sub-domain for DPE Kubernetes cluster ingress: https://sagebionetworks.jira.com/browse/IT-3931 - We may have a subdomain to use for all DNS records and some additional work may be needed to pull TLS certs in a centralized manner as defined by IT. This may also influence some decisions on handling authentication between schematic and the k8s cluster.

[BryanFauble]

These are not new changes (Git is showing them in the diff for some reason). Please ignore changes here.

[BryanFauble]

These are not new changes (Git is showing them in the diff for some reason). Please ignore changes here.

[BryanFauble]

These are not new changes (Git is showing them in the diff for some reason). Please ignore changes here.