Skip to content

Definition of eLife's k8s cluster and deployments to it. Automatically applied via Flux.

Notifications You must be signed in to change notification settings

elifesciences/elife-flux-cluster

Repository files navigation

eLife k8s/Flux Production Cluster

EKS cluster name: kubernetes-aws--flux-prod

Use this git repo to control the cluster state (no kubectl or helm cli action needed/wanted).

  • Flux will try to apply any yaml file in this repo to the cluster
  • HelmController allows use of helm charts
  • We currently have three Kustomizations defined: crds, system and deployments (each pointed at the root directory named the same). Only Yaml files found in these folders are loaded, in a dependency order (see "Kustomizations" below)

Cluster infrastructure is defined in builder in the kubernetes-aws section.

Users can configure kubectl for this cluster with:

    aws eks update-kubeconfig --name kubernetes-aws--flux-prod

Dashboards

The #cluster-alerts slack channel receives alerts from:

  • Alertmanager
  • Healthchecks.io (monitors Alertmanager heartbeat)

How to make a change

  • Ensure that you have run mise install to get up-to-date dependencies.
  • Ensure that you have run make validate and it completed correctly.
  • Follow conventional commits for commit messages.
  • Add a suffix to the commit message containing the issue e.g. elifesciences/issues#1234
  • Look at the GitOps dashboard to see the change being applied.

Adding/Editing Deployments

Kustomizations

Cluster level Kustomizations

  • crds: Cluster managed CustomResourceDefinitions.

  • system: Cluster services that are not directly serving production usecases. Some infrastructure components needs CRDs to exist before upgrading, so infrastructure kustomization depends on crds kustomization

  • deployments: These are the production services. As these all depend on infrastructure to serve traffic correctly, system kustomization is a dependency of this kustomization

  • flux tries to apply any .yaml file in the kustomization directories above

  • within that root folder, the structure is only used for humans

  • namespaces are managed using .yaml files

  • flux will always apply the HEAD of master

Each namespace is organised around an application, or an environment for an application, favouring the latter.

Individual Kustomizations

There are a growing number of kustomizations for apps or system that abstract complexity. We can then deploy them with a flux Kustomization object from one of the cluster kustomizations above. These kustomizations are stored in kustomzations/.

Adding Helm Charts

Debugging

Services available on the Cluster

  • nginx-ingress (docs)
    • provides SSL termination
    • host entries ending in .elifesciences.org will be added to our zone by ExternalDNS
  • cert-manager with letsencrypt (docs/letsencrypt)
    • obtain letsencrypt SSL certs via ingress definitions
  • VictoriaMetricsOperator (docs/monitoring-alerting)
  • oauth2-proxy (docs/oauth-proxy)
    • limit access to elifesciences github org
  • SealedSecrets (docs/sealed-secrets.md)
    • encrypt secrets for safe storage in this repo
  • VictoriaLogs
    • Stores logs for services in cluster, is queriable from Grafana as a data source.
  • Percona Server for MongoDB operator
    • Used to run a MongoDB cluster, with support for automated backup, reconvery and upgrades.
    • Deployed in "cluster-wide" mode. Each namespace can deploy it's own cluster of pods from the central operator.

Administration

About

Definition of eLife's k8s cluster and deployments to it. Automatically applied via Flux.

Resources

Stars

Watchers

Forks