diff --git a/canary-checker/docs/concepts/metrics-exporter.md b/canary-checker/docs/concepts/metrics-exporter.md index 2bfc808d..e3878be2 100644 --- a/canary-checker/docs/concepts/metrics-exporter.md +++ b/canary-checker/docs/concepts/metrics-exporter.md @@ -2,8 +2,6 @@ title: Metrics Exporter --- - - Canary checker can export custom metrics from any check type, replacing and/or consolidating multiple standaline Prometheus Exporters into a single exporter. In the example below, exchange rates against the USD are exported by first calling an HTTP api and then using the values from the JSON response to create the metrics: @@ -59,20 +57,122 @@ exchange_rate{from=USD, to=ILS} 3.849 exchange_rate_api 260.000 ``` -| Field | Description | Scheme | Required | -| --------- | ------------------------------------ | ------------------- | -------- | +## Stateful Metrics + +Metrics can be generated from time based data, e.g. logs per minute, logins per second by using the output of one check execution as the input to the next. + +```yaml +apiVersion: canaries.flanksource.com/v1 +kind: Canary +metadata: + name: "container-log-counts" + namespace: observability + # The schedule can be as short or as long as you want, the query will always search for log + # since the last query + schedule: "@every 5m" + http: + - name: container_log_volume + url: "https://elasticsearch/logstash-*/_search" + headers: + - name: Content-Type + value: application/json + templateBody: true + test: + # if no logs are found, fail the health check + expr: json.?aggregations.logs.doc_count.orValue(0) > 0 + # query for log counts by namespace, container and pod that have been created since the last check + body: >- + { + "size": 0, + "aggs": { + "logs": { + "filter": { + "range": { + "@timestamp" : { + {{- if last_result.results.max }} + "gte": "{{ last_result.results.max }}" + {{- else }} + "gte": "now-5m" + {{- end }} + } + } + }, + "aggs": { + "age": { + "max": { + "field": "@timestamp" + } + }, + "labels": { + "multi_terms": { + "terms": [ + { "field": "kubernetes_namespace_name.keyword"}, + { "field": "kubernetes_container_name.keyword"}, + { "field": "kubernetes_pod_name.keyword"} + ], + "size": 1000 + } + } + } + } + } + } + transform: + # Save the maximum age for usage in subsequent queries and create a metric for each pair + expr: | + json.orValue(null) != null ? + [{ + 'detail': { 'max': string(json.?aggregations.logs.age.value_as_string.orValue(last_result().?results.max.orValue(time.Now()))) }, + 'metrics': json.?aggregations.logs.labels.buckets.orValue([]).map(k, { + 'name': "namespace_log_count", + 'type': "counter", + 'value': double(k.doc_count), + 'labels': { + "namespace": k.key[0], + "container": k.key[1], + "pod": k.key[2] + } + }) + }].toJSON() + : '{}' +``` + +This snippet will retrieve the `last_result.results.max` value from the last execution ensuring data is not duplicated or missed +```go + "@timestamp" : { + {{- if last_result.results.max }} + "gte": "{{ last_result.results.max }}" + {{- else }} + "gte": "now-5m" + {{- end }} + } + +``` + +The max value is saved in the `transform` section using: + +```yaml +#... + 'detail': { 'max': string(json.?aggregations.logs.age.value_as_string.orValue(last_result().?results.max.orValue(time.Now()))) }, +#... +``` + +## Result Variables + +| Field | Description | Scheme | Required | +| --------- | ------------------------------------ | ----------------- | -------- | | `metrics` | Metrics to export from check results | [Metric](#metric) | | ### Metric -| Field | Description | Scheme | Required | -| -------- | ------------------------------------------------------ | ---------- | -------- | -| `name` | Name of the metric | `string` | Yes | -| `value` | An expression to derive the metric value from | `Expression` | Yes | -| `type` | Prometheus Metric Type (counter, gauge or histogram) | `string` | Yes | +| Field | Description | Scheme | Required | +| -------- | ------------------------------------------------------ | ----------------- | -------- | +| `name` | Name of the metric | `string` | Yes | +| `value` | An expression to derive the metric value from | `Expression` | Yes | +| `type` | Prometheus Metric Type (counter, gauge or histogram) | `string` | Yes | | `labels` | Labels for prometheus metric (values can be templated) | [[]Label](#label) | | -#### Label +### Label | Field | Description | Scheme | Required | | ----------- | ------------------------------------------------------ | ------------------- | -------- | @@ -83,17 +183,18 @@ exchange_rate_api 260.000 Expresions can make use of the following variables: -#### **Expression Variables** - -| Fields | Description | Scheme | -| ------------------- | ------------------------------------------ | ----------------------------------------- | -| **`result`** | Check Result | See result variables section of the check | -| `check.name` | Check name | `string` | -| `check.description` | Check description | `string` | -| `check.labels` | Dynamic labels attached to the check | `map[string]string` | -| `check.endpoint` | Endpoint (usally a URL) | `string` | -| `check.duration` | Duration in milliseconds | `int64` | -| `canary.name` | Canary name | `string` | -| `canary.namespace` | Canary namespace | `string` | -| `canary.labels` | Labels attached to the canary CRD (if any) | `map[string]string` | +### **Expression Variables** + +| Fields | Description | Scheme | +| ------------------------- | ------------------------------------------ | ----------------------------------------- | +| **`result`** | Check Result | See result variables section of the check | +| **`last_result.results`** | The last result | | +| `check.name` | Check name | `string` | +| `check.description` | Check description | `string` | +| `check.labels` | Dynamic labels attached to the check | `map[string]string` | +| `check.endpoint` | Endpoint (usally a URL) | `string` | +| `check.duration` | Duration in milliseconds | `int64` | +| `canary.name` | Canary name | `string` | +| `canary.namespace` | Canary namespace | `string` | +| `canary.labels` | Labels attached to the canary CRD (if any) | `map[string]string` | diff --git a/canary-checker/docs/concepts/authentication.md b/canary-checker/docs/concepts/secret-management.md similarity index 50% rename from canary-checker/docs/concepts/authentication.md rename to canary-checker/docs/concepts/secret-management.md index ab71e90e..440c0b67 100644 --- a/canary-checker/docs/concepts/authentication.md +++ b/canary-checker/docs/concepts/secret-management.md @@ -1,16 +1,16 @@ -# Authentication +# Secret Management Canary checker uses the Kubernetes ValuesFrom pattern to retrieve sensitive values like usernames, password and access keys. -Whenever a field uses the `EnvVar` object type you have the option of specifying the value in 3 ways: +Whenever a field uses the `EnvVar` object type you have the option of specifying the value in multiple ways. -## EnvVar +1. Statically in the `value` +1. Via a Kubernetes Config Map via the `configMapKeyRef` +1. Via a Kubernetes Secret via the `secretKeyRef` +1. From a value of a deployed helm chart using `helmRef` +1. From a service account using `serviceAccount` -1. Statically in the `value` field -1. Via a Kubernetes Config Map via the `configMapKeyRef` field -1. Via a Kubernetes Secret via the `secretKeyRef` field - -### Static Values +## Static Values Using a HTTP health check as an example for static values: @@ -30,7 +30,7 @@ spec: value: world ``` -### Kubernetes Configmaps +## Configmaps To use a configmap, we first need to create the configmap: @@ -60,7 +60,7 @@ spec: key: pass ``` -### Kubernetes Secrets +## Secrets To use a secret, first we create the secret: @@ -75,22 +75,111 @@ metadata: name: http-basic-auth-configmap spec: http: - - url: https://httpbin.org/basic-auth/hello/world - responseCodes: [200] - authentication: - username: + - url: https://httpbin.demo.aws.flanksource.com/basic-auth/hello/world + username: + valueFrom: + secretKeyRef: + name: basic-auth + key: user + password: + valueFrom: + secretKeyRef: + name: basic-auth + key: pass +``` + +## Helm Values + +To use a secret, first we deploy a helm chart + +```bash +helm install podinfo podinfo/podinfo -n podinfo --set ingress.enabled=true +``` + +```yaml title="http-from-helm.yaml" +apiVersion: canaries.flanksource.com/v1 +kind: Canary +metadata: + name: http-from-helm +spec: + http: + - env: + - name: url valueFrom: - secretKeyRef: - name: basic-auth - key: user - password: + helmRef: + name: podinfo + key: .ingress.hosts[0].host + + url: $(url) +``` + +## Service Accounts + +Checks can use service accounts for authentication with external services that have existing trust established + +```yaml title="http-service-accounts.yaml" +apiVersion: canaries.flanksource.com/v1 +kind: Canary +metadata: + name: http-basic-auth-configmap +spec: + http: + + interval: 30 + http: + - name: vault-example-sre + description: "HashiCorp Vault functionality check." + url: https://vault.example/v1/auth/kubernetes/login + env: + - name: TOKEN valueFrom: - secretKeyRef: - name: basic-auth - key: pass + serviceAccount: default-account + templateBody: true + body: | + { + "jwt": "$(TOKEN)", + "role": "example-role" + } + +``` + +:::note + +For service account token issuing the canary-checker service account `canary-checker-sa` will need to be granted permissions to issue tokens using: + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: canary-checker-sa-issuing-rolebinding +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: canary-checker-sa-issuing +subjects: + - kind: ServiceAccount + name: canary-checker-sa + namespace: canary-checker +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + creationTimestamp: null + name: canary-checker-sa-issuing +rules: + +- apiGroups: [""] + resources: + - "serviceaccounts/token" + - "serviceaccounts" + verbs: + - "create" + - "get ``` -### Recommendations +::: + +# Recommendations Kubernetes Secrets are, by default, stored unencrypted in the API server's underlying data store (etcd). Anyone with API access can retrieve or modify a Secret, and so can anyone with access to etcd. With this in mind, it is recommended to implement some level of security to prevent unauthorized access to your Kubernetes secrets. You may consider the following for your encryption and security needs: diff --git a/canary-checker/docs/helm.md b/canary-checker/docs/helm.md index 14485a76..cf485884 100644 --- a/canary-checker/docs/helm.md +++ b/canary-checker/docs/helm.md @@ -7,14 +7,14 @@ image: /static/img/icons/helm.svg The recommended method for installing Canary Checker is using [helm](https://helm.sh/) -### 1. Add the Flanksource helm repository +## 1. Add the Flanksource helm repository ```bash helm repo add flanksource https://flanksource.github.io/charts helm repo update ``` -### 2. Deploy Canary Checker using Helm +## 2. Deploy Canary Checker using Helm To install into a new `canary-checker` namespace, run @@ -83,11 +83,18 @@ kubectl port-forward svc/canary-checker-ui 8080:80 [http://localhost:8080](http://localhost:8080) -The canary checker itself only presents an API. To view the data graphically, the Flanksource UI is required, and is installed by default. The UI should be configured to allow external access to via ingress -| ---------------------------------- | ------------------------------------------------------------ | -| flanksource-ui.ingress.host | URL at which the UI will be accessed | -| flanksource-ui.ingress.annotations | Map of annotations required by the ingress controller or certificate issuer | -| flanksource-ui.ingress.tls | Map of configuration options for TLS | +To deploy an ingress for the dashboard, update the `values.yaml`: -More details regarding ingress configuration can be found in the [kubernetes documentation](https://kubernetes.io/docs/concepts/services-networking/ingress/) +```yaml +flanksource-ui: + enabled: true + ingress: + annotations: + kubernetes.io/tls-acme: "true" + host: + tls: + - hosts: + - + secretName: ingress-tls +``` diff --git a/canary-checker/docusaurus.config.js b/canary-checker/docusaurus.config.js index 0b1ced55..10ff2805 100644 --- a/canary-checker/docusaurus.config.js +++ b/canary-checker/docusaurus.config.js @@ -15,7 +15,8 @@ const config = { customFields: { oss: process.env.CANARY_CHECKER_OSS === 'true', links: { - "authentication": '/concepts/authentication' + "authentication": '/concepts/secret-management', + "connection": '/concepts/authentication' } }, markdown: { diff --git a/canary-checker/sidebars.js b/canary-checker/sidebars.js index 26af2fd3..98dd618a 100644 --- a/canary-checker/sidebars.js +++ b/canary-checker/sidebars.js @@ -49,10 +49,12 @@ module.exports = { id: 'concepts/metrics-exporter', label: 'Metrics Exporter', }, + + { type: 'doc', - id: 'concepts/grafana', - label: 'Grafana', + id: 'concepts/secret-management', + label: 'Secret Management', }, { @@ -83,11 +85,13 @@ module.exports = { ] }, + { type: 'doc', - id: 'concepts/authentication', - label: 'Authentication', + id: 'concepts/grafana', + label: 'Grafana', }, + { type: 'doc', id: 'concepts/troubleshooting', @@ -268,31 +272,13 @@ module.exports = { id: 'reference/exec', label: 'Exec', }, - { - type: 'doc', - id: 'reference/jmeter', - label: 'JMeter', - }, + { type: 'doc', id: 'reference/junit', label: 'JUnit', }, - { - type: 'doc', - id: 'reference/k6', - label: 'K6', - }, - { - type: 'doc', - id: 'reference/newman', - label: 'Newman / Postman', - }, - { - type: 'doc', - id: 'reference/playwright', - label: 'Playwright', - }, + { type: 'doc', id: 'reference/azure-devops', @@ -332,6 +318,32 @@ module.exports = { }, ], }, + { + type: 'category', + label: "Examples", + items: [ + { + type: 'doc', + id: 'reference/k6', + label: 'K6', + }, + { + type: 'doc', + id: 'reference/newman', + label: 'Newman / Postman', + }, + { + type: 'doc', + id: 'reference/playwright', + label: 'Playwright', + }, + { + type: 'doc', + id: 'reference/jmeter', + label: 'JMeter', + }, + ] + }, { type: 'category', label: 'Scripting', diff --git a/mission-control/Makefile b/mission-control/Makefile index 578c6702..266f268a 100644 --- a/mission-control/Makefile +++ b/mission-control/Makefile @@ -7,5 +7,5 @@ sync: cp -r docs/canary-checker/scripting docs/reference rm -rf static/img/icons cp -r ../canary-checker/static/img/icons static/img - cp docs/canary-checker/concepts/authentication.md docs/reference/ + cp docs/canary-checker/concepts/secret-management.md docs/reference/ cp docs/canary-checker/concepts/connections.md docs/reference/ diff --git a/mission-control/docs/install.md b/mission-control/docs/install.md index f88467e8..ccd88ef2 100644 --- a/mission-control/docs/install.md +++ b/mission-control/docs/install.md @@ -2,13 +2,13 @@ How to Install Mission control with helm -### Prerequisite +## Prerequisites -To properly install and run the Mission Control chart on your Kubernetes Cluster, you need to have the following prerequisites; +To install and run the Mission Control chart on your Kubernetes Cluster, you need to have the following prerequisites; -- A Kubernetes installation of version 1.21 or higher. +- A Kubernetes installation of version 1.26 or higher. -### Install Chart +## Install Chart ```console helm install [RELEASE_NAME] flanksource/mission-control @@ -52,7 +52,7 @@ _See [configuration](#configuration) below._ _See [helm install](https://helm.sh/docs/helm/helm_install/) for command documentation._ -### Uninstall Chart +## Uninstall Chart ```console helm uninstall [RELEASE_NAME] @@ -62,7 +62,7 @@ This removes all the Kubernetes components associated with the chart and deletes _See [helm uninstall](https://helm.sh/docs/helm/helm_uninstall/) for command documentation._ -### Upgrading Chart +## Upgrading Chart ```console helm upgrade [RELEASE_NAME] [CHART] --install @@ -70,7 +70,7 @@ helm upgrade [RELEASE_NAME] [CHART] --install _See [helm upgrade](https://helm.sh/docs/helm/helm_upgrade/) for command documentation._ -### Configuration +## Configuration See [Customizing the Chart Before Installing](https://helm.sh/docs/intro/using_helm/#customizing-the-chart-before-installing). To see all configurable options with detailed comments, visit the chart's [values.yaml](https://github.com/flanksource/config-db/blob/main/chart/values.yaml), or run these configuration commands: diff --git a/mission-control/docusaurus.config.js b/mission-control/docusaurus.config.js index 963576ce..190df3e5 100644 --- a/mission-control/docusaurus.config.js +++ b/mission-control/docusaurus.config.js @@ -22,7 +22,7 @@ const config = { customFields: { links: { - "authentication": '/reference/authentication', + "authentication": '/reference/secret-management', "connection": '/reference/authentication' }