Merge pull request #151 from flanksource/updates

Updates

canary-checker/docs/concepts/artifacts.md

@@ -29,7 +29,7 @@ The following checks support archiving artifacts
 
 The only configuration required is to provide the path(s) of the artifacts generated by the check.
 
-### Ex1. Archiving `/tmp/results/` directory
+### Archiving `/tmp/results/` directory
 
 For the following script in an exec check
 

docs/incidents/concepts/responders.md

@@ -41,8 +41,8 @@ responder_clients:
 | Field      | Description   | Scheme                                                                        | Required |
 | ---------- | ------------- | ----------------------------------------------------------------------------- | -------- |
 | `url`      | Jira url      | `string`                                                                      | `true`   |
-| `username` | Jira username | [`types.EnvVar`](https://pkg.go.dev/github.com/flanksource/duty/types#EnvVar) | `true`   |
-| `password` | Jira password | [`types.EnvVar`](https://pkg.go.dev/github.com/flanksource/duty/types#EnvVar) | `true`   |
+| `username` | Jira username |  <CommonLink to="secrets">*EnvVar*</CommonLink> | `true`   |
+| `password` | Jira password |  <CommonLink to="secrets">*EnvVar*</CommonLink> | `true`   |
 | `defaults` | Jira defaults | `string`                                                                      | `option` |
 | `values`   | Jira values   | `string`                                                                      | `option` |
 
@@ -53,7 +53,7 @@ responder_clients:
 | `tenant_id` | MS Planner tenant id | `string`                                                                      | `true`   |
 | `client_id` | MS Planner client id | `string`                                                                      | `true`   |
 | `group_id`  | MS Planner group id  | `string`                                                                      | `true`   |
-| `username`  | MS Planner username  | [`types.EnvVar`](https://pkg.go.dev/github.com/flanksource/duty/types#EnvVar) | `true`   |
-| `password`  | MS Planner password  | [`types.EnvVar`](https://pkg.go.dev/github.com/flanksource/duty/types#EnvVar) | `true`   |
+| `username`  | MS Planner username  |  <CommonLink to="secrets">*EnvVar*</CommonLink> | `true`   |
+| `password`  | MS Planner password  |  <CommonLink to="secrets">*EnvVar*</CommonLink> | `true`   |
 | `defaults`  | MS Planner defaults  | `map[string]string`                                                           | `option` |
 | `values`    | MS Planner values    | `map[string]string`                                                           | `option` |

mission-control/docs/architecture.md

@@ -9,58 +9,39 @@
 
 
 
-Mission Control has a micro-service architecture with a shared data source with multiple deployment models. 
+Mission Control has a micro-service architecture with a shared data source with multiple deployment models.
 
 
 
-1. CLI 
+1. CLI
 2. Kubernetes (Helm Chart)
 3. SaaS
 
 
 
 
 
-Communication Model
+## Communication Model
 
 Communication between services happen in 3 ways:
 
-1. **Database** - A shared database with interface library enables services to query the data owned by other services by directly hitting the database using an interface provided by a shared library 
+1. **Database** - A shared database with interface library enables services to query the data owned by other services by directly hitting the database using an interface provided by a shared library
 2. **Messaging** - A postgres based message bus is used, database triggers insert events into queues which are then consumed by various services
 3. **HTTP/REST** - This model is primarily used when the service need to interact with services outside the DB (e.g. the APM hub needs to connect to log stores to retrieve logs)
 
+## Postgres
 
+Postgres is the only data store used by Mission Control and is also used as a JSON document database and message queue. This limits the dependencies and complexity especially when self-hosting.
 
-Shared Data Source
-
-
-
-All microservices use a shared database and model via the https://github.com/flanksource/duty project, this provides the following benefits:
-
-
-
-* Each microservice can update the database directly with full referential integrity between tables owned by other services
-* 
-
-
+All services use a shared database and model via the [duty](https://github.com/flanksource/duty) project, this provides the following benefits:
 
+* Limit RPC calls improving latency and performance
+* Services can run with slightly different versions of the library, limiting the need for coordinated migrations
+* Library updates happen automatically using dependabot
 
 
 ## Kubernetes & Gitops
 
-
-
 Mission control is kubernetes-native with all configuration being possible by Custom Resource Definition (CRD's)
 
 The single source of truth is still the database, the operators only function is to synchronize CRD's into the database and update the CRD status periodically.
-
-
-
-
-
-
-
-
-
-
-

mission-control/docs/config-db/concepts/transform.md

@@ -3,7 +3,7 @@
 Transformation can be performed after the configs have been scraped from the target. This can be useful when you want to
 
 - hide/remove sensitive data from the scraped configuration (e.g. passwords, tokens, etc.)
-- transform the scraped configuration using Javascript and [Go templates](https://pkg.go.dev/text/template).
+- transform the scraped configuration using CEL
 - remove certain fields from the scraped configuration
 
 | Field           | Description                                                                              | Scheme                                  | Required |
@@ -12,7 +12,7 @@ Transformation can be performed after the configs have been scraped from the tar
 | `javascript`    | Specify javascript syntax for script                                                     | `string`                                |          |
 | `jsonpath`      | Specify JSONPath                                                                         | `string`                                |          |
 | `expr`          | Specify Cel expression                                                                   | `string`                                |          |
-| `change`        | Apply transformaion on the scraped changes                                               | [`[]TransformChange`](#transformchange) |          |
+| `change`        | Apply transformaion on the scraped changes                                               | [`[]Changes`](#changes) |          |
 | `exclude`       | Fields to remove from the config, useful for removing sensitive data and fields          | [`[]Exclude`](#exclude)                 |          |
 |                 | that change often without a material impact i.e. Last Scraped Time                       |                                         |          |
 | [`mask`](#mask) | Specify configurations to replace sensitive fields with hash functions or static string. | [`[]Mask`](./mask.md)                   |          |
@@ -154,7 +154,7 @@ file:
 This configuration specifies 2 different masks. The first one will replace the value of the field `password` with the md5sum of the value. The second one will replace the value of the field `secret` with `***`.
 
 :::info
-All the masks will be applied in the order they are specified in the configuration file.
+Masks are applied in the order they are specified in the configuration file.
 :::
 
 ### Supported hash functions
@@ -171,25 +171,23 @@ At the moment, only `md5sum` is supported. More hash functions will be added in
 
 `Config DB` allows selectively applying masks to certain types of configs. So you could apply a mask to all the `Config` types and another mask to all the `Secret` types.
 
-### TransformChange
+## Changes
 
 | Field     | Description                                                       | Scheme     | Required |
 | --------- | ----------------------------------------------------------------- | ---------- | -------- |
-| `exclude` | Exclude is a list of CEL expressions that excludes a given change | `[]string` |          |
+| `exclude` | A list of CEL expressions that excludes a given change | `[]string` |          |
 
 The scraped changes can be accessed using the `details` field.
 
-#### Examples:
 
-1. Excluding canary pass/fail events
-
-```yaml title=""
+```yaml title="exclude-canary-pass-fail.yaml"
 spec:
   kubernetes:
     - clusterName: local-kind-cluster
       transform:
         change:
           exclude:
+            # Canary checker events are handled natively, no need to import the K8S events
             - 'details.source.component == "canary-checker" && details.reason == "Failed"'
             - 'details.source.component == "canary-checker" && details.reason == "Succeeded"'
 ```
@@ -200,7 +198,6 @@ This feature allows you to specify custom creation and deletion times for config
 
 You'll be making use of the `createFields` and `deleteFields` fields that are supported by all the scrapers. They are both a list of [JSONPath expression](../concepts/templating.md#jsonpath) and are used to extract the created/deleted time of the config item from the scraped configuration. If multiple fields are specified, the first non-empty value will be used.
 
-### Example
 
 Consider the following configuration file
 
@@ -233,13 +230,12 @@ aws:
 
 When the scraped configuration is saved in the database, the created date will be `2017-03-06T21:04:11Z` instead of being the current time and the deleted date will be `2017-04-04T15:04:05Z` instead of being empty.
 
-### Custom timestamp format
+### Timestamp Format
 
 By default, the timestamp format is RFC3339 (`2006-01-02T15:04:05Z07:00`). If the scraped configuration follows a different timestamp format, then you can specify it in the `timestampFormat` field. The format is specified using the [Go time format](https://golang.org/pkg/time/#Time.Format).
 
 In the above example if the value of `made_at` was `2017/03/06 21:04:11Z`, then the `timestampFormat` file would look like this
 
 ```yaml
----
 timestampFormat: '2006/01/02 15:04:05Z'
 ```

mission-control/docs/config-db/scrapers/kubernetes-file.md

@@ -39,9 +39,9 @@ The KubernetesFile config type is used to scrape configurations contained in you
 | `deleteFields`    | DeleteFields is a JSONPath expression used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used         | `[]string`                               | `false`  |
 | `selector`        | Specify Kubernetes resource for configuration based on `namespace`, `kind`, `name` and more.                                                                            | [`ResourceSelector`](./resourceselector) | yes      |
 | `container`       | Set container name                                                                                                                                                      | `string`                                 |          |
-| `files`           | Specify path to file contained in Pod                                                                                                                                   | `[]PodFile`                              |          |
+| `files`           | Specify path to file contained in Pod                                                                                                                                   | `[]File`                              |          |
 
-### PodFile
+### File
 
 | Field    | Description                       | Scheme     | Required |
 | -------- | --------------------------------- | ---------- | -------- |