Skip to content

Commit

Permalink
Merge pull request #139 from flanksource/config-retention
Browse files Browse the repository at this point in the history 
feat: add config retention rules
  • Loading branch information
@moshloop
moshloop authored Jan 18, 2024
2 parents 71abf6c + fc6879c commit f949e3e
Show file tree
Hide file tree
Showing 10 changed files with 159 additions and 41 deletions.
3 changes: 3 additions & 0 deletions mission-control/Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,6 @@ sync:
cp -r ../canary-checker/static/img/icons static/img
cp docs/canary-checker/concepts/secret-management.md docs/reference/
cp docs/canary-checker/concepts/connections.md docs/reference/

watch: sync
npm exec -c 'docusaurus start'
33 changes: 33 additions & 0 deletions mission-control/docs/config-db/concepts/retention.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Retention

After scraping we can choose to retain results on the basis of name, age, count and more.

The retention rules are applied for each unique catalog item. If `changes` is specified with type `X` and count `20`, last 20 changes of `X` type would be kept for each catalog item

| Field | Description | Scheme | Required |
| --------- | ----------------------------------- | ----------------------------------------- | -------- |
| `changes` | Specify retention rules for changes | [`[]RetentionChanges`](#retentionchanges) | `false` |

## RetentionChanges

```yaml
retention:
changes:
- name: CreateRole
age: 30d # Any change older than 30 days is removed
count: 50 # Only 50 last changes will be retained

- name: PullSuceeded
age: 7d # Only keep one week of PullSuceeded changes

- name: ProvisioningFailed
count: 5 # Only 5 latest events are kept

```

| Field | Description | Scheme | Required |
| ------- | ------------------------------------------------------- | -------- | -------- |
| `name` | Name of the change type | `string` | `true` |
| `age` | Maximum age of the change type to retain (`12h`, `30d`) | `string` | `false` |
| `count` | Maximum count to retain the change type | `bool` | `false` |

17 changes: 17 additions & 0 deletions mission-control/docs/config-db/scrapers/aws.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,13 @@
This config type is used to scrape information about your AWS infrastructure.

```yaml
logLevel: ""
schedule: "@every 15m"
retention:
changes:
- name: CreateRole
age: 30d # Any change older than 30 days is removed
count: 50 # Only 50 last changes will be retained
aws:
- region:
- eu-west-2
Expand Down Expand Up @@ -32,6 +39,16 @@ transform:
- jsonpath: subnetArn
```
### Scraper
| Field | Description | Scheme | Required |
| ----------- | ---------------------------------------------------------------------------------- | -------------------------------------------- | -------- |
| `logLevel` | Specify the level of logging. | `string` | `false` |
| `schedule` | Specify the interval to scrape in cron format. Defaults to every 60 minutes. | `string` | `false` |
| `full` | Set to `true` to extract changes from scraped configurations. Defaults to `false`. | `bool` | `false` |
| `retention` | Settings for retaining changes, analysis and scraped items | [`Retention`](/config-db/concepts/retention) | |
| `aws` | Specifies the list of AWS configurations to scrape. | [`[]AWS`](#aws-1) | `false` |

### AWS

| Field | Description | Scheme | Required |
Expand Down
10 changes: 10 additions & 0 deletions mission-control/docs/config-db/scrapers/azure-devops.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,16 @@ azureDevops:
type: Release # the change type the pipeline produces
```
## Scraper
| Field | Description | Scheme | Required |
| ------------- | ---------------------------------------------------------------------------------- | -------------------------------------------- | -------- |
| `logLevel` | Specify the level of logging. | `string` | `false` |
| `schedule` | Specify the interval to scrape in cron format. Defaults to every 60 minutes. | `string` | `false` |
| `full` | Set to `true` to extract changes from scraped configurations. Defaults to `false`. | `bool` | `false` |
| `retention` | Settings for retaining changes, analysis and scraped items | [`Retention`](/config-db/concepts/retention) | |
| `azureDevops` | Specifies the list of Azure DevOps configurations to scrape. | [`[]AzureDevops`](#azuredevops-1) | `false` |

### AzureDevops

| Field | Description | Scheme | Required |
Expand Down
36 changes: 23 additions & 13 deletions mission-control/docs/config-db/scrapers/file.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,19 +13,29 @@ file:
For more examples of configuration file, please check the [GitHub repo](https://github.com/flanksource/config-db/tree/main/fixtures)
## Scraper
| Field | Description | Scheme | Required |
| ----------- | ---------------------------------------------------------------------------------- | -------------------------------------------- | -------- |
| `logLevel` | Specify the level of logging. | `string` | `false` |
| `schedule` | Specify the interval to scrape in cron format. Defaults to every 60 minutes. | `string` | `false` |
| `full` | Set to `true` to extract changes from scraped configurations. Defaults to `false`. | `bool` | `false` |
| `retention` | Settings for retaining changes, analysis and scraped items | [`Retention`](/config-db/concepts/retention) | |
| `file` | Specifies the list of File configurations to scrape. | [`[]File`](#file-1) | `false` |

### File

| Field | Description | Scheme | Required |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | -------- |
| `id` | A static value or JSONPath expression to use as the ID for the resource. | `string` | `true` |
| `name` | A static value or JSONPath expression to use as the Name for the resource. Default value is the `id`. | `string` | `false` |
| `items` | A JSONPath expression to use to extract individual items from the resource | `string` | `false` |
| `type` | A static value or JSONPath expression to use as the type for the resource. | `string` | `true` |
| Field | Description | Scheme | Required |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ | -------- |
| `id` | A static value or JSONPath expression to use as the ID for the resource. | `string` | `true` |
| `name` | A static value or JSONPath expression to use as the Name for the resource. Default value is the `id`. | `string` | `false` |
| `items` | A JSONPath expression to use to extract individual items from the resource | `string` | `false` |
| `type` | A static value or JSONPath expression to use as the type for the resource. | `string` | `true` |
| `transform` | Specify field to transform result | [`Transform`](../concepts/transform) | `false` |
| `format` | Format of config item, defaults to JSON, available options are JSON | `string` | `false` |
| `timestampFormat` | TimestampFormat is a Go time format string used to parse timestamps in createFields and DeletedFields. If not specified, the default is `RFC3339`. | `string` | `false` |
| `createFields` | CreateFields is a list of JSONPath expression used to identify the created time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` |
| `deleteFields` | DeleteFields is a JSONPath expression used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` |
| `url` | Specify URL e.g github repository containing the configuration(s) | `string` | `false` |
| `paths` | Specify paths to configuration(s) for scraping | `[]string` | `true` |
| `ignore` | Set configurations to ignore | `[]string` | `false` |
| `format` | Format of config item, defaults to JSON, available options are JSON | `string` | `false` |
| `timestampFormat` | TimestampFormat is a Go time format string used to parse timestamps in createFields and DeletedFields. If not specified, the default is `RFC3339`. | `string` | `false` |
| `createFields` | CreateFields is a list of JSONPath expression used to identify the created time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` |
| `deleteFields` | DeleteFields is a JSONPath expression used to identify the deleted time of the config. If multiple fields are specified, the first non-empty value will be used | `[]string` | `false` |
| `url` | Specify URL e.g github repository containing the configuration(s) | `string` | `false` |
| `paths` | Specify paths to configuration(s) for scraping | `[]string` | `true` |
| `ignore` | Set configurations to ignore | `[]string` | `false` |
10 changes: 10 additions & 0 deletions mission-control/docs/config-db/scrapers/kubernetes-file.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,16 @@ The KubernetesFile config type is used to scrape configurations contained in you
format: properties
```
## Scraper
| Field | Description | Scheme | Required |
| ---------------- | ---------------------------------------------------------------------------------- | -------------------------------------------- | -------- |
| `logLevel` | Specify the level of logging. | `string` | `false` |
| `schedule` | Specify the interval to scrape in cron format. Defaults to every 60 minutes. | `string` | `false` |
| `full` | Set to `true` to extract changes from scraped configurations. Defaults to `false`. | `bool` | `false` |
| `retention` | Settings for retaining changes, analysis and scraped items | [`Retention`](/config-db/concepts/retention) | |
| `kubernetesFile` | Specifies the list of Kubernetes File configurations to scrape. | [`[]KubernetesFile`](#kubernetesfile-1) | `false` |

### KubernetesFile

| Field | Description | Scheme | Required |
Expand Down
17 changes: 13 additions & 4 deletions mission-control/docs/config-db/scrapers/kubernetes.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,16 @@ kubernetes:
- nodeoutofmemory
```
## Scraper
| Field | Description | Scheme | Required |
| ------------ | ---------------------------------------------------------------------------------- | -------------------------------------------- | -------- |
| `logLevel` | Specify the level of logging. | `string` | `false` |
| `schedule` | Specify the interval to scrape in cron format. Defaults to every 60 minutes. | `string` | `false` |
| `full` | Set to `true` to extract changes from scraped configurations. Defaults to `false`. | `bool` | `false` |
| `retention` | Settings for retaining changes, analysis and scraped items | [`Retention`](/config-db/concepts/retention) | |
| `kubernetes` | Specifies the list of Kubernetes configurations to scrape. | [`[]Kubernetes`](#kubernetes-1) | `false` |

### Kubernetes

| Field | Description | Scheme | Required |
Expand All @@ -57,7 +67,7 @@ kubernetes:
| `exclusions` | Specify Kubernetes resources to be excluded from scraping | `[]string` | |
| **`kubeconfig`** | Specify kubeconfig for access to your Kubernetes Cluster | [`kommons.EnvVar`](https://pkg.go.dev/github.com/flanksource/kommons#EnvVar) | yes |
| `event` | Specify configuration to handle Kubernetes events. See [**KubernetesEvent**](#kubernetesevent) | [`KubernetesEvent`](#kubernetesevent) | yes |
| `relationships` | Create relationships between kubernetes objects. See [**KubernetesRelationships**](#kubernetesrelationships) | [`[]KubernetesRelationships`](#kubernetesrelationships) | `false` |
| `relationships` | Create relationships between kubernetes objects. See [**KubernetesRelationships**](#kubernetesrelationships) | [`[]KubernetesRelationships`](#kubernetesrelationships) | `false` |

### KubernetesEvent

Expand Down Expand Up @@ -85,20 +95,19 @@ are created automatically, there are cases where we want to link objects on the
We support static values, expressions and label lookups to find the parent

| Field | Description | Scheme | Required |
| ------------| ---------------------------------|-----------------------------------------------------------------| ---------|
| ----------- | -------------------------------- | --------------------------------------------------------------- | -------- |
| `kind` | `kind` of Kubernetes Object | [`KubernetesRelationshipLookup`](#kubernetesrelationshiplookup) | `true` |
| `name` | `name` of Kubernetes Object | [`KubernetesRelationshipLookup`](#kubernetesrelationshiplookup) | `true` |
| `namespace` | `namespace` of Kubernetes Object | [`KubernetesRelationshipLookup`](#kubernetesrelationshiplookup) | `true` |

#### KubernetesRelationshipLookup

| Field | Description | Scheme | Required |
| --------|----------------------------------------------------|----------| ---------|
| ------- | -------------------------------------------------- | -------- | -------- |
| `value` | Static string value of the resource | `string` | |
| `expr` | CEL Expression to evaluate | `string` | |
| `label` | Label key containing the value of the the resource | `string` | |


#### Kubernetes Relationship Example

```yaml
Expand Down
Loading

0 comments on commit f949e3e

Please sign in to comment.