Scan Kubernetes Custom Resources?

[tschran]

Question

Is there a way to configure the Kubernetes scanner to understand custom resources? We're trying to scan some clusters that are running Argo Rollouts and Argo Workflows. Both of those services use Custom Resources that define resources to create. Rollouts is basically a replacement for Deployments, so it will create and manage ReplicaSets, and Argo Workflows creates a bunch of pods. Right now, I'm getting separate reports for hundreds of Argo Workflow related pods, even though they're all coming from the same source. This, in turn, is causing the scan run to get OOMKilled frequently, because the run is producing a report that's over a gigabyte.
For the built-in resources, it looks like you're rolling up and deduping the scan findings. For instance, there may be a hundred pods on a cluster, but they all come from the same Deployment, so Trivy just reports one set of findings against that Deployment. I'd like to do the same thing for these custom resources.

Target

Kubernetes

Scanner

Vulnerability

Output Format

JSON

Mode

Standalone

Operating System

Ubuntu 22.04

Version

Version: 0.56.2
Vulnerability DB:
  Version: 2
  UpdatedAt: 2024-10-25 12:17:51.722810731 +0000 UTC
  NextUpdate: 2024-10-26 12:17:51.72281046 +0000 UTC
  DownloadedAt: 2024-10-25 15:58:00.246567921 +0000 UTC
Java DB:
  Version: 1
  UpdatedAt: 2024-10-23 02:37:26.813900849 +0000 UTC
  NextUpdate: 2024-10-26 02:37:26.813900579 +0000 UTC
  DownloadedAt: 2024-10-23 19:51:06.355657256 +0000 UTC
Check Bundle:
  Digest: sha256:ef2d9ad4fce0f933b20a662004d7e55bf200987c180e7f2cd531af631f408bb3
  DownloadedAt: 2024-09-18 20:21:26.698079409 +0000 UTC

[simar7]

AFAIK With trivy k8s scanning, you are only able to scan core Kubernetes components, so scanning custom resources might not be possible, correct me if I'm wrong @afdesk

Although having said that, if you're looking to scan custom resources with the trivy config misconfiguration scanner (i.e. scanning deployment YAML files, for instance) you could write your own schema and checks to be able to support custom resources like described here https://aquasecurity.github.io/trivy/v0.56/docs/scanner/misconfiguration/custom/schema/#custom-checks-with-custom-schemas

[afdesk]

@simar7 you're right. Trivy k8s doesn't scan custom resources now.
Trivy receives only next resources from k8s: https://github.com/aquasecurity/trivy-kubernetes/blob/b6099eb824b5be348f257560f2544ce20eeb6031/pkg/k8s/k8s.go#L406-L432

[itaysk]

it should be possible to create a custom check (for Argo resources) also without schema, isn't it?
there's a demo of this here: https://youtu.be/nGcvPAQdpVg?si=qXL9Tjkxfx_BRMak&t=545 some things might have changed since, but in principle it should work

[simar7]

@itaysk the demo uses a custom schema like I mentioned.

The project structure uses Rego convention so Trivy is automatically able to pick up the schema. This is documented here

[tschran]

We're only using the vulnerability scanner at this point.

[simar7]

@tschran sorry about the delayed response. As @itaysk mentioned, scanning is possible without a schema.

For instance, take a blue-green deployment strategy for a Rollout kind.

# This example demonstrates a Rollout using the blue-green update strategy, which contains a manual
# gate before promoting the new stack.
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rollout-bluegreen
spec:
  replicas: 2
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: rollout-bluegreen
  template:
    metadata:
      labels:
        app: rollout-bluegreen
    spec:
      containers:
      - name: rollouts-demo
        image: argoproj/rollouts-demo:blue
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
  strategy:
    blueGreen: 
      # activeService specifies the service to update with the new template hash at time of promotion.
      # This field is mandatory for the blueGreen update strategy.
      activeService: rollout-bluegreen-active
      # previewService specifies the service to update with the new template hash before promotion.
      # This allows the preview stack to be reachable without serving production traffic.
      # This field is optional.
      previewService: rollout-bluegreen-preview
      # autoPromotionEnabled disables automated promotion of the new stack by pausing the rollout
      # immediately before the promotion. If omitted, the default behavior is to promote the new
      # stack as soon as the ReplicaSet are completely ready/available.
      # Rollouts can be resumed using: `kubectl argo rollouts promote ROLLOUT`
      autoPromotionEnabled: false

Now you can write a Rego check as such:

# METADATA
# title: BlueGreen deployments must have Auto Promotion Enabled 
# description: Ensure that BlueGreen deployments have autoPromotionEnabled set to true
# schemas:
#   - input: schema["input"]
# custom:
#   id: ARGO001
#   severity: LOW
package user.argo001

deny[res] {
    not input.spec.strategy.blueGreen.autoPromotionEnabled
    res := result.new(
        sprintf("Auto promotion must be enabled, currently set to: %s", [input.spec.strategy.blueGreen.autoPromotionEnabled]),
        input.spec.strategy.blueGreen.autoPromotionEnabled
    )
}

And finally running this custom check:

 trivy  config --misconfig-scanners=yaml --config-check ./rollout-check.rego  --check-namespaces user . 
2024-12-06T22:40:20-07:00       INFO    [misconfig] Misconfiguration scanning is enabled
2024-12-06T22:40:21-07:00       INFO    Detected config files   num=1

rollout.yaml (yaml)

Tests: 1 (SUCCESSES: 0, FAILURES: 1)
Failures: 1 (UNKNOWN: 0, LOW: 1, MEDIUM: 0, HIGH: 0, CRITICAL: 0)

 (LOW): Auto promotion must be enabled, currently set to: false
════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
Ensure that BlueGreen deployments have autoPromotionEnabled set to true
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Just as a reminder, scanning arbitrary YAML or JSON files was added as part of Trivy v0.55, so please make sure you have at least that version. You can read more on this here https://trivy.dev/latest/docs/scanner/misconfiguration/#scan-arbitrary-json-and-yaml-configurations - do let us know if you have any other follow up questions.