OpenTelemetry Integration

[saada]

We would like to get traces out of Flux to help debug issues with Flux and get visibility on what's happening. This is even more valuable now that Flux is broken up into different controllers. Would it be possible to integrate with OpenTelemetry?

Example use case:

• I want to see everything that happened from when a git change occurred in a single trace. All the applied yaml files in Source Controller, all the notifications that went out to Notifications Controller, all the HelmReleases that were applied in Helm-Controller, Kustomize Controller, etc...
• I want to build a UI using trace data to track release changes and integrate deeply with a Git commit/tag, a Docker image tag, and the GOTK flow of operations

[stefanprodan]

Similar to to how we defile alerts we could extend notification-controller to collect and push traces to an OpenTelemetry provider.

Define a tracing provider:

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Provider
metadata:
  name: jaeger
  namespace: default
spec:
  type: jaeger
  address: http://jaeger-collector.jaeger-system.svc.cluster.local:9411

Enable tracing for specific reconciliations:

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Trace
metadata:
  name: webapp
  namespace: default
spec:
  providerRef: 
    name: jaeger
  eventSources:
    - kind: Kustomization
      name: webapp-backend
    - kind: HelmRelease
      name: webapp-frontend

Based on the above config, notification-controller would parse the events received and create a span for each source revision.

[stanfordpeng]

jaegar is not one of the provider type(https://fluxcd.io/flux/components/notification/providers/#type). Will it work?

[stanfordpeng]

Oh it is still a proposal

[stoetti]

@stefanprodan Are there any plans on adding OpenTelemetry-Support in the near future?