notification-controller: Events to Git Provider

[phillebaba]

The current work done with the Notification Controller would support sending events to Slack, MSTeams etc. in a similar way to how flux-cloud works today. The downside of consuming events through ChatOps is that the message could be considered ephemeral. Additionally it would be difficult to build a CI that could fetch the information in a reasonable way.

Most Git providers offer a status API for tools outside of the "native" CI to feedback the status of a build job. Meaning that there exists an alternative destination for the notification events.

• Github
• Gitlab
• Azure DevOps
• Bitbucket

I started building flux-status to try out my idea which I felt worked pretty well. The downside of flux-status is that it works the same way flux-cloud does, which means that it would have to be rewritten when FluxV2 gets released. After discussing the matter with @stefanprodan we came to the agreement that having yet another project to solve a small problem would be counter productive. Instead it might be better to add this functionality as a new provider in the notification controller.

There would have to be some differences between the current providers and those that talk to git providers. First of all we would need to find the source repository for the event as the event only includes the sha of the commit. We would also need to translate the event type to a status state specific to that API. It is also unclear if all events can/should be sent to the commit status. As an example should a helm chart failure after it has been successfully running for days update the status of the commit or not.

In my head a typical workflow would look something like this:

1. A new commit is pushed to the source repository
2. Flux applies the manifests to the cluster (HelmRelease and Kustomization)
3. Set the status of the commit that the apply has either failed or succeeded
4. The HelmController will reconcile the HelmRelease and install the chart
5. Set the status of the commit that the Helm chart has either failed or succeded

We would end up with n+1 commit statuses. One for the initial apply of the commit and then one for the health check of every HelmRelease and Kustomization. As far as I have checked all of the git providers would support sending multiple statuses per commit, I have seen that Github has a limit of 1000 statuses per SHA so it might be worthwhile looking into how we could bundle the events into a single or a couple of statuses instead. Doing it this way would cause an edge case, as it would be possible for a commit to apply but the time for the Pod to start might be long, allowing for a new commit to be made and applied during that time. We now end up with a situation where changes made in the first commit will have its status posted to the latest commit SHA.

There are some challenges when working with commit status. The main one is the message size limit for each status, which are not documented. Azure DevOps seems to have a limit at 1000 characters while Github and the other providers have a limit at 200 characters. Stefan had an idea that if we used PRs we could post a lot more information as there are no limits on how large the comment can be. But I think that it would require a lot more work to achieve the functionality, so it is better to start off with commit statuses and see where that takes us.

[phillebaba]

Getting notifications to update a commit status was not too difficult, but my current solution is hard coded to a single repository. Now that I understand the relationship between events and their sources I don't think there are going to be any issues implementing the lookup.

I have some points that need to be discussed though, as I think they will cause issues in the future.

• The revision in the event is linked to the repository from which for example the Kustomization comes from. This may cause issues if the end user to not own the source of the Kustomization. On the other hand it is now possible in theory to have an infinite chain of Sources and Kustomizations meaning that it would be difficult to determine the root source. My initial solution was based on the axiom that flux would only work with a single repository, so we need to rethink the strategy.
• Access tokens need to be passed to the provider so an additional field is required. Additionally in theory a single provider may be used for multiple repositories in different git providers. We need some sort of configuration parameter to link personal access tokens with specific repositories, unless we use a solution similar to what I have build for Azure DevOps.
• A single commit will cause many events, so should the commit status be updated every time an event occurs. Even if it may have been successful before.

Would love any feedback on the points above.

[stefanprodan]

Proposal

We can introduce notification providers of type <GIT-PROVIDER>-commit-status.

Github provider example:

apiVersion: notification.fluxcd.io/v1alpha1
kind: Provider
metadata:
  name: podinfo-gh-status
  namespace: gitops-system
spec:
  type: github-commit-status
  address: https://github.com/stefanprodan/podinfo # repo URL
  secretRef:
    name: github-token # GH token with 'repo:status' scope

Alert example:

apiVersion: notification.fluxcd.io/v1alpha1
kind: Alert
metadata:
  name: podinfo-status-update
  namespace: gitops-system
spec:
  providerRef: 
    name: podinfo-gh-status 
  eventSeverity: info
  eventSources:
    - kind: Kustomization
      name: podinfo-staging
    - kind: Kustomization
      name: podinfo-prod

The kustomize-controller issues events with the source revision stored as metadata:

{
  "severity": "info",
  "reportingController": "kustomize-controller",
  "reason": "ApplySucceed",
  "message": "service/podinfo create/ndeployment/podinfo created/n....apply log can be huge",
  "involvedObject": {
    "kind": "Kustomization",
    "name": "podinfo-prod",
    "namespace": "gitops-system"
  },
  "metadata": {
    "revision": "master/a1afe267b54f38b46b487f6e938a6fd508278c07",
  }
}

For alerts providers of type github-commit-status, the notification-controller will parse the event and extract the metadata.revision,involvedObject.name,severity and reason.

Using the above information, notification-controller will post to GH API a status of type success with the description Kustomization podinfo-prod apply succeeded for the commit and branch extracted from revision.

{
  "severity": "error",
  "reportingController": "kustomize-controller",
  "reason": "HealthCheckFailed",
  "message": "health check failed for Deployment podinfo-prod/podinfo: long error message....",
  "involvedObject": {
    "kind": "Kustomization",
    "name": "podinfo-prod",
    "namespace": "gitops-system"
  },
  "metadata": {
    "revision": "master/a1afe267b54f38b46b487f6e938a6fd508278c07",
  }
}

If the severity is error then the status type is error, the message is composed the same.

Message format: {{ .involvedObject.kind }} {{ .involvedObject.name }} {{ .reason }}

[phillebaba]

The revision comes from source-controller, and is the branch and commit of the repository that is being reconciled on the cluster by the kustomize-controller. Note that the repo doesn't have to contain a kustomization.yaml. The kustomize-controller will generate one on the fly so that the repo structure can be the same as what Flux v1 expects. There is no other repo involved.

I get that, and I am not sure if that behavior will work if promotion is done from the repo Kustomization resource originates from. For example the tk cli example would not work for anyone other than you as the notification controller would attempt to update the commit status of the pod-info repository.

I suspect people with self hosted Gitlab instances will have problems if the provider type is not specified. It would be difficult to determine the type from just the URL if its a custom domain.

[stefanprodan]

For example the tk cli example would not work for anyone other than you as the notification controller would attempt to update the commit status of the pod-info repository.

It will work, promotion means switching from a branch to another, or to a git tag. The notification controller has that info in the revision so it knows where to post the status, if it's a commit on a branch or if it's tag and so on.

  "metadata": {
    "revision": "v4.0.5/a1afe267b54f38b46b487f6e938a6fd508278c07",
  }

I suspect people with self hosted Gitlab instances will have problems if the provider type is not specified. It would be difficult to determine the type from just the URL if its a custom domain.

We should make the API explicit, having git as a provider is not an option, we don't do that anywhere in the toolkit APIs. The current notification providers have explicit names Slack, Teams, etc and so will the status providers.

[phillebaba]

Ok sounds good! I think the best thing to do now is for me to finish my implementation for Github and create a PR for it. That allows us to see if there are any unforeseen problems in the proposed solution.

[stefanprodan]

@phillebaba thank you! Please create an issue in notification-controller with the API proposal for GitHub. PRs can't link to discussions, GH discussions are beta and lack some features.

[phillebaba]

Created an Issue to track the implementation.

fluxcd/notification-controller#26

[gerardgorrion]

Hi all. We are triying to use alert to point commit used into bitbucket repo, with multiple clusters running with flux and same repo. What is the way to do? install podinfo is necessary? is necessary to add every crd at the same repo by every cluster? Thanks,

[NasAmin]

@stefanprodan Is there a way to specify a branch for the Git provider? I need to be able to tell the GitHub notification provider to listen to changes in a specific branch e.g. a Config branch.
Also does it have to be http/https. Is ssh not supported?

[NasAmin]

Never mind about the branch. Just had to do
https://github.com/owner/repo?ref=branch