Workflow + debugging FluxCD errors -- Any advice?

[Nathan-Nesbitt]

Hey all,

I'm currently trying to play around with flux and get a basic K8s cluster going using it. Maybe I missed something in the docs, or there's some good docs on best practices, but I'm finding it very difficult to debug/find the core issue of problems in flux and get basic applications running.

Currently my workflow is:

1. Create a local cluster bootstrapped with flux
2. Make small change to flux to try to get an app to run
3. Commit the change
4. Wait for it to reconcile
5. Check the logs from flux get all for errors thrown
6. Repeat from step 2

My issues are:

• This process takes a really long time and doesn't produce very useful logs
• This process makes a mess of the git repository
• There's no way to check if your changes are going to work until after you commit them
• A lot of the errors are very abstract and appear to be thrown by all kustomizations at the same time? (Maybe this is just because I am new and don't know how to check the logs a better way, but it makes it hard to tell what is actually broken...)

What I'm hoping to get answers for:

• What is the best way to test this without having to make small pushes + hope that it fixes the problems with flux?
• Is there some script that will throw the errors before committing?
• What is the suggested workflow going from designing the development --> staging --> production environments?

Thanks!

[Alexander-N]

What has been helpful for me is building and applying the kustomizations locally and commit after everything works

kustomize build . | kubectl apply -f -

If you want to do the same thing with SOPS encrypted secrets you have to decrypt them first

kustomize build . \
| yq 'select(.kind == "Secret")' \
| sops --input-type yaml --output-type yaml -d --ignore-mac /dev/stdin \
| kubectl apply -f -

When using post build variable substitution, you could use flux build kustomization instead.

To avoid polluting the git history and test changes before merging you can configure your local cluster to use a feature branch like this . I can also recommend this script to catch obvious problems.

[stefanprodan]

There is also flux diff kustomization that allows you to review the local changes before pushing them to Git.

[stefanprodan]

What is the suggested workflow going from designing the development --> staging --> production environments?

Please see https://fluxcd.io/flux/guides/repository-structure/ and https://github.com/fluxcd/flux2-kustomize-helm-example

For Helm charts there is a dedicated guide for promotions here: https://fluxcd.io/flux/use-cases/gh-actions-helm-promotion/

[arthur-c]

@Nathan-Nesbitt I just discovered Flux this week but I already spent many hours learning and playing with it (I love it!) I believe you can implement tests in your pipeline before merging your changes in the main branch. And even if you push directly to the branch, you can have a kustomization test in your pre-commit config.

This gitOps workflow is new to me so I am exploring a lot, but I can already foresee these kinds of tests and feedbacks for the end-user without having to go to a different interface. Small example: jobs on the merge event pipeline to wait for the reconciliation and display the Flux and release statuses.

[stefanprodan]

Flux can update the merge commit status on its own, see: https://fluxcd.io/flux/components/notification/providers/#git-commit-status-updates