Repeating Kustomization configured alerts every five minutes

[edutcher-cg]

Recently after upgrading our RabbitMQ Operator and cluster we have been getting repeating alerts about the cluster getting configured every five minutes. This is similar to this issue. In this thread it is mentioned that this can happen if two Kustomizations are fighting over the same resource and to check the flux tree. We have looked at the tree and it looks clean, no resource appears in two different kustomizations. Also, this message did not repeat with our kustomization setup before upgrading versions of the operator and cluster(went from Operator version 1.7 to 2.9 and RMQ version 3.8 to 3.13).

Our repo structure is:

• Clusters
  • Production
    • Infra
      • rmq
        • cluster-Kustomization
        • operator-Kustomization

Where the Operator Kustomization has the namespace, CustomReourceDefinition for the RMQ cluster, service account, Role and RoleBinding, ClusterRole and ClusterRoleBinding, and Deployment.

The cluster kustomiztion has the certs and the rabbitmq CustomResource.

Any help or thoughts would be greatly appreciated, thanks.

[kingdonb]

The alert body should tell which resource is "configured" - this can happen if there is a conflict between the YAML in the git repo and the spec body that the operator applies to itself. Usually operators do not write back to their own resources in the spec but only write to the status - but this is not a hard and fast rule.

You should be able to tell by looking at the resource filtered through kubectl neat (a krew plugin) what differs, if the spec is rewritten by the operator. If you can post more details about how this happens, we can advise you how best to proceed, but in brief: the default mode of Flux's kustomization apply is to wipe out any diff between the resource spec in git and the spec in the cluster.

You can use flux diff kustomization to see this diff live in the cluster.

You can probably use the ssa: merge annotation on the resource that conflicts to avoid this issue, if there is some difference in the spec that is caused by the new version of the operator, and it isn't an error, (doesn't belong in git, etc...)

https://fluxcd.io/flux/components/kustomize/kustomizations/#controlling-the-apply-behavior-of-resources

[edutcher-cg]

Thank you very much for this detailed response, this helped us track down the issue. It seems the old rabbitmq version is still listed in the spec:

spec:
delayStartSeconds: 30
image: rabbitmq:3.8.21-management
override:
service:
spec:
ports:
- name: new-tls-port
port: 5670
protocol: TCP
targetPort: 5670
statefulSet:
spec:
template:
metadata:
labels:
fargate: "no"
spec:
containers:
- image: rabbitmq:3.13.2-management

This explains the flux behavior. Just need to figure out why that old version is in the spec still, but that's likely just a RMQ issue we'll need to track down.