Image update intermittently stops working: "object not found"

[Poundex]

Hello,

We are having a problem with our image update automation, specifically that it sporadically stops working. It will do one or two Git commits and then lock up saying "object not found". But it doesn't specify which object cannot be found, and every now and then it will do a successful push.

The only log message in the controller is
{"level":"error","ts":"2023-09-21T10:17:33.875Z","msg":"Reconciler error","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind" :"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"xxxxx","namespace":"xxxxxx"},"namespace":"xxxxxx","name":"xxxxx","reconcileID":"354 e2231-3c0a-470e-86c2-7a3b07ec3e61","error":"object not found"}

Trying to manually reconcile I see:
$ flux reconcile --verbose image update -n xxxxx xxxxx ► annotating ImageUpdateAutomation xxxxx in xxxxx namespace ✔ ImageUpdateAutomation annotated ◎ waiting for ImageUpdateAutomation reconciliation ✗ ImageUpdateAutomation reconciliation failed: 'object not found'

I can see that all the configuration is good. The ImageRepositories have tag lists, the ImagePolicies have their Latest Image set correctly and like I said it does an update every now and then but then goes immediately back to being stuck.

Different deployments are working/not working at the same time. It is not the case that they are either all working or not working, which means the cluster-wide shared objects (such as the GitRepository and credentials etc) are also fine (which they would have to be as like I said the updates do work sometimes).

We did not see this in testing and went ahead and migrated our development environments to Kubernetes and Flux and now many of them are stuck so if anybody could shed any light I would be extremely grateful.

I made sure the image update controller was latest and updated it to 0.36.1

[Poundex]

It seems that deploying a new unreleated ImageUpadeAutomation in a different namespace can temporarily unstick other ImageUpdateAutomations that are "object not found"ing and they will all commit their changes successfully at once

[Poundex]

For a particular deployment I can remove all of the automation resources (ImageUpdateAutomation/ImagePolicies/ImageRepositories) and then re-create them (this is just a Helm flag so the recreated objects are exactly the same). This gets me exactly one successful commit before it locks up again:

Status:
  Conditions:
    Last Transition Time:    2023-09-21T17:06:32Z
    Message:                 object not found
    Reason:                  ReconciliationFailed
    Status:                  False
    Type:                    Ready
  Last Automation Run Time:  2023-09-21T17:06:28Z
  Last Push Commit:          2db050ab4846cf962ee57bec3a8b25bdb0b84a33
  Last Push Time:            2023-09-21T17:06:28Z
  Observed Generation:       1
Events:
  Type    Reason  Age   From                         Message
  ----    ------  ----  ----                         -------
  Normal  info    2s    image-automation-controller  committed and pushed commit '2db050ab4846cf962ee57bec3a8b25bdb0b84a33' to branch 'master'
00000000000.dkr.ecr.eu-west-2.amazonaws.com/xxxxx/xxxxx:0.6.0-branch-master.t20230921-164402.c1.gef327ee
  Warning  error  0s (x2 over 1s)  image-automation-controller  object not found

Even though it says "Warning" it is more than that because it doesn't matter how many actual image pushes are done (and how successfully the ImageRepository and ImagePolicy are able to work together to get the new tag name) it will not commit any more updates.

[PaulSxxxs]

Are you using bitbucket?

[PaulSxxxs]

Similar: #498