Better error handling of "unexpected EOF"

[bendikp]

When something is wrong in the parsing of the scan Job logs, we get a "unexpected EOF" error in the controller logs and the scan Jobs is never retried.

{
  "level": "error",
  "ts": "2023-01-17T20:06:40.568Z",
  "msg": "Reconciler error",
  "controller": "job",
  "controllerGroup": "batch",
  "controllerKind": "Job",
  "Job": {
    "name": "deployment-vuln-app-app-eccf8-89288",
    "namespace": "image-scanner-jobs"
  },
  "namespace": "image-scanner-jobs",
  "name": "deployment-vuln-app-app-eccf8-89288",
  "reconcileID": "b13fa5da-bb8d-4a65-9463-8d5e0f7b68ab",
  "error": "unexpected EOF",
  "stacktrace": "sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/internal/controller/controller.go:235"
}

[erikgb]

I am thinking we should start by just improving the error handling and update the CIS status with the error. Then we can see how to improve this further. WDYT?

[padlar]

I agree to temporarily update the CIS status with error. As I see it, the problem is triggered from outside and the best we can do is to fail and retry the scan job -
-- if the trivy-server is sending different json output in the failed scenario? If yes, then the above solution is good enough.
-- if the problem is with the transporation layer or k8s log bytes limitation, we gotta figure out the root cause