Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Early error when failing to get alert status from coralogix #41

Merged
merged 1 commit into from
Aug 3, 2023

Conversation

ArthurSens
Copy link
Contributor

After deploying the coralogix-operator to staging and euplatform clusters, we noticed that it wold eventually get nilpointer exceptions on apis/coralogix/v1alpha1/alert_types.go:1082:

{"level":"info","ts":"2023-08-02T14:58:25Z","msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","controller":"alert","controllerGroup":"coralogix.com","controllerKind":"Alert","Alert":{"name":"recordtoolarge-alerting-kafka-connect","namespace":"platform"},"namespace":"platform","name":"recordtoolarge-alerting-kafka-connect","reconcileID":"20721bc0-68af-4920-a506-339c860480ce"}panic: runtime error: invalid memory address or nil pointer dereference [recovered]	panic: runtime error: invalid memory address or nil pointer dereference[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0xddfa64]goroutine 447 [running]:sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:119 +0x1fapanic({0x1829240, 0x2992280})	/usr/local/go/src/runtime/panic.go:884 +0x213github.com/coralogix/coralogix-operator/apis/coralogix/v1alpha1.(*AlertSpec).DeepEqual(0x1cc4460?, 0xc004f9c868?)	/workspace/apis/coralogix/v1alpha1/alert_types.go:1082 +0x44github.com/coralogix/coralogix-operator/controllers/alphacontrollers.(*AlertReconciler).Reconcile(0xc00043bee0, {0x1cdb2f8, 0xc0051b2e40}, {{{0xc000653d40?, 0x0?}, {0xc0002f38f0?, 0x40de67?}}})	/workspace/controllers/alphacontrollers/alert_controller.go:211 +0x1a25sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1cdb2f8?, {0x1cdb2f8?, 0xc0051b2e40?}, {{{0xc000653d40?, 0x179cf20?}, {0xc0002f38f0?, 0x0?}}})	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:122 +0xc8sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0000a2780, {0x1cdb250, 0xc0002ffef0}, {0x18b4280?, 0xc000143ac0?})	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:323 +0x35fsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0000a2780, {0x1cdb250, 0xc0002ffef0})	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274 +0x1d9sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235 +0x85created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:231 +0x587

After a few hours trying to reproduce it in a Kind cluster 😅, I noticed that we were calling DeepEqual(line 211) with a nil AlertStatus. This is happening because we're not handling the error received from flattenAlert, or rather handling it too late.

@ArthurSens ArthurSens requested a review from a team as a code owner August 2, 2023 17:13
Signed-off-by: Arthur Silva Sens <arthur.sens@coralogix.com>
@ArthurSens
Copy link
Contributor Author

After leaving the operator running for a few more hours using this branch's code, unfortunately, I got a nil pointer in that same place... so the PR doesn't solve all problems :/

@ArthurSens ArthurSens merged commit e9b8fb6 into main Aug 3, 2023
4 of 5 checks passed
@ArthurSens ArthurSens deleted the fix-nilpointer branch August 3, 2023 18:18
@github-actions
Copy link
Contributor

github-actions bot commented Aug 4, 2023

🎉 This PR is included in version 0.1.9 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants