Skip to content

Fix informer delete handling#2039

Open
27rohan wants to merge 1 commit into
Altinity:0.27.2from
27rohan:fix/informer-delete-handling
Open

Fix informer delete handling#2039
27rohan wants to merge 1 commit into
Altinity:0.27.2from
27rohan:fix/informer-delete-handling

Conversation

@27rohan

@27rohan 27rohan commented Jul 17, 2026

Copy link
Copy Markdown

Summary

When Kubernetes tells the operator that something was deleted, the notification sometimes arrives in an unexpected wrapper format. The operator's code didn't handle that case and would crash.

This PR teaches the code to unwrap both formats safely — and if the notification is garbage, it just logs an error instead of crashing. Tests were added to cover all these cases.

Why this is needed

client-go may wrap a deleted object in DeletedFinalStateUnknown when a watch deletion is missed and a subsequent relist discovers that the cached object is gone. The existing handlers use bare type assertions, so receiving this valid client-go payload panics the operator.

This exact failure has occurred in released versions of the upstream operator:

This PR only makes existing delete callbacks safe. It does not enable the dormant CHI ReconcileDelete queue path or otherwise change deletion policy.

Fix

Routes both direct objects and DeletedFinalStateUnknown tombstones through a single typed delete handler in the CHI controller's informer callbacks, so malformed delete events are reported instead of causing panics. Includes regression tests for direct, tombstone, malformed, and nil payloads.

  • decode direct objects and DeletedFinalStateUnknown tombstones through one typed delete handler, reporting malformed events instead of panicking
  • protect all CHI controller informer delete callbacks
  • add regression coverage for direct, tombstone, malformed, and nil payload cases

Test plan

  • go build -mod=readonly ./...
  • go test -mod=readonly -race -count=1 -vet=off ./pkg/controller/chi/...

Decode direct objects and DeletedFinalStateUnknown tombstones through one typed
delete handler, reporting malformed events instead of panicking.

Signed-off-by: Rohan Thakkar <rohant@twitter.com>
@27rohan
27rohan force-pushed the fix/informer-delete-handling branch from d6a290d to 320703a Compare July 17, 2026 19:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant