Skip to content

Log reconciliation conflicts at info instead of error - #73

Merged
antiguru merged 2 commits into
MaterializeInc:mainfrom
antiguru:quiet-transient-conflicts
Oct 19, 2026
Merged

antiguru merged 2 commits into
MaterializeInc:mainfrom
antiguru:quiet-transient-conflicts

Conversation

@antiguru

@antiguru antiguru commented Oct 17, 2026 •

Copy link
Copy Markdown
Member

A reconciliation that fails because the object changed since the controller read it resolves on its own when the object is retried at its newer version. With leader election this happens on every leadership change, since the new leader reconciles from a cache that lags the API server. Each failure is logged at error, so orchestratord sends a Sentry event per failure, which uses up our error budget.

Error::is_conflict treats two cases as conflicts. The first is a 409 Conflict anywhere in the source chain. The second is a failed test operation in the finalizer patch, which the API server reports as a 422 without object details. reconcile logs these at info. AlreadyExists and validation 422s are still logged at error.

🤖 Generated with Claude Code

A reconciliation that fails because the object changed since the
controller read it resolves on its own when retried against the newer
version. This happens routinely whenever the cache lags the API server,
for instance after a leadership change, and logging each one at error
floods error trackers.

Error::is_conflict classifies a 409 Conflict anywhere in the source
chain and a failed test operation in the finalizer patch, which the API
server reports as a 422 without object details.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Alphadelta14
Alphadelta14 previously approved these changes Oct 19, 2026
Comment thread src/controller.rs Resolved
@antiguru
antiguru marked this pull request as ready for review October 19, 2026 15:44
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@antiguru antiguru changed the title Log reconciliation conflicts at warn instead of error Log reconciliation conflicts at info instead of error Oct 19, 2026
@antiguru
antiguru requested a review from Alphadelta14 October 19, 2026 15:47
@antiguru
antiguru merged commit 75bffc1 into MaterializeInc:main Oct 19, 2026
2 checks passed
@github-actions github-actions Bot mentioned this pull request Oct 19, 2026
@antiguru
antiguru deleted the quiet-transient-conflicts branch October 20, 2026 07:46
antiguru added a commit to MaterializeInc/materialize that referenced this pull request Oct 20, 2026
k8s-controller 0.14.1 logs reconciliation conflicts at info instead of
error (MaterializeInc/k8s-controller#73).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants