Repository navigation
[PP.5][Ops] Drain blockers may count DELETED workloads as resident #1042
Description
Current evidence status (2026-10-16)
ACCEPTANCE_PENDING. The defect was confirmed and the projection fix landed via #1046 (fbb5de0e), with deleted_tombstones_are_not_resident_drain_blockers and live_resources_remain_honest_drain_blockers regressions. The original investigation below is historical, not an assertion that current main still has the defect. Later protected maintenance/integrated passes are recorded in #1053/#1055; closure still requires mapping the real drain → delete → blocker-requery evidence to the checklist below, including storage/attachment behavior. No new product defect or completed final acceptance is inferred.
Context
Found during the PP.5 #1035 / PostgreSQL parity review. Pre-existing, not introduced by the parity fix, but it bears directly on the #1033 drain contract's promise to report resident blockers honestly.
Original defect (confirmed and repaired; historical description)
bins/o3kd/src/native_adapters/building_block.rs::derived_blockers counts every compute_instance resource whose create intent names one of the block's resource_provider_ids, and it does not consult observed_state.
OperationJournal::finish_lifecycle leaves desired_state intact when it writes observed_state = "DELETED", and the store retains DELETED as a deliberate tombstone (issue #89). So a workload that has been deleted may still be counted as a resident workload blocker for its block.
This was already true on SQLite, which has always returned DELETED tombstones from list_resources_by_kind. The PostgreSQL adapter used to hide it by filtering tombstones; that filter was a divergence and has been removed, so the behaviour is now the same on both backends.
Why it matters
The #1033 supported maintenance contract requires the operator to drain a BuildingBlock and see an honest list of what is still resident before taking the host down. If a deleted workload still counts, either:
- the operator is told a resolved workload is still resident (dishonest blocker report), or
- a block can never become drainable without an unrelated workaround.
It also affects the P15.7 drain journey's "blocker cleared before removal" step.
Required work
- Establish empirically whether a deleted workload clears the blocker, using the real drain path (the S5 drain journey is the natural place). Do not assume — the P15.7 journey passes today, so something may already compensate.
- If it is a defect, decide the correct semantics: a blocker is a resident workload, so a terminally deleted instance should not count. Repair at the projection layer (
derived_blockers/ the resource projection), not by filtering in each caller. - Add a regression that drains a block with a live workload, deletes the workload, and asserts the blocker report is honest at each step — and that a terminally deleted workload no longer counts while a live or in-flight one still does.
- Check the same question for the local-storage and attachment blocker kinds.
Non-goals
No evacuation, live migration or HA claim. Does not change the drain state machine.
Activity
zoorpha commentedon Sep 23, 2026
Status: intentionally NOT pre-judged; empirical validation still pending (blocked on committed-SHA S5 campaign).
Static suspicion unchanged this run: no fix attempted without the real drain path. Required sequence once the S5 campaign runs: workload ACTIVE on block-a -> Draining -> blocker present -> delete workload -> wait terminal -> blockers must not count the terminally DELETED workload; also inspect local-storage and attachment blockers. If defective, repair the shared blocker projection semantics (not caller-specific filtering). No evacuation work is in scope.
fix(operator): exclude terminal tombstones from drain blockers (#1042)
zoorpha commentedon Sep 23, 2026
Adjudicated: REAL defect — fixed in fbb5de0 (pushed on PR #1046).
Empirical + static evidence
The committed-SHA S5 campaign (RUN 990923001) at dd5cf40 was attempted 3x; the journey died earlier at the external-PG restart race (harness defect, separately fixed in c5a5993) before reaching drain, so the drain re-query was additionally adjudicated statically and by two independent review agents (Reviewer D architecture review + dedicated analysis), with fail-before/fix-after regression evidence:
BuildingBlockAdapter::derived_blockerscounted retained DELETED resource tombstones as resident Workload/LocalStorage/Attachment blockers:list_resources_by_kinddeliberately returns tombstones (callers must filter),resource_matches_provider_idsmatches on the preserved create intent, and terminal delete preserves the create intent in desired_state. On PostgreSQL this became user-visible with this PR's store parity fix; on SQLite it was latent since the drain-first contract landed.Fix
Skip observed_state == DELETED in the blocker projection (bins/o3kd/src/native_adapters/building_block.rs), mirroring the flavors terminal-filter idiom.
Regressions (fail-before verified)
The exact post-fix semantic (drain -> blocker present -> delete workload -> wait terminal DELETED -> blockers re-query shows not resident) will be re-exercised on the real host during the S5 rerun at c5a5993; closing this issue waits for that campaign evidence per PP.5 rules.
fix(o3kd): re-derive draining-block drain blockers in the read projec…
feat(pp5): exact-S5 eligibility-counted harness with empirical legs (#…