Skip to content

[PP.5][Ops] Drain blockers may count DELETED workloads as resident #1042

Open

Description

@zoorpha

Current evidence status (2026-10-16)

ACCEPTANCE_PENDING. The defect was confirmed and the projection fix landed via #1046 (fbb5de0e), with deleted_tombstones_are_not_resident_drain_blockers and live_resources_remain_honest_drain_blockers regressions. The original investigation below is historical, not an assertion that current main still has the defect. Later protected maintenance/integrated passes are recorded in #1053/#1055; closure still requires mapping the real drain → delete → blocker-requery evidence to the checklist below, including storage/attachment behavior. No new product defect or completed final acceptance is inferred.

Context

Found during the PP.5 #1035 / PostgreSQL parity review. Pre-existing, not introduced by the parity fix, but it bears directly on the #1033 drain contract's promise to report resident blockers honestly.

Original defect (confirmed and repaired; historical description)

bins/o3kd/src/native_adapters/building_block.rs::derived_blockers counts every compute_instance resource whose create intent names one of the block's resource_provider_ids, and it does not consult observed_state.

OperationJournal::finish_lifecycle leaves desired_state intact when it writes observed_state = "DELETED", and the store retains DELETED as a deliberate tombstone (issue #89). So a workload that has been deleted may still be counted as a resident workload blocker for its block.

This was already true on SQLite, which has always returned DELETED tombstones from list_resources_by_kind. The PostgreSQL adapter used to hide it by filtering tombstones; that filter was a divergence and has been removed, so the behaviour is now the same on both backends.

Why it matters

The #1033 supported maintenance contract requires the operator to drain a BuildingBlock and see an honest list of what is still resident before taking the host down. If a deleted workload still counts, either:

  • the operator is told a resolved workload is still resident (dishonest blocker report), or
  • a block can never become drainable without an unrelated workaround.

It also affects the P15.7 drain journey's "blocker cleared before removal" step.

Required work

  1. Establish empirically whether a deleted workload clears the blocker, using the real drain path (the S5 drain journey is the natural place). Do not assume — the P15.7 journey passes today, so something may already compensate.
  2. If it is a defect, decide the correct semantics: a blocker is a resident workload, so a terminally deleted instance should not count. Repair at the projection layer (derived_blockers / the resource projection), not by filtering in each caller.
  3. Add a regression that drains a block with a live workload, deletes the workload, and asserts the blocker report is honest at each step — and that a terminally deleted workload no longer counts while a live or in-flight one still does.
  4. Check the same question for the local-storage and attachment blocker kinds.

Non-goals

No evacuation, live migration or HA claim. Does not change the drain state machine.

Activity

  1. zoorpha commented on Sep 23, 2026

    @zoorpha
    CollaboratorAuthor

    Status: intentionally NOT pre-judged; empirical validation still pending (blocked on committed-SHA S5 campaign).

    Static suspicion unchanged this run: no fix attempted without the real drain path. Required sequence once the S5 campaign runs: workload ACTIVE on block-a -> Draining -> blocker present -> delete workload -> wait terminal -> blockers must not count the terminally DELETED workload; also inspect local-storage and attachment blockers. If defective, repair the shared blocker projection semantics (not caller-specific filtering). No evacuation work is in scope.

  2. zoorpha commented on Sep 23, 2026

    @zoorpha
    CollaboratorAuthor

    Adjudicated: REAL defect — fixed in fbb5de0 (pushed on PR #1046).

    Empirical + static evidence

    The committed-SHA S5 campaign (RUN 990923001) at dd5cf40 was attempted 3x; the journey died earlier at the external-PG restart race (harness defect, separately fixed in c5a5993) before reaching drain, so the drain re-query was additionally adjudicated statically and by two independent review agents (Reviewer D architecture review + dedicated analysis), with fail-before/fix-after regression evidence:

    • BuildingBlockAdapter::derived_blockers counted retained DELETED resource tombstones as resident Workload/LocalStorage/Attachment blockers: list_resources_by_kind deliberately returns tombstones (callers must filter), resource_matches_provider_ids matches on the preserved create intent, and terminal delete preserves the create intent in desired_state. On PostgreSQL this became user-visible with this PR's store parity fix; on SQLite it was latent since the drain-first contract landed.
    • Consequence: a block whose workloads reached terminal DELETED stayed falsely drain-blocked and unremovable; GET /operator/building-blocks/{id} also returned stale stored blockers.

    Fix

    Skip observed_state == DELETED in the blocker projection (bins/o3kd/src/native_adapters/building_block.rs), mirroring the flavors terminal-filter idiom.

    Regressions (fail-before verified)

    • deleted_tombstones_are_not_resident_drain_blockers: drain reports no blockers with DELETED server+volume+attachment tombstones and remove succeeds — failed before with Workload/LocalStorage/Attachment count 1 each.
    • live_resources_remain_honest_drain_blockers: a live workload still blocks drain and still rejects removal.

    The exact post-fix semantic (drain -> blocker present -> delete workload -> wait terminal DELETED -> blockers re-query shows not resident) will be re-exercised on the real host during the S5 rerun at c5a5993; closing this issue waits for that campaign evidence per PP.5 rules.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions