Fix interrupted RocksDB table drops

[kriszyp]

Validation found that this Harper diff does not fix the storage-layer defect reported in #1381. The failing catalog write uses a metadata column-family handle invalidated by a concurrent native database destroy/open lifecycle race; the root fix is rocksdb-js PR #787, “Serialize database destruction with concurrent opens”, which is not yet merged or released. This draft should not merge as-is.

The column-family postcondition, cross-worker source-cache guard, and async entry-handler rejection handling in this diff may be useful defense-in-depth changes, but they are not a substitute for enforcing the process-global path lifecycle invariant in rocksdb-js. Keep this draft parked until #787 is merged and published, then pin the released dependency and rerun the regression before deciding which Harper-side changes remain justified.

Refs #1381.

For the human reviewer

1. Do not merge this draft as the Cross-worker write can race RocksDB table drop and poison catalog cleanup #1381 fix. The observed failure crosses the rocksdb-js native registry boundary, where database destruction releases a path before physical teardown is complete and a concurrent worker can reopen it. A Harper-side CF sweep cannot restore a metadata handle after that lifecycle violation.
2. After rocksdb-js Skip license enforcement for model-* metrics in dev mode (no installed license) #787 is released, decide whether to retain any of this diff as separately scoped defense in depth. Keeping it would require resolving the open review feedback and adding coverage for the persisted-tombstone-only source-cache path; dropping it avoids carrying unrelated Scope behavior in a storage fix.

Verification

• npm run build — passed on Node 26.2.0.
• npx mocha unitTests/resources/dropTableGhost.test.js unitTests/components/Scope.test.js — 49 passed, 1 LMDB-only pending.
• npm run test:integration -- --isolation=none "integrationTests/apiTests/blob.test.mjs" — 22 passed on Node 26.2.0.
• The same integration command — 22 passed on Node 24.19.0.
• PR CI passed every Node 24 integration shard. Its Node 22/24/26 unit jobs each failed on the unrelated DeploymentRecorder.ingestPayload transaction context timeout-budget assertion.

These green runs confirm the branch is mechanically healthy; they do not reproduce the intermittent native destroy/open race. The added tests exercise simulated redundant CF drops and orphaned CF cleanup, not the catalog-handle invalidation seen in the failing integration logs.

Review coverage

Authored by GPT-5 Codex. Claude and Gemini reviewed commit 3e271283cb75; both left open comments on the Harper diff. Root-cause validation traced the lifecycle invariant to rocksdb-js #787, whose current Node 22/24/26, native, stress, and benchmark checks are green but which remains unreleased.

— GPT-5 Codex

_{Human-Review-Need: 4 @ 3e27128}

[gemini-code-assist]

Code Review

This pull request improves the robustness of table dropping and promise rejection handling. It ensures that redundant or interrupted table drops on RocksDB are handled safely by verifying that all prefixed column families are completely removed before clearing catalog rows, and it prevents unhandled promise rejections during scope initial loads. Feedback on the changes suggests using the centralized openRocksDatabase helper in resources/Table.ts instead of instantiating RocksDatabase directly to ensure database configurations are consistently applied.

[gemini-code-assist]

Bypassing the centralized openRocksDatabase helper and directly instantiating new RocksDatabase means that important configurations such as disableWAL, compression (and compressionForAllColumnFamilies), and readOnly mode are not applied to the opened column family store. To maintain consistency and ensure all database options are correctly respected, please export openRocksDatabase from resources/databases.ts, import it in resources/Table.ts, and use it here.

Suggested change

	const columnStore = new RocksDatabase(rootStore.path, { name: columnName }).open();
	const columnStore = openRocksDatabase(rootStore.path, { name: columnName });

[claude]

File: resources/Table.ts:406, 5984, 6109
What: tableIsDropping() adds a new OR branch — reading the persisted tombstone via dbisDb.getSync(tableName + '/')?.dropping — specifically so a different worker's drop (which never sets this worker's in-memory droppingTable) is still caught before a source-cache write lands. This is the cross-worker race the PR description says it fixes ("Prevents cross-worker source-cache writes from racing a table drop"). Every existing test (dropTableGhost.test.js, Resource-get-context.test.js) only exercises the droppingTable leg, by calling dropTable() in the same worker/process before triggering the write. None sets the tombstone directly (as several ghost tests already do, e.g. line 90-92) and then drives a getFromSource-style write through the commit callback (line 5984) or the pre-stage check (line 6109) to confirm it aborts.
Why it matters: This is the exact "production" leg of a new two-branch guard, and it's the one the in-process tests structurally cannot reach (calling dropTable() always also flips droppingTable true locally). Without a test for the persisted-only path, a regression here (e.g. wrong key, wrong table variable, or the check being dropped in a future refactor) would silently reintroduce the cross-worker write-after-drop race this PR sets out to close, undetected by the suite.
Suggested fix: Add a test that, without calling dropTable(), writes { dropping: true } onto the table's primary catalog row (mirroring the existing ghost tests' tombstone-injection pattern) and then drives a source-populated write (via sourcedFrom + get(), as Resource-get-context.test.js already does) to assert the cache write is skipped rather than staged.

[claude]

Suggestion (non-blocking): This new orphaned-column-family sweep opens each column with new RocksDatabase(rootStore.path, { name: columnName }).open() directly — the only such direct use in this file. Every other RocksDB open in the codebase goes through openRocksDatabase() in resources/databases.ts, which applies WAL/compression settings and the read-only-mode guard (resources/databases.ts:286-341), and resources/databases.ts's own completeInterruptedDrop (which this block closely mirrors, including the new remainingColumnFamilies check) already uses it. Consider exporting openRocksDatabase and reusing it here (or extracting the whole drop-and-verify loop into one helper shared by dropTable() and completeInterruptedDrop()) instead of duplicating the column-sweep logic with a divergent open path.

[claude]

Reviewed. One blocker: the new cross-worker leg of tableIsDropping() (Table.ts:406) — the core mechanism this PR adds — has no test exercising the persisted-tombstone-only path (see inline comment). One non-blocking suggestion posted inline on the column-family sweep's open path.

[kriszyp]

Superseded by draft #2206, “Quiesce workers before dropping RocksDB tables”.

The captured #1381 timeline shows an HTTP worker committing through a retained table column-family handle after another worker begins the table drop. That is distinct from rocksdb-js #787, “Serialize database destruction with concurrent opens”, which protects whole-database destroy/open lifecycle. #2206 handles the demonstrated Harper race directly with a pre-destructive worker barrier, targeted drain/cancellation, handle closure, and generation-guarded cleanup; it does not depend on #787.

Closing this draft avoids two competing Harper-side designs for the same symptom while preserving its investigation history.

— GPT-5 Codex