Serialize database destruction with concurrent opens

[kriszyp]

Summary

Harper currently works around a rocksdb-js lifecycle race by locking root opens in JavaScript. This moves the invariant into the native registry: physical destruction now owns a database path across read-write/read-only descriptors, concurrent opens wait, and shutdown is serialized with both operations.

The change also makes teardown failures observable and recoverable, and prevents destroy/shutdown from releasing the native database beneath directory backups, streaming backups, or checkpoints.

For the human reviewer

• destroy() intentionally changes from refusing while peer handles exist to closing every in-process handle for the physical path before removing it. This is what allows Harper schema propagation to race safely with a database drop. Cross-process coordination remains RocksDB's lock responsibility.
• A failed native close quarantines the path and emits database:closeFailed. A failed post-destroy directory cleanup is visible in registryStatus() and can only be retried by the explicit destructive verbs, destroy() or shutdown(); open() remains non-destructive.
• Close-time flush/compaction errors are reported after native teardown completes. This can surface an error from close() in a finally, but silently ignoring a failed flush would hide possible data loss.
• Destroy waits for registered in-flight backups/checkpoints before teardown. Streaming backups poll the closing state so their JS backpressure handshake cannot deadlock the synchronous destroy.
• The registry and path gates are process-global. Raw path spellings are not canonicalized, matching the existing registry key behavior.
• Cross-thread handle close can still race an owner-thread close on the same DBHandle; shutdown already exercised that path before this change. N-API reference deletion is now owner-thread-only, and worker lifecycle fixtures pass, but the remaining shared-handle synchronization is a follow-up decision for the storage maintainer.

This is the rocksdb-js root-cause fix for Harper PR #2169, "Prevent job wedges on runtime database opens". Harper's JavaScript .open lock should remain out of the released path once a package containing this change is available.

Verification

• node_modules/.bin/node-gyp build
• node_modules/.bin/tsc --noEmit
• node_modules/.bin/oxlint
• node_modules/.bin/oxfmt --check
• Full JavaScript suite: 768 passed, 1 skipped
• Native suite: 101 passed, 3 expected macOS MADV_COLD skips
• Focused lifecycle suite: 69 passed

Review coverage

• Full and incremental reviews by Claude, Gemini, Cursor Grok, Cursor Composer, and Harper storage-domain adjudication.
• Review-found blockers fixed: shutdown no longer abandons healthy databases; destroy-cleanup tombstones are explicit and recoverable; directory backups participate in teardown accounting; cancelled queued operations release their in-flight claims; close-failure events cover every registry teardown path.
• Human decisions remain around force-destroy semantics, synchronous wait behavior, and the pre-existing concurrent close race described above.

— GPT-5 Codex

_{Human-Review-Need: 4 (decisions: close-throws-vs-reports, quarantine-blocks-reopen, global-shutdown-as-the-retry-api, shutdown-throws-while-tombstone-exists, destroy-forces-foreign-teardown, unbounded-inflight-wait-outside-the-deadline, one-global-lifecycle-timeout, retry-skips-the-inflight-drain, iterator-serialized-by-mutex-not-lifetime-guard, destroy-readonly-flag-crosses-the-js-boundary) @ e9bc308}

[gemini-code-assist]

Code Review

This pull request introduces robust database lifecycle management for RocksDB JS bindings. It implements a timed-wait mechanism (lifecycleWaitSeconds) for open, destroy, and shutdown operations to prevent concurrent lifecycle conflicts. It also introduces a "quarantine" state for database paths when a native close, flush, compaction, or physical directory cleanup fails, preventing subsequent opens until the cleanup is retried via destroy() or shutdown(). Additionally, it ensures that in-flight operations (like backups and checkpoints) are safely awaited before destruction, and that thread-affine N-API references are cleaned up safely. There are no review comments, so I have no feedback to provide.

[github-actions]

📊 Benchmark Results

get-sync.bench.ts

getSync() > random keys - small key size (100 records)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	24.65K ops/sec	40.57	39.22	616.2	0.115	123,254
🥈 rocksdb	2	10.95K ops/sec	91.29	86.49	31,330.12	1.24	54,773

getSync() > sequential keys - small key size (100 records)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	28.83K ops/sec	34.69	33.48	581.614	0.101	144,145
🥈 rocksdb	2	10.96K ops/sec	91.24	88.49	2,929.388	0.124	54,802

ranges.bench.ts

getRange() > small range (100 records, 50 range)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 lmdb	1	26.20K ops/sec	38.17	35.46	2,509.233	0.290	131,003
🥈 rocksdb	2	15.64K ops/sec	63.92	56.31	1,063.846	0.124	78,220

realistic-load.bench.ts

Realistic write load with workers > write variable records with transaction log

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	375.47 ops/sec	2,663.335	90.59	69,982.207	19.85	759
🥈 lmdb	2	25.95 ops/sec	38,531.476	363.193	1,220,596.403	136.479	64.00

transaction-log.bench.ts

Transaction log > read 100 iterators while write log with 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	39.93K ops/sec	25.04	10.67	13,681.903	0.592	199,675
🥈 lmdb	2	439.99 ops/sec	2,272.777	227.646	21,946.699	1.43	2,200

Transaction log > read one entry from random position from log with 1000 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	755.27K ops/sec	1.32	1.13	4,875.267	0.205	3,776,355
🥈 lmdb	2	436.44K ops/sec	2.29	1.22	5,862.241	0.717	2,182,177

worker-put-sync.bench.ts

putSync() > random keys - small key size (100 records, 10 workers)

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	834.17 ops/sec	1,198.795	1,027.59	2,987.324	0.442	1,669
🥈 lmdb	2	1.12 ops/sec	889,448.582	816,589.515	994,597.711	4.45	10.00

worker-transaction-log.bench.ts

Transaction log with workers > write log with 100 byte records

Implementation	Rank	Operations/sec	Mean (ms)	Min (ms)	Max (ms)	RME (%)	Samples
🥇 rocksdb	1	22.37K ops/sec	44.70	30.37	461.572	0.565	44,742
🥈 lmdb	2	822.27 ops/sec	1,216.139	165.957	13,488.481	5.44	1,646

---

Results from commit c68d8a2

[cb1kenobi]

Medium: a failed destroy cleanup now poisons shutdown() permanently

This round removed Shutdown()'s destroy-cleanup retry loop, so a tombstone entry (descriptor == nullptr with a non-empty closeError) is no longer repaired here — it is only reported, by throwing. Nothing else in the registry ever clears one: PurgeAll() skips every entry with a closeError (line 682), OpenDB() refuses the path (line 460), and DestroyDB() is the sole remover. So once a single destroy() fails its physical cleanup, every later shutdown() call re-scans the map, finds the tombstone, and throws — forever, even after the underlying cause is gone.

test/destroy.test.ts:162-166 pins exactly that: it restores the directory permissions with chmodSync(lockedDirectory, 0o700) first and still asserts shutdown() throws. The cleanup is now recoverable only through destroy() on that specific path, which needs a previously-opened writable instance the caller may no longer hold.

Two consequences beyond the throw itself:

• The README recommends shutdown() from a process exit listener. That listener now throws on every exit for the rest of the process's life.
• binding.cpp's Shutdown() returns on the throw before GlobalEvents::Shutdown() (already flagged separately, still open at this SHA), so an unremovable directory also leaks every global listener threadsafe function at exit — and PurgeAll() at the end of DBRegistry::Shutdown() never runs either.

Suggested fix: keep the report, drop the throw. The tombstone is already visible through registryStatus().destroyCleanupPending and was already emitted as database:closeFailed when the destroy failed, so shutdown() can surface it without aborting its own flush + PurgeAll() + GlobalEvents::Shutdown(). If a hard signal is wanted, restore the destroyPhysicalPath() retry that lived here at 6bba0884 and throw only when the retry itself fails.

Minor, same function: the destroyCleanupError.empty() term in the loop-exit condition at line 1012 is dead — line 1008 already threw on every path that could make it non-empty.

—
Generated by Barber AI

[cb1kenobi]

Low: compactSync() now holds an OperationGuard across an unbounded compaction, stalling an untimed teardown drain

Adding the guard here is the right call — it is what stops finishClose() from resetting columnDescriptor/db underneath the compaction. But DBDescriptor::finishClose() drains with a bare, untimed atomic wait (db_descriptor.cpp:305-307: while ((current = this->operationsInFlight.load()) != 0) this->operationsInFlight.wait(current);), and CompactSync never checks isClosing() once it starts. A manual compaction of a large column family therefore blocks destroy()/shutdown() for its full duration with no deadline.

That matters because the blocked thread is holding the new path gate: destroyingPaths / shutdownInProgress stay set, so every concurrent OpenDB() for the path waits on lifecycleCondition against its own getLifecycleWaitSeconds() deadline and fails with Timed out opening database ... destruction is still in progress. One slow compactSync() turns into cascading open failures across the process — the wedge class this PR set out to remove.

The other entry points that gained the guard in this round are all short; CompactSync and GetCount are the two that can run arbitrarily long.

Suggested fix: make the compaction cancellable so the guard is released promptly — set rocksdb::CompactRangeOptions::canceled from the closing path (or poll descriptor->isClosing() between column families) so a teardown can interrupt it, rather than leaving the drain unbounded.

—
Generated by Barber AI

[cb1kenobi]

Medium: Return()/Throw() now throw when the iterator is already closed, which is exactly the forced-teardown state this commit creates

closeIfOpen() returns false when iterator == nullptr, and both Return (here) and Throw (line 390) turn that into a thrown Iterator not initialized. Before this commit close() was idempotent and neither method could throw.

The iterator reaches iterator == nullptr through the very path this commit adds serialization for: finishClose()'s closables sweep (db_descriptor.cpp:393-406) calls DBIteratorHandle::close() on the destroying thread. After that, on the owner thread:

• for (const x of store.getRange(...)) { ... break; } — the break runs IteratorClose → DBIterator.return() (src/dbi-iterator.ts:92) → this throw. A clean loop exit becomes an exception.
• DBIterator.throw() (src/dbi-iterator.ts:97) calls this.iterator.throw?.(err) before throw err, so the native throw replaces the caller's real error with Throw failed: Iterator not initialized.

It is also reachable without any concurrency: src/dbi-iterator.ts:56 already calls this.iterator.return?.() when #limit is reached, so any later explicit .return() (legal on a done iterator, and a common finally pattern) is a second call.

There is no upside to the signal — dbi-iterator.ts discards the return value in both places, so the bool only ever materializes as a spurious error. Next is the only method that genuinely needs the null check, and it already has one under the mutex (line 283).

Suggested fix: revert Return/Throw to the idempotent close and keep closeIfOpen() for internal callers that care:

Suggested change

	if (!(*itHandle)->closeIfOpen()) {
	(*itHandle)->close();

(and the same at line 390).

—
Generated by Barber AI

[cb1kenobi]

Low: the VT fast-path exception is overstated — those paths still dereference teardown-owned state

verifyVersion / populateVersion (database.cpp:1294, :1330) both still start with UNWRAP_DB_HANDLE_AND_OPEN(), which reads descriptor, descriptor->isClosing() and — via opened() — the descriptor object itself. What the cached verificationTableDbId / verificationTableColumnFamilyId actually removed is the getColumnFamilyHandle() dereference and the OperationGuard, not the descriptor access. A future agent reading "do not touch teardown-owned native state" could reasonably add a new VT fast path with no descriptor gate at all.

The same paragraph is also now the only statement of an invariant that the code applies inconsistently: database.cpp:1815 (PutSync), database.cpp:1884 (RemoveSync) and transaction_handle.cpp:211 still compute the VT address as descriptor->vtEpoch + getColumnFamilyHandle()->GetID() rather than reading the new cached fields, so there are two spellings of the same address computation that must stay in agreement.

Suggested fix: narrow the claim to what is true — the fast paths avoid the column-family dereference and the in-flight registration, but still gate on UNWRAP_DB_HANDLE_AND_OPEN() — and either migrate the three remaining sites to dbHandle->verificationTableDbId / verificationTableColumnFamilyId or note why they keep the descriptor form.

—
Generated by Barber AI

[cb1kenobi]

Low: a teardown failure now leaks the benchmark database directory with no diagnostic

This round moved the rmSync from the worker (workerInit, which ran it unconditionally) to the parent, and gated it on !teardownError. Since the worker no longer deletes anything, a benchmark whose worker fails teardown leaves benchmark/data/rocksdb-benchmark-<random>/ behind permanently — and dbPath is freshly randomized per run (line 385), so repeated failures accumulate directories rather than reusing one.

The thrown teardownError says nothing about the retained path, so the leak is invisible until someone looks at benchmark/data/.

If retaining the data for post-mortem inspection is the intent, that is reasonable — but it should say so. Suggested fix: log the retained path before rethrowing, e.g. console.warn(\Benchmark teardown failed; retaining ${dbPath} for inspection`)in theif (teardownError)branch, or delete it anyway since a failedclose()` already left the directory in an undefined state.

—
Generated by Barber AI

[cb1kenobi]

Low: this test covers the constructor gate, not the per-iterator mutex that is the substance of the commit

The timing here is deterministic and does exercise something real: the worker posts destroying and waits 50ms (destroy-open-worker.mts:12) while the parent sits in the 250ms ROCKSDB_JS_ITERATOR_SETUP_DELAY_MS sleep, so db_iterator.cpp:154's isClosing() check reliably fires. That half is covered.

What is not covered is the other half — iteratorMutex serializing DBIterator::Next against DBIteratorHandle::closeIfOpen() running from finishClose()'s closables sweep. The only delay seam is in the constructor, so no test can position a foreign forced close during a Next() call. If the mutex were removed the suite would still pass.

Separately, the fixture's assertion is broad: test/fixtures/fork-destroy-open.mts:34-44 accepts success, Database not open, and Database is closing alike, so it is a crash guard rather than a behavior assertion.

Suggested fix: add a testDelayMs("ROCKSDB_JS_ITERATOR_NEXT_DELAY_MS") seam inside DBIterator::Next after the lock is taken, and a fixture variant that starts a getRange walk, then triggers the destroy mid-next() — that is the case the mutex exists for.

—
Generated by Barber AI