Skip to content

Cross-worker write can race RocksDB table drop and poison catalog cleanup #1381

Description

@heskew

Summary

Intermittently, a block of resources unit tests fail in their before all / after all hooks because resetDatabases() (called from setupTestDBPath) hits a column-family error while completing an interrupted table drop left over from a prior test. One run produced ~12 cascading hook failures from this single root cause. Re-running the job goes green (verified: attempt 2 passed on the same commit), so it's a test-isolation / state-leak flake.

Exact errors

[main/0] [error]: Error committing cache update [Error: Transaction commit failed: Invalid argument: Invalid column family specified in write batch] { code: 'ERR_INVALID_ARGUMENT' }
[main/0] [error] [storage]: Failed to complete interrupted drop of table <db>.TestTableForContext; will retry on next start Error: Remove failed: Invalid argument: Invalid column family specified in write batch

Stack:

Store.removeSync (@harperfast/rocksdb-js/src/store.ts:824)
RocksDatabase.removeSync (@harperfast/rocksdb-js/src/dbi.ts:461)
completeInterruptedDrop (resources/databases.ts:1638)
initStores (resources/databases.ts:541)
readRocksMetaDb (resources/databases.ts:439)
getDatabases (resources/databases.ts:331)
resetDatabases (resources/databases.ts:747)
setupTestDBPath (unitTests/testUtils.js:386)

Resulting cascade (sample): "after all" hook for "should handle source.get returning null", "before all" hook for "It can search_by_conditions on a resource", plus indexing/subscription/txn before all hooks.

Why it's a flake

A prior test leaves a TestTableForContext drop half-completed (tombstoned) in the rocks meta DB; the next test's setupTestDBPath → resetDatabases → readRocksMetaDb → initStores → completeInterruptedDrop tries to finish it and the column family is already gone/inconsistent, so removeSync throws. Order/timing-dependent; green on re-run, same commit.

Affected job / runtime

Observed on Unit Test (Node.js v26), but the mechanism (shared-process unit run, leaked drop state) is not runtime-specific.

Evidence

Related


Filed by Claude (Opus 4.8) during CI flake triage while shepherding #1363/#1371/#1374.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:storageStorage engine, LMDB/RocksDB, compactionbugSomething isn't working

Type

Fields

Priority

P2

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions