remove() blocks on drain_rotations — consider non-blocking alternative

[titusz]

Problem

ShardedIndex.remove() calls drain_rotations() unconditionally at the top of the method. This blocks the calling thread until all pending background shard rotations complete (disk serialization), which can take 1-2 seconds for large shards.

This creates a performance bottleneck in iscc-search's add_assets() flow, which calls remove() before add() for update semantics. Even when the remove is necessary (updated assets), the blocking drain serializes the entire ingestion pipeline.

Impact measured in iscc-search

With 512 MB default shard size, we observed:

• 6–15 second stalls during bulk ingestion (pre-fix)
• Group throughput dropping from 44 to 13 files/sec as shards approach rotation threshold
• The stalls are caused by drain_rotations() inside remove(), not by the remove operation itself

Why remove() needs cross-shard visibility

remove() must find keys across all shards to tombstone them. If a rotation is pending, the frozen shard is not yet registered as a view shard, so remove() would miss keys in that shard without draining first. This is a correctness requirement.

Proposed alternative

Instead of blocking on drain_rotations(), remove() could search the pending rotation shard objects directly. They are still alive in memory (referenced in _pending_rotations), so they can be searched without waiting for the background thread to finish writing to disk.

Rough sketch:

def remove(self, keys, *, compact=False):
    self._check_writable()
    self._register_completed_rotations()  # non-blocking: register finished ones
    # Also search _pending_rotations shards directly for keys
    ...

This would give the same correctness guarantees without blocking on disk I/O.

Workaround in iscc-search

We've worked around this on our side by tracking which assets are updates vs creates and only calling remove() for actual updates. This eliminates the drain for pure-insert workloads, but the blocking drain still affects update-heavy workloads.