Skip to content

fix: bound Nydus cache backfill under ChunkDB contention - #6

Merged
rcyw merged 2 commits into
inclusionAI:mainfrom
iamWHTWD:distill-fs-cache-backpressure
Nov 12, 2026
Merged

rcyw merged 2 commits into
inclusionAI:mainfrom
iamWHTWD:distill-fs-cache-backpressure

Conversation

@iamWHTWD

Copy link
Copy Markdown
Contributor

Problem

A Nydus mount could queue up to 4,096 decompressed chunks while its background writer waited for the shared ChunkDB LMDB transaction. At 1 MiB per chunk, that is about 4 GiB of queued payload per mount. If the map filled, backfill also ran synchronous GC and retried, adding pressure to the same writer lock.

Changes

  • Bound optional decompressed-chunk backfill to 1 GiB per shared ChunkDB on Linux (64 OFD-locked slots of 16 MiB). Charge buffer capacity through queueing, lock wait, and writing. Release the reservation with the buffer or on process exit. Skip optional backfill when the budget is unavailable and wait 50 ms before rescanning unavailable slots.
  • Discard expired backfill before writing. Check the 2 s deadline again after acquiring the LMDB write lock. Waiting for that lock still counts against the budget.
  • Abort the failing backfill transaction on MapFull, preserve previously committed sub-batches, and pause that mount's backfill worker for 5 s. The public ChunkDB::add_chunks path keeps its synchronous GC and retry behavior.

ECS validation

Linux x86_64 with Rust 1.85.0. cargo fmt --all -- --check passed. cargo test --locked passed 138 tests, and 3 Nydus/FUSE tests passed separately. Regression coverage includes MapFull rollback and cooldown, expiry, cross-process budget exhaustion, and permit recovery after a process is killed.

An isolated release-build stress test ran 72 independent Nydus FUSE mounts against one 512 MiB ChunkDB with per-mount caches, 2.25 GiB of cold reads, real gc-chunk reclamation, and a controlled 20 s LMDB writer lock.

Check Result
Cold reads during the held lock 72/72 completed before release with matching SHA-256. P95 was 3.34 s.
Backfill budget and memory All 64 slots were occupied, giving a 1 GiB payload ceiling. Container anonymous memory peaked at 1.15 GiB and fell to about 138 MiB. No OOM or OOM kills.
Recovery after lock release GC waited 20.09 s and reclaimed 128 chunks. All slots were released. 16 subsequent cold reads passed and ChunkDB entries increased from 419 to 459.

The limit covers pending decompressed backfill payloads. The ECS workload used local RAFS blobs and injected lock contention. Production OSS/Dragonfly traffic and prolonged I/O stalls were outside the test scope.

Cap queued and in-flight decompressed chunks across mounts sharing a
ChunkDB. Drop expired backfill and pause the worker after MapFull while
preserving synchronous GC and retries for explicit batch writes.

Cover cross-process reservations, MapFull, and Nydus read and backfill
behavior with regression tests.

rcyw commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator

Reviewed fd5e69922a5a7d050b2e2df6e5fcf5ffd53cfe13. No blocking findings.

Checked the cross-process budget, payload/permit drop ordering, MapFull rollback and cooldown, preservation of committed sub-batches, and deadline checks around LMDB writer-lock acquisition.

Local verification on Linux x86_64 with Rust 1.85.1:

  • Formatting check passed.
  • cargo +1.85.1 test --locked passed all 138 default tests.
  • An additional temporary regression passed: hold the LMDB write lock, enqueue backfill with a future deadline, let it expire while waiting, then release the lock. The reservation remains charged during the wait; afterward the request returns TimedOut, writes no chunk, and releases its budget.

Non-blocking suggestion: consider keeping that lock-wait expiry scenario as a permanent regression test. The existing expired_cache_backfill_releases_its_reservation test starts with an already-expired deadline, so it does not exercise the second deadline check after lock acquisition.

The three ignored Nydus/FUSE tests and the ECS stress workload described in the PR were not rerun in this review.

Hold the LMDB writer lock past the batch deadline. Verify the writer
returns TimedOut after acquiring the lock, leaves the cache unchanged,
and releases the reservation.
@rcyw
rcyw merged commit edbe147 into inclusionAI:main Nov 12, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants