Skip to content

[Bugfix] Propagate shared cache load failures - #1158

Draft
dante159753 wants to merge 1 commit into
ModelEngine-Group:developfrom
dante159753:cache-shared-load-error
Draft

[Bugfix] Propagate shared cache load failures#1158
dante159753 wants to merge 1 commit into
ModelEngine-Group:developfrom
dante159753:cache-shared-load-error

Conversation

@dante159753

Copy link
Copy Markdown
Contributor

Purpose

Prevent shared Cache buffers from being reused while an asynchronous backend still accesses them, and propagate backend load failures to every rank waiting on the same shared entry.

Modifications

  • replace the shared cache ready bit with LOADING/READY/FAILED state and retain the backend failure status
  • keep a cache buffer reference in backend task shards through Posix AIO/psync and Mooncake task lifetimes
  • make non-owner loads consume the shared success or failure result with a bounded wait
  • drain submitted backend tasks before releasing cache task state
  • add unit coverage for cross-mapping failure propagation, retry after references are released, backend lifetime retention, timeout, and cleanup paths

User impact

When one rank owns a shared Cache load and the backend fails or times out, peer ranks now observe the same failure instead of waiting indefinitely or treating incomplete KV data as ready. Backend IO also cannot outlive and silently overwrite a reused cache buffer.

Verification

  • git diff --check
  • clang-format 20 check on changed lines
  • codespell 2.4.1 on changed files
  • A2, latest origin/develop: Debug/Ascend build passed; ucmstore.test 89/89 passed
  • A2 ucmshared.test: 35/37 in the full run; the two logger ordering tests passed 2/2 when rerun together in isolation
  • A2 DeepSeek-V2-Lite, TP2, Cache|Posix, AIO, O_DIRECT, layerwise, recompute: after truncating one Posix KV file and injecting delayed reads, only the owning rank issued backend loads, the peer consumed the shared failure, recovery completed, and generated token IDs matched the baseline

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant