Skip to content

Support request-level asynchronous KV cache loading - #1246

Open
lc1314555 wants to merge 9 commits into
ModelEngine-Group:developfrom
lc1314555:develop_lc_request_async
Open

Support request-level asynchronous KV cache loading#1246
lc1314555 wants to merge 9 commits into
ModelEngine-Group:developfrom
lc1314555:develop_lc_request_async

Conversation

@lc1314555

@lc1314555 lc1314555 commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Purpose

Add request-level asynchronous KV cache loading for vLLM UCMDirectConnector.
While one request is waiting for external KV data, vLLM can process other runnable requests and resume the waiting request after loading completes, reducing head-of-line blocking.

Modifications

  • Add the use_request_async configuration.
  • Record the UCM-to-vLLM block mapping after GPU block allocation.
  • Submit KV load tasks without blocking model execution.
  • Poll load completion through get_finished().
  • Prevent duplicate loads after rescheduling or preemption.
  • Report load failures and invalidate affected GPU blocks.
  • Add unit tests for dispatch, completion, duplicate prevention, and failure handling.
  • Currently support only UCMDirectConnector with a single KV cache group, PP=1, and without CP or layerwise loading.

Comment thread ucm/integration/vllm/ucm_connector.py Outdated
self._async_dump_req_ids: set[str] = set()
self._pending_dump_tasks: list[PendingDumpTask] = []
request_async_configured = bool(
self.launch_config.get("use_request_async", False)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

建议修改为把参数值修改为use_request_async_load

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants