[Feat] add extra info into request hash for multimodal/lora/salt support - #1326
Open
harrisonyhq wants to merge 1 commit into
Open
[Feat] add extra info into request hash for multimodal/lora/salt support#1326harrisonyhq wants to merge 1 commit into
harrisonyhq wants to merge 1 commit into
Conversation
Reuse vLLM's generate_block_hash_extra_keys so UCM block hashes include the multimodal feature identifier+position, LoRA identity, request cache salt, and prompt-embedding content. This prevents different images that share identical placeholder-token prefixes from sharing KV (false hit), while keeping the shared text prefix reusable. Unsupported request semantics fail closed (no token-only fallback). One common RequestHasher closure is used across Direct/LayerWise/Mock/Lite/CP/HLA/FAWA; CacheBlend stays plain-text only. Keys remain reproducible across processes without PYTHONHASHSEED. Includes unit tests: shared-prefix hit semantics, multi-image cursor advance, pure-image divergence, edge cases, connector-level fail-closed, and cross-instance reproducibility.
harrisonyhq
requested review from
FangRun2,
Infinite666,
Tarrei,
Wwwzff,
flesher0813,
mag1c-h,
qyh111 and
ygwpz
as code owners
September 4, 2026 02:22
flesher0813
reviewed
Sep 4, 2026
| self.request_hasher = RequestHasher(vllm_config, 0) | ||
| self._other_rank_hashers = self._make_other_rank_hashers(vllm_config) | ||
| self._seed = self.request_hasher("UCM_HASH_SEED") | ||
| self._seed = self.request_hasher.seed |
Contributor
There was a problem hiding this comment.
Why we need save seed here? Can we just use it in request_hasher?
| ) | ||
| try: | ||
| ucm_block_ids = self.request_block_hasher(request) | ||
| except RequestHashError as e: |
Contributor
There was a problem hiding this comment.
RequestHashError only inherits from RuntimeError. Could this cause any other exception? Would it be better to catch Exception directly instead?
|
|
||
| ## 文档状态 | ||
|
|
||
| - 状态：已实施（Implemented） |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Make UCM external KV-cache keys request-semantic-aware so that requests sharing identical placeholder tokens (e.g., two different images expanded to the same multimodal placeholder IDs) can no longer false-hit each other's KV, while the shared text prefix stays reusable. Keys remain reproducible across processes without PYTHONHASHSEED.
Modifications
Add ucm/integration/vllm/request_hasher.py: RequestHasher + a reusable make_request_block_hasher closure that folds vLLM's generate_block_hash_extra_keys (multimodal identifier+position, LoRA identity, cache salt, prompt-embedding content) into the chained block hash; unsupported semantics fail closed via RequestHashError (no token-only fallback).
Switch Direct/LayerWise/Mock/Lite/CP/HLA/FAWA connectors to the common closure; CacheBlend stays plain-text-only (validates and rejects extra semantics).
Keep the existing model/dtype/TP/rank/spec/sparse namespace and physical layout unchanged; the new 3-tuple hash input naturally supersedes old token-only keys (cold-cache rollout).
Add design docs (vllm_request_aware_prefix_hashing_design*.md) and unit tests (test_ucm_request_hasher.py, test_ucm_blend_hash.py, test_ucm_hla_hash.py).
Test
Unit: 28 passed / 1 skipped on Ascend (vllm 0.23.0) — covers shared-prefix hit semantics, multi-image cursor advance, pure-image divergence, edge cases, connector-level fail-closed, cross-instance reproducibility; blend tests skipped where triton is unavailable.
E2E: Qwen3.8-27B-w8a8 (hybrid) + UCM Cache|Posix, TP=2 — anti-false-hit confirmed in both HBM prefix-cache off and on: reqA(imgX) dumps, then reqB(same tokens, different img) hits only the text-prefix block (~1536 tokens) and correctly misses all image blocks (request-aware hash diverges), then dumps its own image blocks. Text/image reuse (E2/E3) also verified.
Lint: format.sh (black/codespell/isort/actionlint) all pass.