Llm governance middleware foundation

[Divyam19]

Description

Adds an alpha governance middleware layer for Ray Serve LLM so users can inspect or block OpenAI-compatible chat, completions, and transcription traffic without changing vLLM or SGLang.

GovernanceIngress subclasses OpenAiIngress and runs an ordered LLMMiddleware chain around _process_llm_request:

• before_inference — required. Return the request to proceed, or a BlockedResponse to stop before the model is called.
• after_inference — optional. Sees the full non-streaming response, or the first stream chunk.
• on_inference_complete — optional. Runs after the response is returned, including after a stream ends, with the last usage dict.

Users wire it through the existing ingress_cls_config / ingress_extra_kwargs on build_openai_app. The public API is ray.serve.llm.governance (GovernanceIngress, LLMMiddleware, RequestContext, BlockedResponse).

BlockedResponse maps to OpenAI-style JSON errors: PII_DETECTED → 400, BUDGET_EXCEEDED → 402, ACCESS_DENIED → 403, THROTTLED → 429 with Retry-After.

This is the Ray-side hook interface for #65259. Concrete engines (TealTiger, AgentShield, custom policies) live in separate packages and subclass LLMMiddleware.

Out of scope for this PR: stream buffering / redaction, per-model middleware on LLMConfig, embeddings/scoring hooks, and direct streaming (RAY_SERVE_LLM_ENABLE_DIRECT_STREAMING already rejects a custom ingress).

Related issues

Related to #65259.

Additional information

Usage

from ray.serve.llm import LLMConfig, build_openai_app
from ray.serve.llm.governance import (
    BlockedResponse,
    GovernanceIngress,
    LLMMiddleware,
)

class DenyModelMiddleware(LLMMiddleware):
    async def before_inference(self, request, context):
        if context.model_id == "blocked-model":
            return BlockedResponse(
                rule_triggered="ACCESS_DENIED",
                reason="Model is not allowed",
            )
        return request

app = build_openai_app(
    {
        "llm_configs": [llm_config],
        "ingress_cls_config": {
            "ingress_cls": GovernanceIngress,
            "ingress_extra_kwargs": {
                "middlewares": [DenyModelMiddleware()],
            },
        },
    }
)

Docs: doc/source/serve/llm/user-guides/governance.md.

Tests

python -m pytest python/ray/llm/tests/serve/cpu/governance/ -v

Coverage: middleware chain, request-context extraction, reference PII/budget middleware, mocked-handle ingress (block before LLM, 429, stream completion, aclose on block/disconnect), build_openai_app wiring, and HTTP e2e through serve.run + httpx + MockVLLMEngine.

User Guide

User guide: doc/source/serve/llm/user-guides/governance.md (also linked from the Serve LLM user-guides index and the API reference).

[gemini-code-assist]

Code Review

This pull request introduces a new governance middleware framework for Ray Serve LLM, enabling users to inspect, modify, or block OpenAI-compatible traffic via custom LLMMiddleware hooks on GovernanceIngress. Key additions include request context tracking, blocked response handling, reference middlewares for PII detection and budget enforcement, and comprehensive tests. Feedback on the changes highlights opportunities to improve robustness, specifically by supporting Pydantic v1 models in usage_to_dict and the non-streaming response path to prevent potential crashes or silent tracking failures, and extending extract_request_text to support dictionary-based requests consistently.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 7f8c0d7. Configure here.}