Skip to content

fix: preserve queue compatibility and classify admission rejects - #50

Merged
DavidBellamy merged 2 commits into
prodfrom
fix/admission-rejection-metrics
Aug 6, 2026
Merged

DavidBellamy merged 2 commits into
prodfrom
fix/admission-rejection-metrics

Conversation

@DavidBellamy

Copy link
Copy Markdown

Description

Problem

Two compatibility issues block a safe SMG artifact refresh:

  1. Adaptive admission rejections use HTTP 429, but gRPC router metrics classify them as internal_error. Under saturation this makes normal backpressure look like a large internal failure storm.
  2. The recently merged fair-share queue implementation defaults an omitted per-tenant queue cap to 64. Existing configurations omit this field, so merely refreshing the SMG image would silently impose a much smaller queue limit.

Closes #48.
Closes #49.

Solution

  • Classify HTTP 429 as the bounded admission_rejected gRPC error label.
  • Make max_queued_requests_per_tenant optional. When omitted, it inherits the partition's existing shared queue budget and therefore preserves prior behavior. Explicit positive limits remain supported and zero remains invalid.

Neither change alters adaptive admission decisions, response status codes, fair-share ordering, or engine behavior.

Changes

  • Add the admission_rejected router metric label and status mapping.
  • Preserve existing 400, 404, timeout, gateway-timeout, and 5xx mappings.
  • Change the tenant queue limit configuration to an opt-in value.
  • Resolve an omitted tenant limit to the scheduler partition's shared queue budget.
  • Document and test omitted, explicit, and invalid queue-limit behavior.

Test Plan

  • cargo +nightly fmt --all -- --check
  • cargo test --offline -p smg --lib --quiet -- --test-threads=1
  • cargo clippy --offline -p smg --lib --no-deps -- -D warnings -A clippy::ignored-unit-patterns
  • cargo check --offline --manifest-path bindings/python/Cargo.toml
  • Focused gRPC status-mapping tests cover 429 plus all preserved mappings.
  • Focused scheduler tests cover omitted, explicit, and zero tenant caps.
Checklist
  • cargo +nightly fmt passes
  • cargo clippy passes with the documented pre-existing ignored-unit-patterns allowance
  • Documentation updated

Signed-off-by: David <12414531+DavidBellamy@users.noreply.github.com>
Signed-off-by: David <12414531+DavidBellamy@users.noreply.github.com>
@DavidBellamy
DavidBellamy marked this pull request as ready for review August 6, 2026 17:05
@DavidBellamy
DavidBellamy requested a review from a team August 6, 2026 17:05
@DavidBellamy
DavidBellamy merged commit 35713f3 into prod Aug 6, 2026
3 of 9 checks passed
@DavidBellamy
DavidBellamy deleted the fix/admission-rejection-metrics branch August 6, 2026 17:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant