fix: include _internal/mcp in observability tracing paths

[ecthelion77]

🐛 Bug-fix PR

🔗 Issue

Closes #4152

📌 Summary

The _internal/mcp endpoint path is not included in the observability tracing include patterns, causing internal MCP calls between gateway components to be invisible in distributed traces. This makes it impossible to trace the full request flow when meta-servers or virtual servers invoke internal MCP endpoints.

🔁 Reproduction Steps

1. Configure a meta-server that calls internal MCP endpoints via /_internal/mcp/
2. Send a request that triggers the meta-server
3. Check traces in your observability backend (Jaeger/Tempo)
4. Notice that the /_internal/mcp leg is missing from the trace

🐞 Root Cause

The OTEL_TRACE_INCLUDE_PATTERNS default in config.py lists explicit path patterns for tracing (/servers/*/sse, /servers/*/message, /a2a), but /_internal/mcp was omitted.

💡 Fix Description

Add r"^/_internal/mcp(?:/|$)" to the default OTEL_TRACE_INCLUDE_PATTERNS list in Settings, so internal MCP calls are traced by default.

🧪 Verification

Check	Command	Status
Lint suite	make lint	✅
Unit tests	make test	✅
Coverage ≥ 80 %	make coverage	✅
Manual regression no longer fails	Verified traces include /_internal/mcp spans	✅

📐 MCP Compliance (if relevant)

• Matches current MCP spec
• No breaking change to MCP clients

✅ Checklist

• Code formatted (make black isort pre-commit)
• No secrets/credentials committed

[ecthelion77]

Suggested labels: bug, COULD, python, observability

[jonpspri]

Thank you for this fix, @ecthelion77 — the diagnosis in #4152 is accurate and the instinct to add /_internal/mcp to the tracing include-list is correct. However, two things have changed since this PR was opened in April that mean it can no longer be merged as-is.

---

1 — The fix is already in main via a different mechanism

The architecture for path-based trace gating has moved. Since this PR was opened, mcpgateway/observability.py has grown a dedicated _should_trace_request_path() predicate (used by OpenTelemetryRequestMiddleware) that unconditionally enables tracing for /_internal/mcp in code:

# mcpgateway/observability.py  (current main, ~line 816)
if normalized.startswith("/_internal/mcp/"):
    return True

That guard was added as part of the affinity-path OTEL work (PR #6164) and is present on main today. The otel_trace_include_patterns config list is no longer the single gate for this path, so adding the regex there would have no effect on the runtime behaviour described in #4152.

2 — Collateral deletion of unrelated security fields

The diff also removes three fields from Settings that are unrelated to tracing:

-    meta_max_keys: int = Field(default=16, ...)   # CWE-400 guard
-    meta_max_depth: int = Field(default=2, ...)   # CWE-400 guard
-    meta_max_bytes: int = Field(default=4096, ...) # CWE-400 guard

These fields are still present and active on main — they cap user-supplied meta_data forwarded to upstream MCP servers to guard against unbounded input (CWE-400). Merging this PR would silently drop them, removing a security control. This looks like a rebase conflict resolution that went the wrong way.

---

Closing this PR

Given that:

• the bug described in #4152 is already fixed on main,
• the PR carries a regression against active security fields, and
• the otel_trace_include_patterns config approach no longer matches the current tracing architecture,

I'm going to close this PR. Issue #4152 will be closed separately once verified against current main.

If there is a separate desire to also expose /_internal/mcp via otel_trace_include_patterns so operators can override it — that would be a new, narrowly-scoped PR against the current code. But the runtime problem from #4152 is already resolved.

Thank you again for the contribution — the investigation and direction were correct, the implementation just got overtaken by subsequent work.

[jonpspri]

Closing — see review comment above for the full rationale. The runtime fix for #4152 is already on main; this PR cannot be merged without reintroducing the collateral security-field deletions.