feat(observability): auto-instrument httpx for outbound trace propagation

[ecthelion77]

✨ Feature / Enhancement PR

🔗 Epic / Issue

Closes #4155

Improves observability for outbound HTTP calls made by the gateway to upstream MCP servers and OAuth providers.

---

🚀 Summary

This PR adds automatic OpenTelemetry instrumentation for httpx, the HTTP client used by the gateway for all outbound requests (MCP server communication, OAuth token exchanges, health checks). Without this, outbound HTTP calls are invisible in distributed traces — the trace stops at the gateway and does not propagate traceparent headers to upstream services.

Changes:

1. mcpgateway/observability.py: Adds opentelemetry-instrumentation-httpx initialization in setup_observability(), calling HTTPXClientInstrumentor().instrument() to auto-instrument all httpx.AsyncClient and httpx.Client instances.

2. gunicorn.config.py: Re-applies httpx instrumentation in post_fork(). Gunicorn's pre-fork model requires re-initialization of OTel instrumentors in each worker process. The implementation calls uninstrument() first to avoid double-instrumentation warnings, then instrument().

3. pyproject.toml: Adds opentelemetry-instrumentation-httpx as a dependency.

---

🧪 Checks

• make lint passes
• make test passes
• CHANGELOG updated (if user-facing)

---

📓 Notes

Why uninstrument() before instrument() in post_fork?

Gunicorn forks workers from the master process. If the master already instrumented httpx, the forked worker inherits the instrumented state. Calling instrument() again would raise warnings. The safe pattern is uninstrument() + instrument() to ensure clean re-initialization in each worker.

[ecthelion77]

Suggested labels: enhancement, COULD, python, observability

[jonpspri]

Thank you for this, @ecthelion77 — auto-instrumenting httpx for outbound trace propagation is genuinely the right thing to do, and the approach here (calling HTTPXClientInstrumentor().instrument() in setup_observability() and re-applying in gunicorn.config.py's post_fork()) is sound.

However, this work has been superseded by PR #6164 (affinity-path OTEL spans and concurrent forward dispatch), which covers the same capability in a more complete and opt-in form.

What PR #6164 does instead

mcpgateway/config.py on the jps-affinity-concurrency branch (and in the PR #6164 diff) adds three explicit, operator-controllable settings:

otel_httpx_instrumentation_enabled: bool = Field(
    default=False,
    description="Auto-instrument outbound httpx clients (HTTP client spans + W3C trace header injection)"
)
otel_sqlalchemy_instrumentation_enabled: bool = Field(
    default=False,
    description="Emit OTel spans for SQLAlchemy queries (high span volume; enable for debugging)"
)
otel_redis_instrumentation_enabled: bool = Field(
    default=False,
    description="Auto-instrument redis/redis.asyncio clients (per-command CLIENT spans)"
)

And mcpgateway/observability.py wires them up in setup_observability() with graceful fallback when the optional packages are not installed. It also supports both HTTPXClientInstrumentor and HTTPX2ClientInstrumentor, and handles the Gunicorn pre-fork re-init concern you identified.

Key design differences

This PR (#4148)	PR #6164
Always-on (no config toggle)	Opt-in via OTEL_HTTPX_INSTRUMENTATION_ENABLED
Hard dependency on opentelemetry-instrumentation-httpx	Optional: logs a warning if package absent
httpx only	httpx, httpx2, redis, SQLAlchemy
default=True behaviour	default=False — avoids surprises in existing deployments

The opt-in default is important: httpx instrumentation adds a span per outbound call and injects traceparent headers — both are desirable, but they represent a behaviour change for existing deployments that previously had no outbound spans. Making it opt-in (with clear documentation) is the safer production approach.

Closing this PR

Since PR #6164 subsumes this work in a more complete, opt-in, multi-client form, I'm closing this PR. Issue #4155 will be closed with a pointer to PR #6164.

Again — the problem statement and approach here were correct. The subsequent affinity work just arrived at a more complete solution.

[jonpspri]

Closing — superseded by PR #6164. See review comment above for the full rationale.