[REG-15] W0.10 - Collector deployment, sampling policy, and cost controls

[singret]

Closes REG-15

Summary

A place for traces to actually go, and a sampling policy that keeps the bill and the storage bounded.

What changed

• deploy/otel-collector/otel-collector-config.yaml (new): the collector pipeline — redaction/pii (defense-in-depth allow-list + email-pattern masking) → tail_sampling (keep 100% of errors, incident-creating, and paging traces; probabilistic 2% of the rest) → batch → export to Tempo.
• deploy/otel-collector/tempo.yaml (new): single-binary Tempo config, 7-day retention default. Pinned to grafana/tempo:2.6.1, not :latest — verified directly that :latest resolves to Tempo v3.0.0, whose config schema has moved since the docs/examples this was written against (see below).
• docker-compose.dev.yml: otel-collector + tempo added under a new opt-in observability profile — verified via docker compose config --services that neither appears without --profile observability.
• Helm chart: new tracing.* values block + templates/otel-collector.yaml (ConfigMap + Deployment + Service, gated by tracing.collector.enabled, off by default — verified via helm template). Deliberately plain resources in this chart rather than the full upstream open-telemetry-collector chart; the values.yaml comment explains when to reach for that one instead.
• OTEL_TRACES_SAMPLER/OTEL_TRACES_SAMPLER_ARG: no code change needed — confirmed via the SDK's own source that NewTracerProvider already applies env-based sampler config before any explicit option, and InitTracer never overrides it. Added 3 tests proving this rather than trusting the SDK's internals on faith.
• incident.id / incident.created span attributes (new internal/observability/incident_span.go), wired into the incident HTTP handlers and the alert-triggered creation path. Without this the runbook's own "find a trace from an incident ID" claim would have been false for most incident traces.
• notification.paged span attribute on SendEscalationDM (its own root span — no ctx to thread one through, per REG-13's documented reasoning) — the signal the tail-sampling policy needs to keep 100% of paging traces.
• docs/OBSERVABILITY.md (new): Tempo vs SigNoz recommendation, the sampling policy and why tail-based sampling can only happen at a collector, retention rationale tied to W13's DSAR work, and the runbook for finding a trace from an incident ID or a customer complaint.

Bugs found and fixed via a real test-isolation bug + two live-verification failures

1. A real, pre-existing test-isolation bug, surfaced by the new sampler tests: clearOTelEnv's cleanup only restored env vars that existed before a test — it never unset ones the test itself set. OTEL_TRACES_SAMPLER=traceidratio from one test leaked into every later test in the binary, silently breaking worker_test.go's unrelated assertions. Fixed the helper.
2. Tempo :latest (v3.0.0) rejected the config's compactor section entirely — a schema change from the docs this was written against. Confirmed by running -config.verify=true directly against candidate versions; pinned to 2.6.1.
3. The tail-sampling policies for incident.created/notification.paged used type: string_attribute against real OTel bool attributes — which never matches a bool-typed value (string_attribute only reads the OTLP string_value field). Confirmed live: traces carrying these attributes were being dropped by sample-the-rest instead of kept at 100%. Fixed by switching to type: boolean_attribute.

Acceptance criteria

• OTel Collector added to docker-compose.dev.yml (opt-in profile) and the Helm chart (optional, gated resources)
• Backend recommendation documented (Tempo, with SigNoz as the alternative and why)
• Tail-based sampling: errors, incident-creating, and paging traces at 100%; 1–5% (default 2%) of the rest
• OTEL_TRACES_SAMPLER/OTEL_TRACES_SAMPLER_ARG configurable — already worked, now tested and documented
• Collector applies an attribute-scrubbing processor (allow-list + regex value masking)
• Retention documented, defaulted to 7 days, W13 link noted
• Runbook: finding a trace from an incident ID, and from a customer complaint

Verification

• go build ./..., go vet ./..., gofmt -l clean on all touched files
• Full test suite green, plus -shuffle=on on all touched packages (multiple runs, confirming the env-leak fix holds)
• golangci-lint run ./...: 0 issues
• docker compose config validates the compose file and confirms profile gating
• helm lint / helm template validate the chart, confirm the collector resources are absent by default and correct when enabled
• Live-verified against a real otel-collector + Tempo: fired an error trace, an incident.created trace, a notification.paged trace, a trace with an attribute outside the redaction allow-list, and a trace with an email address inside an allowed key — all five behaved exactly as configured (kept at 100%, allow-list attribute stripped, email masked to asterisks) — plus 2000 "boring" control traces, of which ~2.5% survived the probabilistic policy, consistent with the configured 2%

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
fluidify	🟢 Ready	View Preview	Sep 1, 2026, 2:37 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.