Skip to content

[REG-157] Thread ctx through the incident-create path (partial) - #168

Merged
singret merged 1 commit into
mainfrom
reg-157-incident-create-ctx-threading
Sep 1, 2026
Merged

singret merged 1 commit into
mainfrom
reg-157-incident-create-ctx-threading

Conversation

@singret

@singret singret commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Partial REG-157 — the incident-create-path slice, per REG-9's own suggested slice-order. Remaining scope (Slack event handler, post-mortem service, the broader IncidentService CRUD surface) documented precisely in REG-157, not silently dropped.

What

Fixes the actual gap REG-9 found: the gorm/redis tracing plugins were registered correctly, but a live smoke test sending a real Alertmanager webhook produced zero DB spans — only the rate limiter's Redis span — because no repository method ever threaded a context carrying a real span into db.WithContext().

Scope

Repository layer — 8 methods across 3 files: IncidentRepository.{Create,GetByID,Update,LinkAlert}, AlertRepository.{Create,GetByExternalID,Update}, TimelineRepository.Create. The rest of each interface intentionally doesn't take ctx — same mixed pattern user_repository.go's Upsert already established.

Service layer, real ctx threaded (not a placeholder): AlertService.{ProcessAlertmanagerPayload,ProcessNormalizedAlerts,createOrUpdateAlert}, IncidentService.{CreateIncidentFromAlert,CreateIncidentFromAlertWithGrouping,LinkAlertToExistingIncident}, AcknowledgeAlertWithTimeline, coordinator.SeedDemoData, and two functions that already had ctx available via their callers but were discarding it (PostMortemAgent.writeTimelineEntry, TeamsEventHandler.syncMessageToTimeline — both had it as _).

Handlers, all 4 alert-intake entry points: prometheus_webhook.go, webhook_handler.go (the shared Grafana/CloudWatch/generic path — same fix covers all three sources with one signature change), alerts.go, neuri.go, setup.go.

REG-10's WithoutCancel criterion, done for these two functions: both incident-creation paths now capture bgCtx := context.WithoutCancel(ctx) before spawning their Telegram/push goroutines and pass it to recoverAsyncPanic — real trace-linked spans on panic recovery, not context.Background().

Where the line was drawn, and why

Once the 4 shared repository methods took ctx, every caller anywhere in the codebase had to compile against the new signature. Tracing the fan-in: GetIncident/CreateIncident/UpdateIncident/CreateTimelineEntry are called from both real Gin handlers and slack_event_handler.go (14 call sites, no ctx plumbed through that file at all) and post_mortem_service.go. AcknowledgeIncident/ResolveIncident/UpdateIncidentStatus turned out to have zero Gin-handler callers — they're used exclusively by the Slack/Teams bot event handlers.

Threading real ctx through those 7 methods would have forced fixing Slack's entire event-handling surface in this same PR — a structurally different problem (bot callback loop, not request/response) that deserves its own scoped design pass. Left them on their existing signatures, passing context.Background() internally to the newly-ctx-aware repo calls: zero caller changes, compiles clean, and it's an explicit, documented placeholder rather than a silent gap — exactly the same discipline as recoverAsyncPanic's 16 call sites in REG-10.

An incidental fix

LinkAlertToExistingIncident's Slack-notification goroutine had no panic recovery at all before this change — any panic inside BuildAlertLinkedMessage or PostMessage would have crashed the whole server. Added recoverAsyncPanic since I was already touching this exact function.

Testing

  • internal/repository/ctx_propagation_test.go: real span-correlation proofs (not mocked) for all three repositories' Create, using the actual gorm tracing plugin against an in-memory sqlite DB with an explicitly injected TracerProvider (never touching global otel state — the REG-7 finding)
  • go test ./... -shuffle=on -count=3 clean on every touched package
  • golangci-lint run ./... clean (0 issues)
  • Live-verified against a real Jaeger collector: sent the exact same real Alertmanager webhook payload that previously produced 1 span (just the Redis eval). After this change: 5 spans under one trace IDincidents Create, two alerts spans (the dedup GetByExternalID check + Create), the Redis eval, and the root HTTP span. Confirmed db.query.text is empty (redacted by default, per REG-9's own criterion) even with real traffic flowing through instrumented repositories.

…-157)

Repository plugins from REG-9 were correctly registered but never fed a
context carrying a real span: a live smoke test sending a real
Alertmanager webhook produced zero DB spans, only the rate limiter's
Redis span. Fixes the actual gap.

Repository layer — 8 methods across 3 files now take ctx and call
db.WithContext(ctx): IncidentRepository.{Create,GetByID,Update,
LinkAlert}, AlertRepository.{Create,GetByExternalID,Update},
TimelineRepository.Create. The rest of each interface intentionally
does not — same mixed pattern user_repository.go's Upsert already
established, not a uniform migration of every method.

Service layer, real ctx threaded: AlertService.
{ProcessAlertmanagerPayload,ProcessNormalizedAlerts,createOrUpdateAlert},
IncidentService.{CreateIncidentFromAlert,
CreateIncidentFromAlertWithGrouping,LinkAlertToExistingIncident},
AcknowledgeAlertWithTimeline, coordinator.SeedDemoData,
PostMortemAgent.writeTimelineEntry and TeamsEventHandler.
syncMessageToTimeline (both already had ctx via their callers, just
weren't using it — renamed from _ to ctx).

Handlers updated to pass c.Request.Context() — all 4 alert-intake
entry points, not just Prometheus: prometheus_webhook.go,
webhook_handler.go (the shared Grafana/CloudWatch/generic path),
alerts.go, neuri.go, setup.go.

REG-10's WithoutCancel criterion, done for these paths: both
CreateIncidentFromAlert and CreateIncidentFromAlertWithGrouping now
capture bgCtx := context.WithoutCancel(ctx) before spawning their
Telegram/push goroutines and pass it to recoverAsyncPanic — real
trace-linked spans on panic recovery. Also added panic recovery to a
goroutine in LinkAlertToExistingIncident that had none before.

Explicit placeholder, not silently dropped: every remaining
IncidentService method sharing the 4 changed repo methods
(GetIncident, CreateIncident, UpdateIncident, AcknowledgeIncident,
ResolveIncident, UpdateIncidentStatus, CreateTimelineEntry, the
private createTimelineEntry helper) keeps its existing signature and
passes context.Background() internally. Their callers include
slack_event_handler.go (14 sites, no ctx plumbed through that file
today) and post_mortem_service.go — a structurally different surface
(bot event loop, not request/response) that deserves its own scoped
pass rather than riding along here. Full scope and rationale recorded
in REG-157.

Testing: internal/repository/ctx_propagation_test.go proves real span
correlation (not mocked) for all three repos' Create, using the actual
gorm tracing plugin against an in-memory sqlite DB with an injected
TracerProvider. Live-verified against a real Jaeger collector: the
same webhook that produced 1 span before this change now produces 5 —
incidents Create, two alerts spans (dedup check + Create), the Redis
eval, and the root HTTP span — all under one trace ID. db.query.text
confirmed empty (redacted by default, per REG-9).

Also fixed: a mock/stub method touched for this change
(fullIncidentRepoForNeuri, mockIncidentRepoForSetup,
mockAlertRepository, mockIncidentService) updated to the new
signatures; ~25 test call sites across 9 test files updated to pass
context.Background() or a real request context as appropriate.
@mintlify

mintlify Bot commented Sep 1, 2026

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
fluidify 🟢 Ready View Preview Sep 1, 2026, 8:27 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@singret
singret merged commit 4206a65 into main Sep 1, 2026
5 checks passed
@singret
singret deleted the reg-157-incident-create-ctx-threading branch September 1, 2026 08:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant