Skip to content

Incident Management v2: flapping-safe AlertGroup reopen lifecycle #85

Description

@Aidaho12

Parent epic: #45
Related proposal: #84

Release allocation

IncidentRelay 2.4 — lifecycle resilience

  • Event Orchestration set_grouping.reopen_window_seconds; missing/zero = disabled;
  • new child Alert on reopen; resolved child Alerts remain terminal;
  • one eligible resolved AlertGroup reused inside the window; new group outside it;
  • route/team/service/grouping boundaries and resolve/reopen concurrency;
  • linked Incident reopen;
  • recovery notification hysteresis and cancellation on quick reopen;
  • Explain/timeline/audit/UI coverage.

IncidentRelay 2.7 — stale-signal follow-up

  • optional stale-signal policy, disabled by default;
  • async scheduler/worker evaluation; inactivity alone is not recovery.

IncidentRelay 2.9 — analytics

This workstream is no longer a 2.3 release blocker.

Architectural rules

  • closed is not added to the technical AlertGroup state machine.
  • closed remains a terminal operational state of first-class Incident.
  • Resolved child Alerts are never mutated back to firing.
  • Reopening an AlertGroup creates a new child Alert occurrence.
  • Event Orchestration controls reopen eligibility; the default is disabled.
  • set_grouping.window_seconds and reopen_window_seconds are separate
    concepts and must not share semantics.
  • Absence of webhook traffic is not proof of recovery.

2.4 scope

  • Extend Event Orchestration validation, execution, simulator and Explain
    output for reopen_window_seconds.
  • Carry the effective reopen value through orchestration runtime into alert
    lifecycle processing.
  • Find an eligible recently resolved AlertGroup using the effective grouping
    identity and security/routing boundaries.
  • Reopen the group atomically and create a new child Alert occurrence.
  • Preserve old resolved child Alert rows and timestamps.
  • Reset/restart only lifecycle state that the existing firing/reopen rules
    require.
  • Reconcile linked first-class Incident lifecycle without overwriting
    AlertGroup technical state.
  • Add API/OpenAPI, audit, timeline, localization and documentation coverage.
  • Clarify/fix the existing set_grouping.window_seconds contract separately:
    it must not be repurposed as the resolved-group reopen window.

Acceptance criteria

  • Without an orchestration reopen_window_seconds, current behavior is
    unchanged and a firing signal after resolution creates a new AlertGroup.
  • reopen_window_seconds = 0 explicitly disables resolved-group reuse.
  • A matching firing signal within the configured window reuses exactly one
    eligible resolved AlertGroup.
  • Reopen creates a new child Alert and does not change an older resolved
    child Alert back to firing.
  • A firing signal after the configured window creates a new AlertGroup.
  • Reopen respects route/team/service/grouping ownership boundaries.
  • Two concurrent matching firing deliveries cannot create duplicate
    reopened groups or duplicate active occurrences.
  • Resolve-vs-reopen races have deterministic final state.
  • Notification/reminder/escalation workers cannot act on stale pre-reopen
    state.
  • Linked Incident reopen is deterministic and audited.
  • Explain/simulation shows the effective reopen policy without mutation.
  • No technical AlertGroup closed state is introduced.
  • Inactivity alone never resolves a firing signal in the default
    configuration.

Out of scope for the 2.4 subset

  • automatic Incident closure;
  • global always-on reopen behavior;
  • treating set_grouping.window_seconds as the reopen window;
  • stale-firing auto-resolution;
  • flapping analytics dashboard.

Dependencies

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    incident-management-v2Incident Management v2 architecture and delivery

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions