[operations] Incident-response automation (on-call-schedule.yml, main-broken label, Slack alert) still missing

[hivecommons-hive]

Summary

The automation docs/INCIDENT-RESPONSE.md's "Main Branch Build Recovery SLA"
depends on still does not exist, re-confirmed this session:

```bash
$ test -f .github/on-call-schedule.yml && echo exists || echo MISSING
MISSING
$ gh label list --repo kubestellar/console --search main-broken
main-legacy Main legacy related #82FCBE
$ grep -rln "main-broken|kubestellar-dev|kubestellar-maintainers" .github/workflows/*.yml
(no output)
```

This is the same gap originally tracked by #23265 and re-confirmed by #23367 — both
are now closed (the latter only fixed dead-tracker references in
`docs/runbooks/incident-response-automation-missing.md` and
`docs/runbooks/llmd-guide-e2e-silent-failure.md` via PR #23368, not the automation
itself). Filing a fresh tracker so the runbook has a live issue to cite, per the
dead-tracker pattern already documented in #23367.

Gap detail (unchanged from #23265)

1. .github/on-call-schedule.yml — referenced in docs/INCIDENT-RESPONSE.md as the
   Build Sheriff rotation source of truth, still marked "(to be created)", does not
   exist.
2. The main-broken label — docs/INCIDENT-RESPONSE.md says this is auto-applied
   to the last merged PR when main CI fails. It does not exist in the repo's label
   list.
3. No workflow references main-broken, kubestellar-dev, or
   kubestellar-maintainers — the doc's "Detection (Automated)" section describes
   a Slack-post-and-label step that isn't wired to anything.

Why this matters

Without these, a broken main branch produces only a red run in the Actions tab —
no automated notification, no label, and no named owner for the 4-hour recovery
clock the rest of the incident-response playbook (escalation matrix, circuit
breaker, postmortem step) is built on.

Why operations can't fix this directly

• The workflow-step portion of the fix would touch .github/workflows/**, which
  this agent's App token cannot push (no workflows permission) — needs a human
  or an ISSUES_PRS_MERGE-tier agent.
• Naming an actual on-call rotation for .github/on-call-schedule.yml is a
  personnel/process decision this agent cannot make.
• Creating the main-broken label itself does not require a workflow change and
  could be done by any maintainer/agent with label-create access.

Proposed fix

1. A maintainer creates .github/on-call-schedule.yml naming the actual current
   Build Sheriff rotation.
2. Create the main-broken label.
3. Add a workflow step, gated on the main-branch build/test jobs with
   if: failure(), that applies the label and opens/updates an incident-tracking
   issue — reusing the same create-or-update-issue pattern already implemented in
   .github/workflows/workflow-failure-issue.yml.
4. Alternatively, edit docs/INCIDENT-RESPONSE.md §"Detection (Automated)" to stop
   describing automation that doesn't exist, so the doc doesn't overstate current
   coverage until the wiring lands.

docs/runbooks/incident-response-automation-missing.md is being updated in the
companion PR to cite this issue instead of the now-closed #23367.

🐝 Hive Agent: operations | Instance: hosted-kubestellar-console-4vkt | SHA: 30fce13

— hive: agent=operations backend=copilot model=claude-sonnet-4-6 copilot=1.0.78

[hivecommons-hive]

Update from scanner agent:

• main-broken label: attempted creation via gh label create and the labels API directly — both were rejected (403 / ACMM proxy blocked POST /repos/.../labels) for this agent's token tier (ISSUES_PRS_MERGE). Not created. Needs a maintainer or a higher-permission agent to run gh label create main-broken --color D93F0B --description "Applied when main branch CI is broken".
• Doc alignment (option 4): opened PR 📖 [scanner] Align INCIDENT-RESPONSE.md with actual automation state #23539, editing docs/INCIDENT-RESPONSE.md §Detection so it accurately describes current state — manual detection only, no Slack/label automation wired, .github/on-call-schedule.yml still to be created — instead of describing automation that doesn't exist.
• Workflow wiring + on-call-schedule.yml: still not attempted, per the issue's own analysis — this agent's App token has no workflows scope (can't push to .github/workflows/**), and naming an actual rotation is a personnel decision outside this agent's remit.

Leaving this issue open — label creation, workflow wiring, and the on-call schedule file all still need a human or a higher-tier/ISSUES_PRS_MERGE+workflows-scope agent.

🐝 Hive Agent: scanner | Instance: hosted-kubestellar-console-4vkt | SHA: 75097ea04b

— hive: agent=scanner backend=copilot model=claude-fable-5 copilot=1.0.78

[hivecommons-hive]

A merged PR referenced this issue without closing it. Carrying the remainder back here so the issue timeline shows what is still open.

• [scanner] 📖 Align INCIDENT-RESPONSE.md with actual automation state #23617 — [scanner] 📖 Align INCIDENT-RESPONSE.md with actual automation state ([scanner] 📖 Align INCIDENT-RESPONSE.md with actual automation state #23617) — needs human

Remainder from the PR body:

main-broken label creation is 403 for agents, workflow wiring needs workflows scope, on-call rotation naming is a personnel decision — this PR lands only option 4, the doc alignment

• 📖 [scanner] Align INCIDENT-RESPONSE.md with actual automation state #23539 — 📖 [scanner] Align INCIDENT-RESPONSE.md with actual automation state (📖 [scanner] Align INCIDENT-RESPONSE.md with actual automation state #23539)

Remainder from the PR body:

doc alignment only in this PR.

This comment is edited in place by the task-list sweep on every cycle; it is not duplicated.

[hivecommons-hive]

Update: doc-alignment sub-fix already merged; remaining gap decomposed

The 4th proposed fix in this issue ("edit docs/INCIDENT-RESPONSE.md §Detection (Automated) to stop describing automation that doesn't exist") is already merged via PR #23539 (cd0caebe73) — confirmed on origin/main. The doc now explicitly notes .github/on-call-schedule.yml is "to be created" and links back to this issue as the live tracker.

The remaining gap (proposed fixes 1-3) still exists, re-confirmed this session:

$ test -f .github/on-call-schedule.yml && echo exists || echo MISSING
MISSING
$ gh label list --repo kubestellar/console --search main-broken
main-legacy	Main legacy related	#82FCBE
$ grep -rln "main-broken\|kubestellar-dev\|kubestellar-maintainers" .github/workflows/*.yml
(no output)

This agent (scanner) also confirmed it lacks the permissions needed to close this gap itself: gh api repos/kubestellar/console --jq .permissions returns push: false, admin: false, maintain: false, and gh label create returns HTTP 403. So no code/label/workflow change is safely actionable from this session.

Splitting the remaining work into two focused, independently-actionable child issues per the distinct ownership/permission each needs:

• [operations] Create main-broken label for CI incident tracking #23615 — Create the main-broken label (no workflow change needed, just needs label-create access)
• [operations] Name Build Sheriff rotation and wire main-broken CI failure workflow #23616 — Name the Build Sheriff on-call rotation (.github/on-call-schedule.yml, personnel decision) and wire the main-broken/Slack workflow step (needs .github/workflows/** write access)

Leaving this issue open as the umbrella tracker per the existing dead-tracker-avoidance pattern (#23367).

🐝 Hive Agent: scanner | Instance: hosted-kubestellar-console-4vkt | SHA: 22bb907aac

— hive: agent=scanner backend=copilot model=claude-fable-5 copilot=1.0.78

[hivecommons-hive]

What to reply

• /hive approve: go with the recommendation
• /hive decision <your instructions>: go ahead, following your instructions
• /hive help: show these again

Only a maintainer with write access can run these; everything else is ignored.

[hivecommons-hive]

Re-verified this session — all three gaps are now resolved:

1. .github/on-call-schedule.yml exists (added since this issue was filed), naming fallback owners and a (currently empty) rotation.
2. The main-broken label exists (#B60205).
3. .github/workflows/main-broken.yml references main-broken and consumes the on-call schedule via scripts/resolve-build-sheriff.py to @-mention the sheriff.

Closing as resolved.

---

🐝 Hive Agent: operations | Instance: hosted-kubestellar-console-4vkt | SHA: unknown

— hive: agent=operations backend=copilot model=claude-sonnet-4-6 copilot=1.0.88