Skip to content

[operations] Add runbook for fuzz-mdx.yml silent-failure monitoring - #6806

Closed
kubestellar-hive[bot] wants to merge 1 commit into
mainfrom
operations/fuzz-mdx-monitoring-runbook
Closed

[operations] Add runbook for fuzz-mdx.yml silent-failure monitoring#6806
kubestellar-hive[bot] wants to merge 1 commit into
mainfrom
operations/fuzz-mdx-monitoring-runbook

Conversation

@kubestellar-hive

Copy link
Copy Markdown
Contributor

What

Adds runbooks/fuzz-mdx-monitoring.md, documenting how to manually detect and recover from a silently-failed weekly run of .github/workflows/fuzz-mdx.yml (the MDX sanitizer fuzz harness).

Why

Fixes the gap tracked in #6805: fuzz-mdx.yml's only if: failure() step uploads a 14-day crash-log artifact — it never opens or updates a tracking issue, unlike generate-leaderboard.yml/generate-acmm-history.yml in this repo. A crash on the unattended weekly scheduled run (as opposed to a PR check) can go unnoticed indefinitely, and the reproduction evidence (crash seed/log) is deleted after 14 days. This follows the same pattern already used for runbooks/scorecard-monitoring.md (#6724) and runbooks/stale-workflow-monitoring.md (#6729).

The workflow-side if: failure() alert fix itself is out of scope for this PR — it requires workflows permission that agent tokens do not have, so a maintainer needs to add it directly to .github/workflows/fuzz-mdx.yml. This runbook is the bounded, non-workflow mitigation in the meantime.

Scope

Docs-only change (new file under runbooks/). No workflow, alert, or code behavior is modified.

Closes: partially addresses #6805 (manual mitigation; workflow fix still needs a maintainer).

— hive: agent=operations backend=copilot model=claude-sonnet-4-6

Documents the gap tracked in #6805: fuzz-mdx.yml's weekly MDX sanitizer
fuzz run only uploads a 14-day crash artifact on failure and never
opens a tracking issue, unlike generate-leaderboard.yml and
generate-acmm-history.yml in this repo. Adds manual detection and
recovery steps until a maintainer can add the workflow-side alert
(agent tokens cannot write .github/workflows/*).

Signed-off-by: kubestellar-hive[bot] <223556219+Copilot@users.noreply.github.com>
@kubestellar-hive

Copy link
Copy Markdown
Contributor Author

Important

Held for human sign-off on the direction, not on the code.

This PR's only tracked rationale is #6805, which the hive filed itself — issue #6805 was filed by kubestellar-hive[bot] and no human has acknowledged it. An agent-filed issue does not, on its own, establish that anyone agreed to the direction (hivecommons/hive#5117).

The change may well be right; nothing here is a review of it. To release the hold, acknowledge the direction on that issue — comment on it, assign yourself, or add the approved-direction label — and remove the hold label here.

@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Sep 8, 2026
@kubestellar-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kproche for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@netlify

netlify Bot commented Sep 8, 2026

Copy link
Copy Markdown

Deploy Preview for kubestellar-docs ready!

Name Link
🔨 Latest commit f507c3d
🔍 Latest deploy log https://app.netlify.com/projects/kubestellar-docs/deploys/6a9fd8269d136d0008708415
😎 Deploy Preview https://deploy-preview-6806--kubestellar-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@kubestellar-prow kubestellar-prow Bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 8, 2026
@github-actions github-actions Bot added documentation Improvements or additions to documentation and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Sep 8, 2026
@kubestellar-prow

Copy link
Copy Markdown

Hi @kubestellar-hive[bot]. Thanks for your PR.

I'm waiting for a kubestellar member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubestellar-hive

Copy link
Copy Markdown
Contributor Author

Closing: superseded by runbooks/fuzz-mdx-failure-detection.md, already merged to main in #6808 documenting this exact fuzz-mdx.yml silent-failure gap. Adding a second differently-named runbook (fuzz-mdx-monitoring.md) for the same gap would create duplicate/conflicting docs. See #6856.

🐝 Hive Agent: operations | Instance: hosted-kubestellar-console-4vkt | SHA: 989d40c5

— hive: agent=operations backend=copilot model=claude-sonnet-4-6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

copilot dco-signoff: yes Indicates the PR's author has signed the DCO. documentation Improvements or additions to documentation hold needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant