Skip to content

fix: alert on stalled rollout-checker CronJobs - #7255

Merged
hivecommons-hive[bot] merged 1 commit into
mainfrom
ops/rollout-checker-alerting
Oct 5, 2026
Merged

hivecommons-hive[bot] merged 1 commit into
mainfrom
ops/rollout-checker-alerting

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Finding

cluster-objects/prometheusrule.yaml alerts on the docs site's own
/api/metrics output (error rate, latency, scrape-target-down), but the
automated deploy pipeline itself had no alerting: nextra-rollout-checker
(cluster-objects/job.yaml, every 2m) and nextra-pr-rollout-checker
(cluster-objects/pr-job.yaml, every 1m) poll OCIR and roll out new
images with set -e, concurrencyPolicy: Forbid, backoffLimit: 0, and
only failedJobsHistoryLimit: 1 — a stuck/failing checker (expired
oci-config-secret, an RBAC regression on nextra-rollout-sa, an OCI API
outage) silently stops deploying new images with no signal.

Closes #7254.

Change

  • Adds DocsRolloutCheckerStalled / DocsPrRolloutCheckerStalled alerts
    to cluster-objects/prometheusrule.yaml, driven by kube-state-metrics'
    kube_cronjob_status_last_successful_time (a near-universal companion
    to a Prometheus Operator install — no new exporter is added, consistent
    with every other resource in cluster-objects/).
  • Adds runbooks/rollout-checker-failure.md (diagnose via kubectl get cronjob/logs, check oci-config-secret and nextra-rollout-sa RBAC)
    and a runbooks/README.md index entry.
  • Extends src/__tests__/cluster-objects-metrics-consistency.test.ts to
    recognize kube-state-metrics' metric/labels as a legitimate external
    source, alongside the existing Prometheus built-in (up).

No existing alert, SLO, or probe is weakened — this only adds two new
alerts and a runbook. npx vitest run src/__tests__/cluster-objects-metrics-consistency.test.ts passes (4/4).

— hive: agent=operations backend=copilot model=claude-sonnet-4-6 copilot=1.0.88

Add DocsRolloutCheckerStalled and DocsPrRolloutCheckerStalled alerts to
cluster-objects/prometheusrule.yaml, backed by kube-state-metrics'
kube_cronjob_status_last_successful_time. Neither nextra-rollout-checker
nor nextra-pr-rollout-checker (cluster-objects/job.yaml, pr-job.yaml) had
any failure alerting: both are concurrencyPolicy: Forbid with
failedJobsHistoryLimit: 1 and no retry beyond the next scheduled tick, so
a stuck/failing checker silently stops deploying new images with no
signal.

Adds runbooks/rollout-checker-failure.md and a runbooks/README.md index
entry, and extends src/__tests__/cluster-objects-metrics-consistency.test.ts
to recognize kube-state-metrics' metric/labels as a legitimate external
source alongside the existing Prometheus built-ins.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: operations <operations@hive.kubestellar.io>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 5, 2026
@kubestellar-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kproche for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@netlify

netlify Bot commented Oct 5, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for kubestellar-docs ready!

Name Link
🔨 Latest commit 602902b
🔍 Latest deploy log https://app.netlify.com/projects/kubestellar-docs/deploys/6ac2f4785518270008a7b8b2
😎 Deploy Preview https://deploy-preview-7255--kubestellar-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@kubestellar-prow kubestellar-prow Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 5, 2026
@kubestellar-prow

Copy link
Copy Markdown

Hi @hivecommons-hive[bot]. Thanks for your PR.

I'm waiting for a kubestellar member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@github-actions github-actions Bot added documentation Improvements or additions to documentation frontend typescript yaml labels Oct 5, 2026
@hivecommons-hive
hivecommons-hive Bot merged commit 78648e2 into main Oct 5, 2026
16 of 20 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the ops/rollout-checker-alerting branch October 5, 2026 00:55
@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

Check out what's new:

Stay connected: Slack #kubestellar-dev | Multi-Cluster Survey

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

copilot dco-signoff: yes Indicates the PR's author has signed the DCO. documentation Improvements or additions to documentation frontend needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. typescript yaml

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[operations] No alerting on nextra-rollout-checker / nextra-pr-rollout-checker CronJob failures — silent deploy-pipeline stalls

0 participants