Give the autoscale guard an org-scoped metrics token - #562
Conversation
There was a problem hiding this comment.
Pull request overview
This PR fixes Autoscale Guard’s inability to query org-level Fly Prometheus metrics by switching authentication from the app-scoped deploy token (FLY_API_TOKEN) to an org-scoped metrics token (FLY_METRICS_TOKEN), and improves HTTP error reporting to make authorization failures actionable.
Changes:
- Prefer
FLY_METRICS_TOKEN(with fallback toFLY_API_TOKEN) when querying the Fly Prometheus API. - Add HTTPError handling that surfaces status code + response body (and provides a specific hint for 403 deploy-token scope issues).
- Update the Autoscale Guard GitHub Actions workflow to pass
FLY_METRICS_TOKENinstead ofFLY_API_TOKEN.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| scripts/check-scale.py | Switch token selection to metrics-first and improve HTTP error handling for Prometheus queries. |
| .github/workflows/autoscale-guard.yml | Pass the new org-scoped metrics secret into the Autoscale Guard run. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| detail = exc.read().decode(errors="replace").strip()[:200] | ||
| hint = "" | ||
| if exc.code == HTTP_FORBIDDEN: | ||
| # A deploy token authenticates but is scoped to one app, so it cannot | ||
| # read org-level metrics. A bad token gives 401, not 403. | ||
| hint = ( | ||
| f" A deploy token cannot read org metrics -- set FLY_METRICS_TOKEN to" | ||
| f" a read-only org token (fly tokens create readonly -o {ORG})." | ||
| ) | ||
| sys.exit(f"error: metrics API returned {exc.code}.{hint} {detail}") |
| # Prefer a metrics-scoped token; FLY_API_TOKEN is the app-scoped deploy token. | ||
| token = os.environ.get("FLY_METRICS_TOKEN") or os.environ.get("FLY_API_TOKEN") | ||
| if not token: | ||
| sys.exit("error: FLY_API_TOKEN is not set") | ||
| sys.exit("error: neither FLY_METRICS_TOKEN nor FLY_API_TOKEN is set") |
50d44b2 to
acacc75
Compare
|
Addressed both review comments and added tests.
message = f"error: metrics API returned {code}.{hint}"
detail = body.decode(errors="replace").strip()
return f"{message} {detail}" if detail else messageCovered by Stale docstring — now names
TestsPulled the decision and the error message out into The two that carry the design: def test_escalates_on_the_2026_08_28_drain(self):
# 2h window peaked at 56,010, five hours before the first 504
escalate, reason = check_scale.decide("shared", 56010.0, 41134.0)
self.assertTrue(escalate)
def test_does_not_escalate_on_a_deploy_dip(self):
# bottomed at 9,983 but recovered inside 90 min, so the window
# still holds a healthy reading. Depth alone would have fired.
escalate, _ = check_scale.decide("shared", 100000.0, 9983.0)
self.assertFalse(escalate)Plus: no escalation on a performance profile or with missing samples, the floor backstop, Env access is isolated with Full suite: 497 passed, 1 pre-existing failure ( Still needs the secret before the guard actually runs: fly tokens create readonly -o upai
gh secret set FLY_METRICS_TOKEN |
Every scheduled run since the guard landed has failed with a 403 from the Fly
metrics API. FLY_API_TOKEN is the app-scoped deploy token created in 2023: it
authenticates fine, which is why the failure is 403 and not the 401 a bad token
gets, but org-level Prometheus is out of its scope.
The guard now reads FLY_METRICS_TOKEN, which needs a read-only org token:
fly tokens create readonly -o upai
Leaving the deploy token app-scoped rather than widening it keeps the blast
radius of the more widely used secret unchanged.
HTTP errors report the status and body instead of surfacing a urllib
traceback, since the traceback said nothing about which of the two plausible
causes it was. The body is read with a cap and only appended when non-empty.
fly.toml is read with a regex rather than tomllib. The workflow runs this on a
bare runner with no install, and pyproject supports Python 3.10, which has no
tomllib -- so importing it broke collection for the whole test suite on CI.
The decision and the error message are separate functions so they can be
tested without the network, covered by server/tests/test_check_scale.py. The
cases that matter are the two the design turns on: the 2026-08-28 drain, whose
2h peak of 56,010 escalates; and a deploy dip, which bottoms at 9,983 and does
not, because the window still holds a healthy reading.
acacc75 to
56b7274
Compare
Every scheduled
Autoscale Guardrun since it landed has failed:Cause
FLY_API_TOKEN(created 2023-07-06) is the app-scoped deploy token. It authenticates correctly — which is why the failure is a 403 and not a 401 — but the org-level Prometheus endpointapi.fly.io/prometheus/upai/is outside its scope.Verified: a deliberately bogus token returns 401 with
something went wrong resolving organization, while a personal org token returns 200 against the identical query. So this is an authorisation gap, not a bad credential.Change
The guard now reads
FLY_METRICS_TOKEN. The deploy token stays app-scoped rather than being widened, so the blast radius of the more widely used secret is unchanged.HTTP errors now report status and body instead of a
urllibtraceback — the traceback didn't distinguish "wrong token" from "wrong scope", which is the whole question here.Required before this works
This PR alone does not fix the failures — the secret has to exist:
Until then the guard fails with
neither FLY_METRICS_TOKEN nor FLY_API_TOKEN is set, which is at least the actionable version of the message.Note the org already has a
Read-only org token(expires 2046). If its value was saved somewhere, it can be reused instead of minting a new one.Testing
./scripts/lintpasses. Ran locally against the live API: a valid token reportshealthy: 2h peak 200000, 15m min 199948; an invalid one exits 1 with the status and body and no traceback.🤖 Generated with Claude Code