chore: this rebuild ships as 0.49.0+fork.2 #13
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Launch benchmark | |
| # The launch-latency trend (map #193). Every merge to main benches the two | |
| # reproducible launch shapes, publishes one median point per stage, and alerts | |
| # on a large relative regression. | |
| # | |
| # ALERT, DO NOT GATE. This workflow must never become a required check and must | |
| # never fail a build on a slow point. One quiet host's five identical | |
| # cold-recreate runs spread 15.2-20.0s; a shared runner is worse, so a hard | |
| # wall-clock gate would be either too loose to fire or flaky. Deterministic | |
| # structural guards keep the gating role: the devpod spawn counts a launch is | |
| # allowed, asserted argv by argv in `flows::launch`'s tests, and their kin | |
| # across the cargo suite. Those run in ci.yml, where a gate belongs. (The | |
| # spawn-count guard was `test_devpod_spawn_counts` in Python until the Python | |
| # tree retired in #267, and the counts came across with it.) | |
| # | |
| # It is a separate file for exactly that reason: house convention is that a new | |
| # job in ci.yml joins `gate`'s `needs` in the same pull request, and `needs` | |
| # cannot reach across files. A job in there would have become a merge gate by | |
| # convention, which is the thing the charter grilled out. `test_bench_workflow.py` | |
| # holds this and the other charter properties down. | |
| on: | |
| push: | |
| branches: [main] | |
| workflow_dispatch: | |
| inputs: | |
| alert-threshold: | |
| description: >- | |
| Relative regression that alerts, as a percentage of the previous | |
| point. Lower it on a dispatch to watch the alert actually fire -- | |
| an alert nobody has ever seen fire is worth nothing -- without | |
| committing a temporary threshold to main. | |
| default: '150%' | |
| required: false | |
| # Deliberately no `paths:` filter, and this is not an omission: a launch can be | |
| # slowed by any file in the repo, and a trend with holes in it cannot compare a | |
| # point against its own previous point. | |
| concurrency: | |
| # Serialise. The publishing action commits to gh-pages, so two runs race on | |
| # that push -- it retries, but not racing is cheaper -- and two benches | |
| # sharing a runner class would also be measuring each other. | |
| group: launch-benchmark | |
| cancel-in-progress: false | |
| permissions: | |
| # One permission covers both halves of publishing: the push to gh-pages and | |
| # the alert's commit comment. The repository default is read-only. | |
| contents: write | |
| jobs: | |
| bench: | |
| runs-on: ubuntu-latest | |
| # Generous, because the priming step below builds a devcontainer image from | |
| # scratch on every run. Bounded, because an unbounded job on a registry | |
| # serving at 640 B/s hangs for six hours and reports a timeout nobody reads. | |
| timeout-minutes: 45 | |
| env: | |
| # The benched repo. This is an interface, not a detail: the trend's whole | |
| # meaning is "the same launch, over time", so changing this repo makes | |
| # every existing point incomparable with every later one. Change it the | |
| # way a stage name is changed -- deliberately, knowing the line breaks. | |
| # | |
| # A repository variable and not a hardcoded literal only so a fork can | |
| # bench its own target instead of upstream's without a diff here; the | |
| # default is the same literal as before, so upstream's trend is | |
| # unaffected and the "deliberately, knowing the line breaks" rule above | |
| # is unchanged -- an unset variable changes nothing. | |
| # | |
| # Its own devcontainer builds an image and runs `pixi install` in | |
| # postCreate, both of which the cold-recreate shape pays for. That is the | |
| # first thing to look at if the cold trend proves noisier than the alert | |
| # threshold can live with. | |
| TARGET: ${{ vars.BENCH_TARGET || 'blooop/mcp-devtasks' }} | |
| # Every byte this job writes, scoped away from anything else on the | |
| # runner: DEVPOD_HOME (plus DEVPOD_SSH_CONFIG, which `devpod up` writes | |
| # outside that home) is devpod's own state, XDG_CACHE_HOME is | |
| # devlaunch's clone cache and bookkeeping. The runner is ephemeral, so | |
| # /tmp is the same scratch recipe the bench script's epilog documents. | |
| DEVPOD_HOME: /tmp/dl-bench/devpod | |
| DEVPOD_SSH_CONFIG: /tmp/dl-bench/ssh_config | |
| XDG_CACHE_HOME: /tmp/dl-bench/cache | |
| # dl forwards a GitHub token into every workspace it opens, via | |
| # `gh auth token` -- which answers from GH_TOKEN when one is set. Without | |
| # this the host-prep stage times a token fetch that fails. | |
| GH_TOKEN: ${{ github.token }} | |
| steps: | |
| - name: Checkout | |
| uses: actions/checkout@v7 | |
| # No devpod install step: devpod is a pixi dependency, already in the | |
| # lockfile setup-pixi restores, and it is the version this tree is pinned | |
| # against rather than whatever a runner image happens to carry. | |
| - uses: prefix-dev/setup-pixi@v0.10.2 | |
| with: | |
| cache: true | |
| frozen: true | |
| environments: default | |
| - name: Let dl's git reach GitHub without an SSH key | |
| # dl clones and probes over `git@github.com:`, and a runner has no key | |
| # for that. The rewrite is the standard CI substitution and touches | |
| # only what git resolves -- devpod never talks to GitHub here, because | |
| # the worktree backend hands it a local path. | |
| run: | | |
| git config --global \ | |
| url."https://x-access-token:${{ github.token }}@github.com/".insteadOf \ | |
| "git@github.com:" | |
| - name: Build the release binaries, and put `dl` and devpod on PATH | |
| # THE THING BEING BENCHED, named explicitly. Until #267 these steps ran | |
| # `pixi run dl`, which resolved to a console script the editable Python | |
| # install put in the environment -- so after the 0.1.0 cutover this trend | |
| # was measuring the Python build while the released `dl` was Rust, and | |
| # nothing said so. There is no `dl` in the environment now, which is what | |
| # turned a wrong number into a missing one. | |
| # | |
| # The *release* profile, because that is what ships and because a debug | |
| # build's startup would swamp the milliseconds this trend is about. | |
| # `pixi run dl` is deliberately not used below for the same reason: it is | |
| # the edit-run loop's debug build (see the task's comment in | |
| # pyproject.toml). | |
| # | |
| # devpod goes on the same PATH, because `dl` shells out to a bare | |
| # `devpod` and there is none on a runner: it is a pixi dependency, which | |
| # every step that runs `dl` inside `pixi run` gets for free and the | |
| # priming launch below -- the one launch that runs `dl` directly -- does | |
| # not. #267 made that launch bare in the same edit that stopped it being | |
| # `pixi run dl`, and every merge to main from there through 0.10.0 died | |
| # in it: `devpod not found on PATH`, exit 127, eighty seconds, nothing | |
| # timed. | |
| # | |
| # A symlink, and not the environment's whole `bin` on PATH: that | |
| # directory carries a python, a git and a few hundred other things, and | |
| # what the benched launch resolves is part of what is being measured. | |
| run: | | |
| set -eu | |
| pixi run build-release | |
| mkdir -p /tmp/dl-bench/bin | |
| ln -sf "$PWD/rust/target/release/dl" /tmp/dl-bench/bin/dl | |
| ln -sf "$(pixi run -q bash -c 'command -v devpod' | tr -d '\r\n')" \ | |
| /tmp/dl-bench/bin/devpod | |
| echo /tmp/dl-bench/bin >> "$GITHUB_PATH" | |
| # Both, resolved through the symlinks, before anything depends on | |
| # them: a dangling link fails at the launch otherwise, which reads as | |
| # a launch bug rather than as this step. | |
| /tmp/dl-bench/bin/dl --version | |
| /tmp/dl-bench/bin/devpod version | |
| - name: Register the docker provider in this job's devpod home | |
| # A fresh DEVPOD_HOME has no provider at all, so this is not optional | |
| # setup -- it is what makes the home usable. Docker itself is already | |
| # on the runner. | |
| run: pixi run dev-add-docker | |
| - name: Let the container's user write the shared pixi cache | |
| # THE BENCH'S SECOND uid MISMATCH, and it is the same one | |
| # scripts/bench_cold_reset.sh recovers from at the other end of a | |
| # launch. dl creates the shared pixi cache's host directory as the | |
| # invoking user with the default umask -- runner:runner, 0755 -- and | |
| # binds it into the container, which runs as the image's remoteUser: | |
| # `vscode`, uid 1000 in every mainstream devcontainer base. A | |
| # GitHub-hosted runner's own user is not that uid, so the container | |
| # cannot create a single entry in the directory it was handed. | |
| # | |
| # That is a hard launch failure and not a lost optimisation, because | |
| # `--workspace-env PIXI_CACHE_DIR` points every pixi in the workspace at | |
| # the mount and pixi does not degrade to a cache it cannot write. The | |
| # benched repo's own postCreate is `pixi install`, so the PRIMING STEP | |
| # BELOW DIES -- `failed to create directory /var/tmp/devlaunch-pixi/pkgs: | |
| # Permission denied (os error 13)`, then `run agent command failed` -- | |
| # and the job has never got as far as timing anything since #232 merged. | |
| # Every run from 31886967581 (2026-08-15) to 32397739366 failed here. | |
| # | |
| # It is the SOURCE's ownership and not the target path: #240 moved the | |
| # container-side target from `~/.cache/devlaunch-pixi` to | |
| # `/var/tmp/devlaunch-pixi` for an unrelated root-owned-parent bug, and | |
| # the failure came through the move character for character with only | |
| # the path in it changed. | |
| # | |
| # README's "shared pixi package cache" section documents this as dl's | |
| # own limitation -- it cannot see the container's uid before it launches, | |
| # so it cannot decide the mode for you -- and names running the image as | |
| # your own uid, or widening the directory, as the fixes available to the | |
| # person who knows. Here we know: one image, one remoteUser, an | |
| # ephemeral runner, and a cache under /tmp that the job created. So the | |
| # recovery belongs to the environment that creates the mismatch, exactly | |
| # as the cold reset's chown does. | |
| # | |
| # `1777` and not `0777`: it is what /var/tmp itself carries, and the | |
| # sticky bit is free here -- everything under the leaf is created by the | |
| # one container user, so nothing needs to unlink an entry it does not | |
| # own. The mode goes on the LEAF only: the parents keep whatever the | |
| # umask gives them, and every umask a runner has already grants the | |
| # world the traverse (`o+x`) that reaching the leaf needs. dl's own | |
| # mkdir is `exist_ok` and does not re-mode a directory it finds, so | |
| # pre-creating is what settles this, and it has to happen before the | |
| # first launch. | |
| # | |
| # Once, rather than per-run like the cold reset: nothing in this job | |
| # removes this directory. `dl <ws> rm` takes the clone and never the | |
| # shared cache, and `--purge` -- the one command that would -- is not | |
| # run here. | |
| run: | | |
| cache="$XDG_CACHE_HOME/devlaunch/pixi" | |
| mkdir -p "$cache" | |
| chmod 1777 "$cache" | |
| # Recorded rather than assumed: the two uids are the whole of this | |
| # step, and a log that names the runner's is what makes the next | |
| # `Permission denied` under this mount readable in one look. | |
| echo "runner uid $(id -u), shared pixi cache now $(stat -c '%U %a' "$cache")" | |
| - name: Prime the image and the clone cache | |
| # THE FIRST-EVER-COLD EXCLUSION, and it is deliberate rather than | |
| # redundant with the benched runs below. Charter decision 4 keeps the | |
| # first-ever cold launch out of the per-merge trend: its dominant cost | |
| # is an image transfer or build this repo does not control, and the | |
| # image build's own duration is already visible in the target's own | |
| # workflows. An ephemeral runner starts every job in that state, so | |
| # without this step the first timed cold run would be a first-ever cold | |
| # one and the trend would carry a number about a registry's day. | |
| # | |
| # It also leaves the workspace up, which is what makes the next step | |
| # warm. | |
| run: dl "$TARGET" -- true | |
| # Five runs per shape, both shapes. The bench script's own recipe says | |
| # five, and the point published is their MEDIAN -- the trend's baseline is | |
| # the immediately previous point and nothing older, so de-noising has to | |
| # happen here or the alert is a coin flip. Three would save about a minute | |
| # on a job that spends longer than that priming, and would halve the | |
| # median's resistance to one slow run: the exact noise this whole map is | |
| # about. | |
| - name: Bench the warm shape | |
| run: pixi run bench -n 5 --record warm.json --shape warm -- dl "$TARGET" -- true | |
| - name: Bench the cold-recreate shape | |
| # `--before` re-establishes the cold state before EVERY run: delete once | |
| # and bench five times and runs 2..5 are warm, i.e. a warm median under | |
| # a cold label. What it removes is the workspace and its container; the | |
| # image layers and the bare clone the priming step left behind survive | |
| # on purpose, which is what makes this "recreate" rather than "first | |
| # ever". | |
| # | |
| # The reset is `scripts/bench_cold_reset.sh`, which takes the clone back | |
| # from the container's uid before removing it and says there why that | |
| # recovery belongs to a runner rather than to `dl` (run 31838698495). | |
| # | |
| # It is a FILE rather than a string here because the string did not | |
| # survive the trip: `--before` reaches the bench through | |
| # `pixi run bench`, whose task shell re-joins and re-parses a task's | |
| # appended arguments, so the quotes nested inside this step's own quotes | |
| # were gone by the time argparse saw them. That run exited 2 on the | |
| # wreckage, pixi's shell ran the tail after the `;` as a command of its | |
| # own, and the STEP passed with no record written (run 31840842480). | |
| # One level of quoting has nothing left to lose. | |
| # | |
| # It stays INSIDE `--before`, the only place with the right cardinality: | |
| # every run recreates those files, so a one-time step before the bench | |
| # would recover run 1 and hand runs 2..5 exactly the clone that broke it. | |
| run: | | |
| pixi run bench -n 5 --record cold-recreate.json --shape cold-recreate \ | |
| --before "scripts/bench_cold_reset.sh $TARGET" \ | |
| -- dl "$TARGET" -- true | |
| - name: Turn the records into this commit's trend points | |
| # `--require-stages-on cold-recreate` is the load-bearing flag: a cold | |
| # recreate that reported no `tools` stage means the launch changed shape | |
| # or stopped saying so, and this step then fails the JOB and publishes | |
| # nothing -- rather than letting a stage go quietly absent from a chart | |
| # nobody reads that week. The warm shape is not asserted: it legitimately | |
| # lends nothing, and requiring `tools` of it would fail every green run. | |
| run: | | |
| pixi run python scripts/bench_points.py warm.json cold-recreate.json \ | |
| --out bench.json --require-stages-on cold-recreate \ | |
| --note "$(pixi run -q devpod version | tr -d '\r\n'), ${{ runner.os }}-${{ runner.arch }}" | |
| - name: Publish the point and alert on a regression | |
| uses: benchmark-action/github-action-benchmark@v1.22.1 | |
| with: | |
| name: devlaunch launch stages | |
| tool: customSmallerIsBetter | |
| output-file-path: bench.json | |
| github-token: ${{ secrets.GITHUB_TOKEN }} | |
| auto-push: true | |
| gh-pages-branch: gh-pages | |
| benchmark-data-dir-path: dev/bench | |
| # One global relative threshold to start with; per-stage thresholds are | |
| # parked until there is false-alarm experience to tune them against. | |
| alert-threshold: ${{ inputs.alert-threshold || '150%' }} | |
| comment-on-alert: true | |
| # The point of the entire map. `fail-threshold` only applies when this | |
| # is true, so leaving it unset is safe. | |
| fail-on-alert: false | |
| # On a push there is no pull request, so the alert is a COMMIT comment | |
| # -- and commit comments notify almost nobody. This @-mention is what | |
| # actually produces a notification; without it the alert is invisible. | |
| # A repository variable, like TARGET above, so a fork's regressions | |
| # notify the fork's own owner rather than CC'ing upstream on a repo | |
| # they do not maintain; unset, the default is the same handle as | |
| # before and upstream is unaffected. | |
| alert-comment-cc-users: '${{ vars.BENCH_ALERT_CC || ''@blooop'' }}' | |
| # The comparison table lands in every run's job summary, so "did it | |
| # move" is answerable without opening the chart or waiting for a breach. | |
| summary-always: true | |
| # Unset means unbounded growth of the published data file. | |
| max-items-in-chart: 500 | |
| - name: Keep the runs behind the medians | |
| # The trend carries medians; this carries the spread that produced them, | |
| # for digging after an alert. `always()` because a job that failed its | |
| # stage assertion is exactly the one whose records somebody wants. | |
| if: always() | |
| uses: actions/upload-artifact@v7 | |
| with: | |
| name: bench-records | |
| path: | | |
| warm.json | |
| cold-recreate.json | |
| bench.json | |
| if-no-files-found: warn |