diff --git a/docs/personal/release-pipeline.md b/docs/personal/release-pipeline.md index 52df0b08335f..85ca51668a99 100644 --- a/docs/personal/release-pipeline.md +++ b/docs/personal/release-pipeline.md @@ -23,6 +23,27 @@ Every Mac runs "T3 Code (Alpha)" from `/Applications`, built from this fork. A L the desktop app with Settings → Connections → Network access on (`0.0.0.0:3773`, pairing QR). Pairing tokens expire in ~5 minutes; mint one right before pairing (`t3 auth pairing create`). +## Nightly and weekly automation + +A Temporal worker on studio runs this runbook unattended. It lives in +[r4iju/temporal-automations](https://github.com/r4iju/temporal-automations) under +`packages/t3-release` and posts every result to Slack #homelab. + +- **Nightly desktop (05:00 JST):** syncs upstream when the merge is clean and typechecks, + then builds and deploys if `origin/main` moved since the last `desktop-v*` tag. A conflicted + or failing sync is left as a PR for a human, and the night ships `main` without it. +- **Weekly mobile (Sunday 06:00 JST):** `t3-mobile-release`, which queues EAS builds only when + the app's code changed since the last `mobile-*` tag. +- **Busy Macs are skipped:** a Mac with an agent turn running is left alone and named in + Slack. Install it by hand with `--force` once it is idle, or let the next night catch it. + +The fork has no CI: its workflows need upstream's Blacksmith runners and never start. The +nightly gate is local instead: typecheck, a signed build, then studio's own install as the +canary before any other Mac. + +The worker runs from a dedicated worktree, `~/code/t3code-wt-nightly`, detached at +`origin/main`, so it never touches the main checkout. + ## 1. Sync upstream Direct pushes to `main` are blocked, so a sync is a PR: @@ -44,19 +65,24 @@ git -C ~/code/t3code worktree remove ~/code/t3code-wt-sync On studio, from a clean `~/code/t3code` on `main`: ```bash -scripts/personal/t3-alpha-build 0.0.47 +scripts/personal/t3-alpha-build # next patch after the latest desktop-v* tag scripts/personal/t3-alpha-deploy release/T3-Code-0.0.47-arm64.zip ``` -- **Version:** the next patch after the last deploy. Every package.json is set to it for the - build and restored afterwards, so the app and its server report the same version. +- **Version:** the next patch after the latest `desktop-v*` tag, or pass one explicitly. A + signed build tags its commit `desktop-v` on origin. Every package.json is set to + the version for the build and restored afterwards, so the app and its server report the + same version. - **Signing:** the build is signed with the Apple Development certificate in studio's keychain. macOS privacy grants (Screen Recording, Accessibility, …), keychain access and Little Snitch rules are tied to that signature, so they carry over between builds. An unsigned build is a new app to all of them; the installer refuses one. -- **Deploy:** copies the zip and installer to matebook and sm-em over SSH, installs, and waits - for each result. Studio goes last because its restart ends any T3 session driving the - deploy. Name hosts to deploy to a subset: `… .zip emanuel@matebook.lan local`. +- **Deploy:** installs on studio first as the canary; if that fails and rolls back, no other + Mac is touched. Then it copies the zip and installer to matebook and sm-em over SSH and + waits for each. Every host ends with a `RESULT installed|busy|unreachable|failed` + line. Name hosts to deploy to a subset: `… .zip emanuel@matebook.lan`. +- **Busy Macs:** a Mac with an agent turn running (including the T3 session driving a manual + deploy) reports `busy` and is left alone. `--force` installs anyway and ends those turns. - **Target Macs must be logged in:** the installer launches the app in the GUI session. - **Failures roll back** to the previous bundle automatically. Each Mac logs to `~/Library/Logs/t3-alpha-install.log`; `scripts/personal/t3-alpha-install --status` shows @@ -76,7 +102,10 @@ T3CODE_IOS_PERSONAL_TEAM=1 T3CODE_IOS_PERSONAL_TEAM_BUNDLE_ID=com.raijustudios.t3code ``` -From `apps/mobile`, logged in to EAS as `expomozdom`: +`scripts/personal/t3-mobile-release` sets them, queues both builds with auto-submit, and tags +the commit `mobile-`. It skips when nothing under `apps/mobile` or `packages` changed +since the last tag (`--force` releases anyway). By hand, from `apps/mobile`, logged in to EAS +as `expomozdom`: ```bash eas build --profile production -p ios --non-interactive --no-wait diff --git a/scripts/personal/t3-alpha-build b/scripts/personal/t3-alpha-build index e91227f0f66b..d87e82af7ec8 100755 --- a/scripts/personal/t3-alpha-build +++ b/scripts/personal/t3-alpha-build @@ -10,15 +10,31 @@ # designated requirement is its own cdhash, so TCC grants, keychain ACLs and # Little Snitch rules never carried over to the next build. # -# Usage: t3-alpha-build # -> release/T3-Code--arm64.zip +# Each build is tagged `desktop-v` on origin once it is signed, so the +# tags record which commit every Mac runs. Without a version the build takes the +# next patch after the latest tag. +# +# Usage: t3-alpha-build [version] # -> release/T3-Code--arm64.zip set -euo pipefail -VERSION="${1:?usage: t3-alpha-build }" SIGN_IDENTITY="${T3_ALPHA_SIGN_IDENTITY:-Apple Development: Emanuel Franzen (T4J48V44Q2)}" REPO="$(cd "$(dirname "$0")/../.." && pwd)" -ZIP="$REPO/release/T3-Code-$VERSION-arm64.zip" cd "$REPO" +git fetch -q origin 'refs/tags/desktop-v*:refs/tags/desktop-v*' +if [[ -n "${1:-}" ]]; then + VERSION="$1" +else + LAST="$(git tag -l 'desktop-v*' --sort=-v:refname | head -1)" + [[ -n "$LAST" ]] || { echo "error: no desktop-v* tag yet; pass a version." >&2; exit 1; } + LAST="${LAST#desktop-v}" + VERSION="${LAST%.*}.$((${LAST##*.} + 1))" +fi +if git rev-parse -q --verify "refs/tags/desktop-v$VERSION" >/dev/null; then + echo "error: desktop-v$VERSION is already tagged." >&2 + exit 1 +fi +ZIP="$REPO/release/T3-Code-$VERSION-arm64.zip" # The artifact build spawns `vp`, which only exists in the repo's node_modules. export PATH="$REPO/node_modules/.bin:$PATH" unset ELECTRON_RUN_AS_NODE @@ -58,5 +74,8 @@ sign_app "$APP" 2> >(grep -v 'replacing existing signature' >&2) rm -f "$ZIP" "$ZIP.blockmap" ditto -c -k --sequesterRsrc --keepParent "$APP" "$ZIP" +git tag "desktop-v$VERSION" HEAD +git push -q origin "desktop-v$VERSION" + echo "Built and signed $ZIP" codesign -d -r- "$APP" 2>&1 | tail -1 diff --git a/scripts/personal/t3-alpha-deploy b/scripts/personal/t3-alpha-deploy index 53ed7e16ee28..bc6fd46e4eb3 100755 --- a/scripts/personal/t3-alpha-deploy +++ b/scripts/personal/t3-alpha-deploy @@ -1,57 +1,98 @@ #!/bin/bash -# t3-alpha-deploy — install a zip from t3-alpha-build on every Mac. Remote hosts -# go first over SSH, `local` last: installing here restarts the T3 Code this -# may be running inside, which ends the session driving the deploy. +# t3-alpha-deploy — install a zip from t3-alpha-build on every Mac and print one +# `RESULT ` line per host: +# installed running the new version +# busy an agent turn is running there; left alone (see t3-alpha-install) +# unreachable asleep or offline +# failed the install failed and rolled back +# skipped not attempted because the canary failed # -# Each host gets this checkout's t3-alpha-install copied next to the zip, so -# no Mac carries its own installer copy that can drift. +# `local` goes first as the canary: if this Mac's install fails, the build is +# broken and no other Mac is touched. A deploy driven from T3 on this Mac finds +# `local` busy (the driving turn itself); install it last with +# `t3-alpha-deploy local --force`. # -# Usage: t3-alpha-deploy [host ...] -# hosts default to: emanuel@matebook.lan emanuelfranzen@Mac.lan local +# Remote hosts get this checkout's installer copied next to the zip, so no Mac +# carries its own installer copy that can drift. +# +# Usage: t3-alpha-deploy [host ...] [--force] +# hosts default to: local emanuel@matebook.lan emanuelfranzen@Mac.lan +# --force install even where agent turns are running +# Exits 1 if any host failed. set -uo pipefail -ZIP="${1:?usage: t3-alpha-deploy [host ...]}" -shift -HOSTS=("$@") -[[ ${#HOSTS[@]} -eq 0 ]] && HOSTS=(emanuel@matebook.lan emanuelfranzen@Mac.lan local) +ZIP="" FORCE="" HOSTS=() +for arg in "$@"; do + case "$arg" in + --force) FORCE=--force ;; + *) if [[ -z "$ZIP" ]]; then ZIP="$arg"; else HOSTS+=("$arg"); fi ;; + esac +done +[[ -f "$ZIP" ]] || { echo "usage: t3-alpha-deploy [host ...] [--force]" >&2; exit 2; } +[[ ${#HOSTS[@]} -gt 0 ]] || HOSTS=(local emanuel@matebook.lan emanuelfranzen@Mac.lan) +ZIP="$(cd "$(dirname "$ZIP")" && pwd)/$(basename "$ZIP")" INSTALLER="$(cd "$(dirname "$0")" && pwd)/t3-alpha-install" -REMOTE_LOG='~/Library/Logs/t3-alpha-install.log' INFO_PLIST="$(unzip -Z1 "$ZIP" | grep -E '^[^/]+\.app/Contents/Info\.plist$')" VERSION="$(unzip -p "$ZIP" "$INFO_PLIST" | plutil -extract CFBundleShortVersionString raw -)" [[ -n "$VERSION" ]] || { echo "error: cannot read the app version from $ZIP" >&2; exit 1; } -deploy_remote() { # - local host="$1" zip_name before pid result - zip_name="$(basename "$ZIP")" - echo "== $host: copying $zip_name" - scp -q "$ZIP" "$INSTALLER" "$host:/tmp/" || return 1 - before=$(ssh "$host" "cat $REMOTE_LOG 2>/dev/null | wc -l" | tr -d ' ') - pid=$(ssh "$host" "/bin/bash /tmp/t3-alpha-install /tmp/$zip_name --expect-version $VERSION /dev/null"; do sleep 3; done - ssh "$host" "rm -f /tmp/$zip_name /tmp/t3-alpha-install" - result=$(ssh "$host" "tail -n +$((before + 1)) $REMOTE_LOG") - sed 's/^/ /' <<<"$result" - grep -q "SUCCESS:" <<<"$result" +# Runs a shell command on : directly here, over ssh elsewhere. Stdin is +# closed so nothing on the far side can swallow this script's input. +on() { # + if [[ "$1" == local ]]; then + /bin/bash -c "$2" ; sets STATUS + local host="$1" zip="$ZIP" installer="$INSTALLER" before out pid rc tries=0 + if [[ "$host" != local ]]; then + on "$host" true >/dev/null 2>&1 || { STATUS=unreachable; return; } + zip="/tmp/$(basename "$ZIP")" installer=/tmp/t3-alpha-install + scp -q -o BatchMode=yes "$ZIP" "$INSTALLER" "$host:/tmp/" || { STATUS=failed; return; } + fi + before=$(on "$host" 'cat ~/Library/Logs/t3-alpha-install.log 2>/dev/null | wc -l' | tr -d ' ') + out=$(on "$host" "/bin/bash '$installer' '$zip' --expect-version '$VERSION' $FORCE") + rc=$? + sed 's/^/ /' <<<"$out" + if [[ $rc == 3 ]]; then + STATUS=busy + elif pid=$(sed -nE 's/.*\(pid ([0-9]+)\).*/\1/p' <<<"$out") && [[ -n "$pid" ]]; then + # The installer detaches so it survives the app it replaces; wait for it to + # exit, riding out dropped connections (ssh exits 255) for up to 15 minutes. + while ((tries++ < 300)); do + on "$host" "kill -0 $pid 2>/dev/null" + rc=$? + [[ $rc == 0 || $rc == 255 ]] || break + sleep 3 + done + out=$(on "$host" "tail -n +$((before + 1)) ~/Library/Logs/t3-alpha-install.log") + sed 's/^/ /' <<<"$out" + if grep -q "SUCCESS:" <<<"$out"; then STATUS=installed; else STATUS=failed; fi + else + STATUS=failed + fi + [[ "$host" == local ]] || on "$host" "rm -f '$zip' '$installer'" +} -if [[ ${#failed[@]} -gt 0 ]]; then - echo "Failed on: ${failed[*]}. Skipping this Mac." >&2 - exit 1 -fi +# The canary first, then the rest in the order given. +ORDERED=() +for host in "${HOSTS[@]}"; do [[ "$host" == local ]] && ORDERED+=(local); done +for host in "${HOSTS[@]}"; do [[ "$host" == local ]] || ORDERED+=("$host"); done -for host in "${HOSTS[@]}"; do - if [[ "$host" == local ]]; then - echo "== local: installing (T3 Code here restarts; follow ~/Library/Logs/t3-alpha-install.log)" - "$INSTALLER" "$ZIP" --expect-version "$VERSION" +any_failed="" canary_failed="" +for host in "${ORDERED[@]}"; do + if [[ -n "$canary_failed" ]]; then + STATUS=skipped + else + echo "== $host: $(basename "$ZIP")" + deploy "$host" fi + [[ "$host" == local && "$STATUS" == failed ]] && canary_failed=1 + [[ "$STATUS" == failed ]] && any_failed=1 + echo "RESULT $host $STATUS" done +[[ -z "$any_failed" ]] diff --git a/scripts/personal/t3-alpha-install b/scripts/personal/t3-alpha-install index ebdb8dc8e61d..00be01447c7f 100755 --- a/scripts/personal/t3-alpha-install +++ b/scripts/personal/t3-alpha-install @@ -38,12 +38,16 @@ # After a successful install only the backup just taken is kept; older # .bak-*/.failed-* bundles are removed. # +# Install and restart refuse (exit 3, "BUSY: ...") while an agent turn is +# running on this Mac, so an update never cuts off work; --force overrides. +# An agent driving this script from T3 on the same Mac is itself a running turn. +# # Usage: -# t3-alpha-install -arm64.zip> [--expect-version X] -# t3-alpha-install --restart [--expect-version X] # quit + relaunch only -# t3-alpha-install --status # what is running right now -# --expect-version fail (and roll back if installing) unless the running -# instance reports version X +# t3-alpha-install -arm64.zip> [--expect-version X] [--force] +# t3-alpha-install --restart [--expect-version X] [--force] # quit + relaunch only +# t3-alpha-install --status # what is running right now +# --expect-version fail (and roll back if installing) unless the installed +# bundle reports version X set -uo pipefail @@ -71,17 +75,38 @@ if [[ "${1:-}" == "--status" ]]; then exit 0 fi +# Agent turns in progress on this Mac's T3 server. A dead server has none, +# whatever its database says: stale rows are only reconciled on its next start. +busy_turns() { + local pid + pid=$(plutil -extract pid raw "$HOME/.t3/userdata/server-runtime.json" 2>/dev/null) && + kill -0 "$pid" 2>/dev/null || { echo 0; return; } + /usr/bin/sqlite3 -readonly -cmd ".timeout 5000" "$HOME/.t3/userdata/state.sqlite" \ + "SELECT COUNT(*) FROM projection_thread_sessions + WHERE (status = 'running' AND active_turn_id IS NOT NULL) OR status = 'starting';" +} + if [[ "${1:-}" != "--detached" ]]; then - MODE="${1:-}" - if [[ "$MODE" == "--restart" ]]; then - ZIP="" - elif [[ -f "$MODE" ]]; then - ZIP="$(cd "$(dirname "$MODE")" && pwd)/$(basename "$MODE")" - else - echo "usage: t3-alpha-install |--restart [--expect-version X]" >&2 - exit 2 + usage() { echo "usage: t3-alpha-install |--restart [--expect-version X] [--force]" >&2; exit 2; } + ZIP="" RESTART="" EXPECT="" FORCE="" + while [[ $# -gt 0 ]]; do + case "$1" in + --restart) RESTART=1 ;; + --expect-version) EXPECT="${2:-}"; shift ;; + --force) FORCE=1 ;; + *) [[ -f "$1" ]] || usage; ZIP="$(cd "$(dirname "$1")" && pwd)/$(basename "$1")" ;; + esac + shift + done + [[ -n "$ZIP" || -n "$RESTART" ]] || usage + if [[ -z "$FORCE" ]]; then + busy=$(busy_turns) + if [[ "$busy" != 0 ]]; then + echo "BUSY: ${busy:-unknown} agent turn(s) running; nothing changed. --force overrides." + exit 3 + fi fi - nohup /bin/bash "$0" --detached "$ZIP" "${2:-}" "${3:-}" >> "$LOG" 2>&1 & + nohup /bin/bash "$0" --detached "$ZIP" --expect-version "$EXPECT" >> "$LOG" 2>&1 & disown echo "Started in background (pid $!). Watch: tail -f $LOG" echo "The app will restart; this session will die with it if it runs inside T3." diff --git a/scripts/personal/t3-mobile-release b/scripts/personal/t3-mobile-release new file mode 100755 index 000000000000..1896800c5c22 --- /dev/null +++ b/scripts/personal/t3-mobile-release @@ -0,0 +1,47 @@ +#!/bin/bash +# t3-mobile-release — queue EAS production builds of the fork's mobile app for +# iOS and Android, each auto-submitted to TestFlight / Play internal testing. +# +# Skips when nothing the app is built from changed since the last release: the +# `mobile-` tags on origin mark each released commit. Builds run in EAS's +# cloud (--no-wait); this script only queues them and tags HEAD. +# +# Needs the gitignored apps/mobile/credentials/ (submit keys) and an EAS login +# as expomozdom. +# +# Usage: t3-mobile-release [--force] # --force releases even without changes + +set -euo pipefail + +REPO="$(cd "$(dirname "$0")/../.." && pwd)" +cd "$REPO" +export PATH="$REPO/node_modules/.bin:$PATH" + +# The fork's identity; upstream's defaults point at its own EAS project and IDs. +export T3CODE_EAS_OWNER=expomozdom +export T3CODE_EAS_PROJECT_ID=041ec0cd-429a-40d9-8d00-9fcf196ebb59 +export T3CODE_ANDROID_PACKAGE=com.raijustudios.t3code +export T3CODE_IOS_PERSONAL_TEAM=1 +export T3CODE_IOS_PERSONAL_TEAM_BUNDLE_ID=com.raijustudios.t3code + +[[ -d apps/mobile/credentials ]] || { echo "error: apps/mobile/credentials is missing." >&2; exit 1; } + +git fetch -q origin 'refs/tags/mobile-*:refs/tags/mobile-*' +LAST="$(git tag -l 'mobile-*' --sort=-creatordate | head -1)" +if [[ "${1:-}" != --force && -n "$LAST" ]] && + git diff --quiet "$LAST" HEAD -- apps/mobile packages pnpm-lock.yaml; then + echo "UNCHANGED: nothing mobile changed since $LAST." + exit 0 +fi + +vp i +TAG="mobile-$(date +%Y%m%d)" +cd apps/mobile +for platform in ios android; do + eas build --profile production -p "$platform" --non-interactive --no-wait \ + --auto-submit-with-profile personal +done + +git tag -f "$TAG" HEAD +git push -qf origin "refs/tags/$TAG" +echo "QUEUED: iOS and Android builds from $(git rev-parse --short HEAD), tagged $TAG."