chore(personal): unattended-safe releases: busy skip, canary deploy, tagged versions

docs/personal/release-pipeline.md

@@ -23,6 +23,27 @@ Every Mac runs "T3 Code (Alpha)" from `/Applications`, built from this fork. A L
 the desktop app with Settings → Connections → Network access on (`0.0.0.0:3773`, pairing QR).
 Pairing tokens expire in ~5 minutes; mint one right before pairing (`t3 auth pairing create`).
 
+## Nightly and weekly automation
+
+A Temporal worker on studio runs this runbook unattended. It lives in
+[r4iju/temporal-automations](https://github.com/r4iju/temporal-automations) under
+`packages/t3-release` and posts every result to Slack #homelab.
+
+- **Nightly desktop (05:00 JST):** syncs upstream when the merge is clean and typechecks,
+  then builds and deploys if `origin/main` moved since the last `desktop-v*` tag. A conflicted
+  or failing sync is left as a PR for a human, and the night ships `main` without it.
+- **Weekly mobile (Sunday 06:00 JST):** `t3-mobile-release`, which queues EAS builds only when
+  the app's code changed since the last `mobile-*` tag.
+- **Busy Macs are skipped:** a Mac with an agent turn running is left alone and named in
+  Slack. Install it by hand with `--force` once it is idle, or let the next night catch it.
+
+The fork has no CI: its workflows need upstream's Blacksmith runners and never start. The
+nightly gate is local instead: typecheck, a signed build, then studio's own install as the
+canary before any other Mac.
+
+The worker runs from a dedicated worktree, `~/code/t3code-wt-nightly`, detached at
+`origin/main`, so it never touches the main checkout.
+
 ## 1. Sync upstream
 
 Direct pushes to `main` are blocked, so a sync is a PR:
@@ -44,19 +65,24 @@ git -C ~/code/t3code worktree remove ~/code/t3code-wt-sync
 On studio, from a clean `~/code/t3code` on `main`:
 
 ```bash
-scripts/personal/t3-alpha-build 0.0.47
+scripts/personal/t3-alpha-build           # next patch after the latest desktop-v* tag
 scripts/personal/t3-alpha-deploy release/T3-Code-0.0.47-arm64.zip
 ```
 
-- **Version:** the next patch after the last deploy. Every package.json is set to it for the
-  build and restored afterwards, so the app and its server report the same version.
+- **Version:** the next patch after the latest `desktop-v*` tag, or pass one explicitly. A
+  signed build tags its commit `desktop-v<version>` on origin. Every package.json is set to
+  the version for the build and restored afterwards, so the app and its server report the
+  same version.
 - **Signing:** the build is signed with the Apple Development certificate in studio's keychain.
   macOS privacy grants (Screen Recording, Accessibility, …), keychain access and Little Snitch
   rules are tied to that signature, so they carry over between builds. An unsigned build is a
   new app to all of them; the installer refuses one.
-- **Deploy:** copies the zip and installer to matebook and sm-em over SSH, installs, and waits
-  for each result. Studio goes last because its restart ends any T3 session driving the
-  deploy. Name hosts to deploy to a subset: `… .zip emanuel@matebook.lan local`.
+- **Deploy:** installs on studio first as the canary; if that fails and rolls back, no other
+  Mac is touched. Then it copies the zip and installer to matebook and sm-em over SSH and
+  waits for each. Every host ends with a `RESULT <host> installed|busy|unreachable|failed`
+  line. Name hosts to deploy to a subset: `… .zip emanuel@matebook.lan`.
+- **Busy Macs:** a Mac with an agent turn running (including the T3 session driving a manual
+  deploy) reports `busy` and is left alone. `--force` installs anyway and ends those turns.
 - **Target Macs must be logged in:** the installer launches the app in the GUI session.
 - **Failures roll back** to the previous bundle automatically. Each Mac logs to
   `~/Library/Logs/t3-alpha-install.log`; `scripts/personal/t3-alpha-install --status` shows
@@ -76,7 +102,10 @@ T3CODE_IOS_PERSONAL_TEAM=1
 T3CODE_IOS_PERSONAL_TEAM_BUNDLE_ID=com.raijustudios.t3code
 ```
 
-From `apps/mobile`, logged in to EAS as `expomozdom`:
+`scripts/personal/t3-mobile-release` sets them, queues both builds with auto-submit, and tags
+the commit `mobile-<date>`. It skips when nothing under `apps/mobile` or `packages` changed
+since the last tag (`--force` releases anyway). By hand, from `apps/mobile`, logged in to EAS
+as `expomozdom`:
 
 ```bash
 eas build --profile production -p ios --non-interactive --no-wait

scripts/personal/t3-alpha-build

@@ -10,15 +10,31 @@
 # designated requirement is its own cdhash, so TCC grants, keychain ACLs and
 # Little Snitch rules never carried over to the next build.
 #
-# Usage: t3-alpha-build <version>   # -> release/T3-Code-<version>-arm64.zip
+# Each build is tagged `desktop-v<version>` on origin once it is signed, so the
+# tags record which commit every Mac runs. Without a version the build takes the
+# next patch after the latest tag.
+#
+# Usage: t3-alpha-build [version]   # -> release/T3-Code-<version>-arm64.zip
 
 set -euo pipefail
 
-VERSION="${1:?usage: t3-alpha-build <version>}"
 SIGN_IDENTITY="${T3_ALPHA_SIGN_IDENTITY:-Apple Development: Emanuel Franzen (T4J48V44Q2)}"
 REPO="$(cd "$(dirname "$0")/../.." && pwd)"
-ZIP="$REPO/release/T3-Code-$VERSION-arm64.zip"
 cd "$REPO"
+git fetch -q origin 'refs/tags/desktop-v*:refs/tags/desktop-v*'
+if [[ -n "${1:-}" ]]; then
+  VERSION="$1"
+else
+  LAST="$(git tag -l 'desktop-v*' --sort=-v:refname | head -1)"
+  [[ -n "$LAST" ]] || { echo "error: no desktop-v* tag yet; pass a version." >&2; exit 1; }
+  LAST="${LAST#desktop-v}"
+  VERSION="${LAST%.*}.$((${LAST##*.} + 1))"
+fi
+if git rev-parse -q --verify "refs/tags/desktop-v$VERSION" >/dev/null; then
+  echo "error: desktop-v$VERSION is already tagged." >&2
+  exit 1
+fi
+ZIP="$REPO/release/T3-Code-$VERSION-arm64.zip"
 # The artifact build spawns `vp`, which only exists in the repo's node_modules.
 export PATH="$REPO/node_modules/.bin:$PATH"
 unset ELECTRON_RUN_AS_NODE
@@ -58,5 +74,8 @@ sign_app "$APP" 2> >(grep -v 'replacing existing signature' >&2)
 rm -f "$ZIP" "$ZIP.blockmap"
 ditto -c -k --sequesterRsrc --keepParent "$APP" "$ZIP"
 
+git tag "desktop-v$VERSION" HEAD
+git push -q origin "desktop-v$VERSION"
+
 echo "Built and signed $ZIP"
 codesign -d -r- "$APP" 2>&1 | tail -1

scripts/personal/t3-alpha-deploy

@@ -1,57 +1,98 @@
 #!/bin/bash
-# t3-alpha-deploy — install a zip from t3-alpha-build on every Mac. Remote hosts
-# go first over SSH, `local` last: installing here restarts the T3 Code this
-# may be running inside, which ends the session driving the deploy.
+# t3-alpha-deploy — install a zip from t3-alpha-build on every Mac and print one
+# `RESULT <host> <status>` line per host:
+#   installed    running the new version
+#   busy         an agent turn is running there; left alone (see t3-alpha-install)
+#   unreachable  asleep or offline
+#   failed       the install failed and rolled back
+#   skipped      not attempted because the canary failed
 #
-# Each host gets this checkout's t3-alpha-install copied next to the zip, so
-# no Mac carries its own installer copy that can drift.
+# `local` goes first as the canary: if this Mac's install fails, the build is
+# broken and no other Mac is touched. A deploy driven from T3 on this Mac finds
+# `local` busy (the driving turn itself); install it last with
+# `t3-alpha-deploy <zip> local --force`.
 #
-# Usage: t3-alpha-deploy <zip> [host ...]
-#   hosts default to: emanuel@matebook.lan emanuelfranzen@Mac.lan local
+# Remote hosts get this checkout's installer copied next to the zip, so no Mac
+# carries its own installer copy that can drift.
+#
+# Usage: t3-alpha-deploy <zip> [host ...] [--force]
+#   hosts default to: local emanuel@matebook.lan emanuelfranzen@Mac.lan
+#   --force  install even where agent turns are running
+# Exits 1 if any host failed.
 
 set -uo pipefail
 
-ZIP="${1:?usage: t3-alpha-deploy <zip> [host ...]}"
-shift
-HOSTS=("$@")
-[[ ${#HOSTS[@]} -eq 0 ]] && HOSTS=(emanuel@matebook.lan emanuelfranzen@Mac.lan local)
+ZIP="" FORCE="" HOSTS=()
+for arg in "$@"; do
+  case "$arg" in
+    --force) FORCE=--force ;;
+    *) if [[ -z "$ZIP" ]]; then ZIP="$arg"; else HOSTS+=("$arg"); fi ;;
+  esac
+done
+[[ -f "$ZIP" ]] || { echo "usage: t3-alpha-deploy <zip> [host ...] [--force]" >&2; exit 2; }
+[[ ${#HOSTS[@]} -gt 0 ]] || HOSTS=(local emanuel@matebook.lan emanuelfranzen@Mac.lan)
+ZIP="$(cd "$(dirname "$ZIP")" && pwd)/$(basename "$ZIP")"
 INSTALLER="$(cd "$(dirname "$0")" && pwd)/t3-alpha-install"
-REMOTE_LOG='~/Library/Logs/t3-alpha-install.log'
 INFO_PLIST="$(unzip -Z1 "$ZIP" | grep -E '^[^/]+\.app/Contents/Info\.plist$')"
 VERSION="$(unzip -p "$ZIP" "$INFO_PLIST" | plutil -extract CFBundleShortVersionString raw -)"
 [[ -n "$VERSION" ]] || { echo "error: cannot read the app version from $ZIP" >&2; exit 1; }
 
-deploy_remote() { # <host>
-  local host="$1" zip_name before pid result
-  zip_name="$(basename "$ZIP")"
-  echo "== $host: copying $zip_name"
-  scp -q "$ZIP" "$INSTALLER" "$host:/tmp/" || return 1
-  before=$(ssh "$host" "cat $REMOTE_LOG 2>/dev/null | wc -l" | tr -d ' ')
-  pid=$(ssh "$host" "/bin/bash /tmp/t3-alpha-install /tmp/$zip_name --expect-version $VERSION </dev/null" |
-    sed -nE 's/.*\(pid ([0-9]+)\).*/\1/p')
-  [[ -n "$pid" ]] || { echo "   installer did not start"; return 1; }
-  # The installer detaches so it survives the app it replaces; wait for it to exit.
-  while ssh "$host" "kill -0 $pid 2>/dev/null"; do sleep 3; done
-  ssh "$host" "rm -f /tmp/$zip_name /tmp/t3-alpha-install"
-  result=$(ssh "$host" "tail -n +$((before + 1)) $REMOTE_LOG")
-  sed 's/^/   /' <<<"$result"
-  grep -q "SUCCESS:" <<<"$result"
+# Runs a shell command on <host>: directly here, over ssh elsewhere. Stdin is
+# closed so nothing on the far side can swallow this script's input.
+on() { # <host> <command>
+  if [[ "$1" == local ]]; then
+    /bin/bash -c "$2" </dev/null
+  else
+    ssh -o BatchMode=yes -o ConnectTimeout=10 "$1" "$2" </dev/null
+  fi
 }
 
-failed=()
-for host in "${HOSTS[@]}"; do
-  [[ "$host" == local ]] && continue
-  deploy_remote "$host" || failed+=("$host")
-done
+deploy() { # <host>; sets STATUS
+  local host="$1" zip="$ZIP" installer="$INSTALLER" before out pid rc tries=0
+  if [[ "$host" != local ]]; then
+    on "$host" true >/dev/null 2>&1 || { STATUS=unreachable; return; }
+    zip="/tmp/$(basename "$ZIP")" installer=/tmp/t3-alpha-install
+    scp -q -o BatchMode=yes "$ZIP" "$INSTALLER" "$host:/tmp/" || { STATUS=failed; return; }
+  fi
+  before=$(on "$host" 'cat ~/Library/Logs/t3-alpha-install.log 2>/dev/null | wc -l' | tr -d ' ')
+  out=$(on "$host" "/bin/bash '$installer' '$zip' --expect-version '$VERSION' $FORCE")
+  rc=$?
+  sed 's/^/   /' <<<"$out"
+  if [[ $rc == 3 ]]; then
+    STATUS=busy
+  elif pid=$(sed -nE 's/.*\(pid ([0-9]+)\).*/\1/p' <<<"$out") && [[ -n "$pid" ]]; then
+    # The installer detaches so it survives the app it replaces; wait for it to
+    # exit, riding out dropped connections (ssh exits 255) for up to 15 minutes.
+    while ((tries++ < 300)); do
+      on "$host" "kill -0 $pid 2>/dev/null"
+      rc=$?
+      [[ $rc == 0 || $rc == 255 ]] || break
+      sleep 3
+    done
+    out=$(on "$host" "tail -n +$((before + 1)) ~/Library/Logs/t3-alpha-install.log")
+    sed 's/^/   /' <<<"$out"
+    if grep -q "SUCCESS:" <<<"$out"; then STATUS=installed; else STATUS=failed; fi
+  else
+    STATUS=failed
+  fi
+  [[ "$host" == local ]] || on "$host" "rm -f '$zip' '$installer'"
+}
 
-if [[ ${#failed[@]} -gt 0 ]]; then
-  echo "Failed on: ${failed[*]}. Skipping this Mac." >&2
-  exit 1
-fi
+# The canary first, then the rest in the order given.
+ORDERED=()
+for host in "${HOSTS[@]}"; do [[ "$host" == local ]] && ORDERED+=(local); done
+for host in "${HOSTS[@]}"; do [[ "$host" == local ]] || ORDERED+=("$host"); done
 
-for host in "${HOSTS[@]}"; do
-  if [[ "$host" == local ]]; then
-    echo "== local: installing (T3 Code here restarts; follow ~/Library/Logs/t3-alpha-install.log)"
-    "$INSTALLER" "$ZIP" --expect-version "$VERSION"
+any_failed="" canary_failed=""
+for host in "${ORDERED[@]}"; do
+  if [[ -n "$canary_failed" ]]; then
+    STATUS=skipped
+  else
+    echo "== $host: $(basename "$ZIP")"
+    deploy "$host"
   fi
+  [[ "$host" == local && "$STATUS" == failed ]] && canary_failed=1
+  [[ "$STATUS" == failed ]] && any_failed=1
+  echo "RESULT $host $STATUS"
 done
+[[ -z "$any_failed" ]]

scripts/personal/t3-alpha-install

@@ -38,12 +38,16 @@
 # After a successful install only the backup just taken is kept; older
 # .bak-*/.failed-* bundles are removed.
 #
+# Install and restart refuse (exit 3, "BUSY: ...") while an agent turn is
+# running on this Mac, so an update never cuts off work; --force overrides.
+# An agent driving this script from T3 on the same Mac is itself a running turn.
+#
 # Usage:
-#   t3-alpha-install <path/to/T3-Code-<ver>-arm64.zip> [--expect-version X]
-#   t3-alpha-install --restart [--expect-version X]   # quit + relaunch only
-#   t3-alpha-install --status                          # what is running right now
-#   --expect-version  fail (and roll back if installing) unless the running
-#                     instance reports version X
+#   t3-alpha-install <path/to/T3-Code-<ver>-arm64.zip> [--expect-version X] [--force]
+#   t3-alpha-install --restart [--expect-version X] [--force]   # quit + relaunch only
+#   t3-alpha-install --status                                    # what is running right now
+#   --expect-version  fail (and roll back if installing) unless the installed
+#                     bundle reports version X
 
 set -uo pipefail
 
@@ -71,17 +75,38 @@ if [[ "${1:-}" == "--status" ]]; then
   exit 0
 fi
 
+# Agent turns in progress on this Mac's T3 server. A dead server has none,
+# whatever its database says: stale rows are only reconciled on its next start.
+busy_turns() {
+  local pid
+  pid=$(plutil -extract pid raw "$HOME/.t3/userdata/server-runtime.json" 2>/dev/null) &&
+    kill -0 "$pid" 2>/dev/null || { echo 0; return; }
+  /usr/bin/sqlite3 -readonly -cmd ".timeout 5000" "$HOME/.t3/userdata/state.sqlite" \
+    "SELECT COUNT(*) FROM projection_thread_sessions
+     WHERE (status = 'running' AND active_turn_id IS NOT NULL) OR status = 'starting';"
+}
+
 if [[ "${1:-}" != "--detached" ]]; then
-  MODE="${1:-}"
-  if [[ "$MODE" == "--restart" ]]; then
-    ZIP=""
-  elif [[ -f "$MODE" ]]; then
-    ZIP="$(cd "$(dirname "$MODE")" && pwd)/$(basename "$MODE")"
-  else
-    echo "usage: t3-alpha-install <zip>|--restart [--expect-version X]" >&2
-    exit 2
+  usage() { echo "usage: t3-alpha-install <zip>|--restart [--expect-version X] [--force]" >&2; exit 2; }
+  ZIP="" RESTART="" EXPECT="" FORCE=""
+  while [[ $# -gt 0 ]]; do
+    case "$1" in
+      --restart) RESTART=1 ;;
+      --expect-version) EXPECT="${2:-}"; shift ;;
+      --force) FORCE=1 ;;
+      *) [[ -f "$1" ]] || usage; ZIP="$(cd "$(dirname "$1")" && pwd)/$(basename "$1")" ;;
+    esac
+    shift
+  done
+  [[ -n "$ZIP" || -n "$RESTART" ]] || usage
+  if [[ -z "$FORCE" ]]; then
+    busy=$(busy_turns)
+    if [[ "$busy" != 0 ]]; then
+      echo "BUSY: ${busy:-unknown} agent turn(s) running; nothing changed. --force overrides."
+      exit 3
+    fi
   fi
-  nohup /bin/bash "$0" --detached "$ZIP" "${2:-}" "${3:-}" >> "$LOG" 2>&1 &
+  nohup /bin/bash "$0" --detached "$ZIP" --expect-version "$EXPECT" >> "$LOG" 2>&1 &
   disown
   echo "Started in background (pid $!). Watch: tail -f $LOG"
   echo "The app will restart; this session will die with it if it runs inside T3."

scripts/personal/t3-mobile-release

@@ -0,0 +1,47 @@
+#!/bin/bash
+# t3-mobile-release — queue EAS production builds of the fork's mobile app for
+# iOS and Android, each auto-submitted to TestFlight / Play internal testing.
+#
+# Skips when nothing the app is built from changed since the last release: the
+# `mobile-<date>` tags on origin mark each released commit. Builds run in EAS's
+# cloud (--no-wait); this script only queues them and tags HEAD.
+#
+# Needs the gitignored apps/mobile/credentials/ (submit keys) and an EAS login
+# as expomozdom.
+#
+# Usage: t3-mobile-release [--force]   # --force releases even without changes
+
+set -euo pipefail
+
+REPO="$(cd "$(dirname "$0")/../.." && pwd)"
+cd "$REPO"
+export PATH="$REPO/node_modules/.bin:$PATH"
+
+# The fork's identity; upstream's defaults point at its own EAS project and IDs.
+export T3CODE_EAS_OWNER=expomozdom
+export T3CODE_EAS_PROJECT_ID=041ec0cd-429a-40d9-8d00-9fcf196ebb59
+export T3CODE_ANDROID_PACKAGE=com.raijustudios.t3code
+export T3CODE_IOS_PERSONAL_TEAM=1
+export T3CODE_IOS_PERSONAL_TEAM_BUNDLE_ID=com.raijustudios.t3code
+
+[[ -d apps/mobile/credentials ]] || { echo "error: apps/mobile/credentials is missing." >&2; exit 1; }
+
+git fetch -q origin 'refs/tags/mobile-*:refs/tags/mobile-*'
+LAST="$(git tag -l 'mobile-*' --sort=-creatordate | head -1)"
+if [[ "${1:-}" != --force && -n "$LAST" ]] &&
+  git diff --quiet "$LAST" HEAD -- apps/mobile packages pnpm-lock.yaml; then
+  echo "UNCHANGED: nothing mobile changed since $LAST."
+  exit 0
+fi
+
+vp i
+TAG="mobile-$(date +%Y%m%d)"
+cd apps/mobile
+for platform in ios android; do
+  eas build --profile production -p "$platform" --non-interactive --no-wait \
+    --auto-submit-with-profile personal
+done
+
+git tag -f "$TAG" HEAD
+git push -qf origin "refs/tags/$TAG"
+echo "QUEUED: iOS and Android builds from $(git rev-parse --short HEAD), tagged $TAG."