Repository navigation

CD #342

Workflow file for this run

	name: CD

	# Manual-only. Production deploys happen automatically in the CI merge queue
	# (ci.yaml deploy-prod on merge_group); this workflow covers the rare
	# direct-push-to-main case, which has no merge_group event to deploy. Run it
	# by hand from the Actions tab. (Dropped the 'v*' release-tag trigger — it
	# re-deployed what the merge queue had already shipped.)
	on:
	workflow_dispatch:
	inputs:
	verify-wedding-backup-staging:
	description: Verify the Wedding backup credential and active dedicated archive after deployment
	required: false
	type: boolean
	default: false

	permissions: {}

	# Shared with the CI merge-queue deploy-prod job (ci.yaml) so a manual deploy
	# and a merge-queue deploy can never run against the prod cluster at once.
	#
	# queue: max is load-bearing, not a tidy-up. cancel-in-progress: false protects
	# the deploy that is already RUNNING; it says nothing about the one waiting
	# behind it. Under the default (queue: single) at most one run may be pending,
	# so a third event replaces the second and that deploy is discarded silently —
	# no failure, no annotation, nothing in the run list. queue: max lets up to 100
	# runs queue FIFO instead, so every deploy in a burst reaches the cluster in
	# order. Cancelling an intermediate deploy is exactly what leaves prod in an
	# inconsistent state, and with main behind a merge queue, several changes
	# landing close together is normal rather than exceptional.
	concurrency:
	group: prod-deploy
	cancel-in-progress: false
	queue: max

	jobs:
	# A workflow-dispatched deployment is the recovery path for a direct push to
	# main, so it has no pull_request or merge_group event for ci.yaml to inspect.
	# Validate every persistence-bearing resource in the dispatched revision
	# here. The shared deploy action separately compares removals with the live
	# Flux-owned objects, because a commit parent is not proof that an earlier
	# direct push ever reached production.
	validate-pvc-prune-safety:
	name: 💾 Validate Two-Stage PVC Retirement
	runs-on: ubuntu-latest
	permissions:
	contents: read # checkout repository and its parent
	steps:
	- name: 📑 Checkout
	uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	with:
	persist-credentials: false

	- name: ⚙️ Setup Go
	uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
	with:
	go-version-file: go.mod
	cache: false

	- name: 💾 Validate two-stage PVC retirement
	env:
	PVC_PRUNE_BASE_SHA: ${{ github.sha }}
	run: \|
	shellcheck scripts/tests/test-pvc-prune-safety.sh
	bash scripts/tests/test-pvc-prune-safety.sh

	# Keep this structural check outside the RGD job whose presence and failure propagation it
	# verifies. This independently required prerequisite still catches deletion or suppression of
	# validate-rgd-templates instead of disappearing with the gate it is meant to protect.
	- name: 🔐 Validate nested RGD gate wiring
	run: \|
	shellcheck scripts/tests/test-rgd-template-static-scan-wiring.sh
	bash scripts/tests/test-rgd-template-static-scan-wiring.sh

	# A manual dispatch can deploy a direct main push that never produced a merge-group revision.
	# Run the same production-pin validator here and make it a hard prerequisite of deploy-prod so
	# neither production route can reconcile an unsupported Kubernetes/Talos pairing.
	validate-talos-kubernetes-compatibility:
	name: 🧪 Validate Kubernetes/Talos Compatibility
	runs-on: ubuntu-latest
	permissions:
	contents: read # checkout repository
	steps:
	- name: 📑 Checkout
	uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	with:
	persist-credentials: false

	- name: ⚙️ Setup Go
	uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
	with:
	go-version-file: go.mod
	cache: false

	- name: ✅ Validate Kubernetes against the pinned Talos release
	run: \|
	go test ./scripts/validate-talos-kubernetes-compatibility
	go run ./scripts/validate-talos-kubernetes-compatibility ksail.prod.yaml

	# A workflow dispatch can deploy a direct push that never had a merge-group revision. Run the
	# same nested-RGD behavioral contract before authorization; validate-eks-authorization depends
	# on this job, and deploy-prod already depends on authorization, preserving the signing
	# validator's load-bearing deploy-prod.needs shape while still making this a deployment gate.
	validate-rgd-templates:
	name: 🔬 Validate Nested RGD Workload Templates
	runs-on: ubuntu-latest
	permissions:
	contents: read # checkout repository
	steps:
	- name: 📑 Checkout
	uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	with:
	persist-credentials: false

	- name: ⚙️ Setup Trivy for nested RGD templates
	uses: aquasecurity/setup-trivy@81e514348e19b6112ce2a7e3ecbafe19c1e1f567 # v0.3.1
	with:
	version: v0.74.0
	cache: true

	- name: 🔬 Validate nested RGD workload templates
	run: \|
	shellcheck scripts/scan-rgd-templates.sh scripts/tests/test-rgd-template-static-scan.sh
	bash scripts/tests/test-rgd-template-static-scan.sh

	# The merge-queue deploy in ci.yaml cannot run without this validator passing,
	# so leaving it out here made the manual path the WAY AROUND the gate: a direct
	# push to main plus a dispatch reconciled the manifests with no authorization
	# check at all. Deliberately a separate job rather than a step folded into the
	# shared deploy-prod action — that action runs inside the prod environment with
	# deploy secrets in scope, and a gate belongs before that, not inside it.
	validate-eks-authorization:
	name: 🔐 Validate Production Authorization
	needs: [validate-pvc-prune-safety, validate-rgd-templates]
	runs-on: ubuntu-latest
	permissions:
	contents: read # checkout repository
	steps:
	- name: 📑 Checkout
	uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	with:
	persist-credentials: false

	- name: 🔐 Enforce approved publication signers
	env:
	APPROVED_REVISIONS_ENFORCE: '1'
	run: ./scripts/guard-publish-workflow-approved-revisions.sh

	- name: ⚙️ Setup Go
	uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
	with:
	go-version-file: go.mod
	cache: false

	- name: ⚙️ Install pinned kubectl renderer
	shell: bash
	env:
	KUBECTL_VERSION: "v1.36.2"
	KUBECTL_SHA256: "1e9045ec32bea85da43de85f0065358529ea7c7a152eca78154fba5b58c27d82"
	run: \|
	set -euo pipefail
	kubectl_path="$RUNNER_TEMP/kubectl"
	curl -fsSL \
	"https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
	-o "$kubectl_path"
	printf '%s %s\n' "$KUBECTL_SHA256" "$kubectl_path" \| sha256sum --check
	sudo install "$kubectl_path" /usr/local/bin/kubectl
	kubectl version --client

	- name: 🔐 Validate static production authorization controls
	# Both commands run unconditionally, and the step fails if either did.
	# They are independent outputs over the same surface: `go test` is the
	# verdict, `go run` is the only caller of the moved-entry diagnostics. A
	# real aggregate mismatch fails BOTH, so short-circuiting on the first
	# left the one failure that most needs explaining with no entry names at
	# all (#3879). Covered by scripts/tests/test-authorization-gate-diagnostics.sh.
	run: \|
	set -euo pipefail
	status=0
	go test ./scripts/validate-eks-ci-role-policy \|\| status=1
	go run ./scripts/validate-eks-ci-role-policy . \|\| status=1
	exit "$status"

	- name: ⚙️ Setup KSail for effective authorization validation
	shell: bash
	env:
	# renovate: datasource=github-releases depName=devantler-tech/ksail extractVersion=^v(?<version>.+)$
	KSAIL_VERSION: "7.198.0"
	GITHUB_TOKEN: ${{ github.token }}
	run: .github/scripts/setup-ksail.sh

	- name: ⚖️ Read Kyverno CLI version for chart admission
	id: chart-kyverno-version
	run: bash .github/scripts/kyverno-version.sh

	- name: ⚖️ Install Kyverno CLI for chart admission
	uses: kyverno/action-install-cli@fcee92fca5c883169ef9927acf543e0b5fc58289 # v0.2.0
	with:
	release: ${{ steps.chart-kyverno-version.outputs.release }}

	- name: 🔐 Validate isolated chart child namespaces
	# Manual recovery deploys may not bypass the rendered namespace proof
	# simply because the staged component is absent from the deploy overlay.
	timeout-minutes: 10
	run: \|
	shellcheck scripts/tests/test-isolated-chart-namespace-rules.sh
	bash scripts/tests/test-isolated-chart-namespace-rules.sh

	- name: 🔐 Validate Helm-rendered production authorization controls
	# Same reasoning as the identical step in ci.yaml: this reaches every
	# upstream chart registry, so an unreachable registry would otherwise
	# stall the step for the job's full budget instead of failing promptly.
	timeout-minutes: 10
	run: \|
	shellcheck scripts/tests/test-production-authorization-rules.sh
	bash scripts/tests/test-production-authorization-rules.sh
	ksail --config ksail.prod.yaml workload validate

	# Same reasoning as the authorization gate above, one property along. The
	# ordering contract — publish to an immutable staging reference, sign and
	# attest it, only then promote those bytes to latest — was wired into ci.yaml,
	# which triggers on pull_request and merge_group. This workflow has neither
	# event, so the direct-push recovery path reached production without the
	# contract being examined once, and the merge-queue gate was the only thing
	# enforcing it. A gate one route can walk around is not a gate.
	validate-publication-contract:
	name: 🖋️ Validate Publication Contract
	runs-on: ubuntu-latest
	# The pin guard gates the deploy THROUGH this job rather than by joining
	# deploy-prod's own `needs`. Two reasons, both load-bearing: this job may run
	# only one command step (see the guard's own job below), and deploy-prod's
	# `needs` line is the literal that validate-dr-signing's ablations mutate — a
	# third entry wraps it past the formatter's width and those ablations silently
	# stop biting, which their "ablation changed nothing" failure caught. Depending
	# on it here keeps that line byte-identical and still blocks the deploy.
	needs: [validate-shared-publish-pin]
	permissions:
	contents: read # checkout repository
	steps:
	- name: 📑 Checkout
	uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	with:
	persist-credentials: false

	- name: ⚙️ Setup Go
	uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
	with:
	go-version-file: go.mod
	cache: false

	- name: 🖋️ Validate DR signing contract
	# Identical invocation to ci.yaml's, deliberately: the two routes to
	# production must be gated by the same check, or they drift and the
	# weaker one becomes the real policy.
	run: \|
	go test ./scripts/validate-dr-signing
	go run ./scripts/validate-dr-signing .github/workflows/dr-rebuild.yaml ksail.prod.yaml

	# Deliberately its OWN job rather than a second step in the contract job above.
	# scripts/validate-dr-signing requires validate-publication-contract to run
	# exactly one command step, because any other step in that job could export
	# BASH_ENV or a fake `go` through $GITHUB_ENV/$GITHUB_PATH and leave the
	# validator looking correct while it was subverted. Adding this guard there
	# broke that invariant, which is the invariant doing its job — so it gets its
	# own job and gates the deploy through `needs` instead.
	#
	# ci.yaml runs this guard on `pull_request` and `merge_group` only, so a matcher
	# widened by a DIRECT PUSH to main reaches production through this manual route
	# without it ever having run — the weakened trust policy would be reconciled to
	# the cluster unchecked (#2818).
	validate-shared-publish-pin:
	name: 🔐 Validate Shared Publish Workflow Pins
	runs-on: ubuntu-latest
	permissions:
	contents: read # checkout repository
	steps:
	- name: 📑 Checkout
	uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	with:
	persist-credentials: false

	- name: 🔐 Validate shared publish workflows are pinned
	run: ./scripts/guard-shared-publish-workflow-pin.sh

	# The apps-base component gate and FANOUT_NAMESPACES must agree, or the
	# post-reconcile reassertion in deploy-prod (--reuse-runtime-proof, no
	# exemption) marks the fan-out incomplete and FAILS THE DEPLOY (#3476).
	#
	# This is the manual recovery path for a direct push to main, which is exactly
	# the case ci.yaml cannot cover: it gates the guard on `pull_request`, and
	# validate-main.yaml runs it asynchronously AFTER a push, so it gates nothing.
	# Without this job a dispatched deploy reconciles an apps-base/fan-out
	# disagreement to production and the guard's verdict arrives too late to stop
	# it — the same way leaving validate-eks-authorization out would have made this
	# route the way around that gate.
	validate-ghcr-fanout-component-gate:
	name: 🔗 Validate GHCR Fan-out Component Gate
	needs: [validate-root-verify]
	runs-on: ubuntu-latest
	permissions:
	contents: read # checkout repository
	steps:
	- name: 📑 Checkout
	uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	with:
	persist-credentials: false

	- name: 🔗 Validate the GHCR fan-out follows the apps-base component gate
	run: \|
	shellcheck scripts/guard-ghcr-fanout-component-gate.sh
	shellcheck scripts/tests/test-guard-ghcr-fanout-component-gate.sh
	bash scripts/guard-ghcr-fanout-component-gate.sh
	bash scripts/tests/test-guard-ghcr-fanout-component-gate.sh

	# A manual dispatch can deploy a direct main push without a merge-group
	# revision. Validate the root-source recovery path on that exact revision.
	# The fan-out gate depends on this job, and deploy-prod depends on fan-out.
	validate-root-verify:
	name: 🛡️ Validate Root Source Repair
	runs-on: ubuntu-latest
	permissions:
	contents: read # checkout repository
	steps:
	- name: 📑 Checkout
	uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	with:
	persist-credentials: false

	- name: 🛡️ Validate the root source repair before deployment
	run: bash scripts/tests/test-reassert-flux-root-verify.sh

	deploy-prod:
	name: 🚀 Deploy to Production
	needs:
	[
	validate-eks-authorization,
	validate-publication-contract,
	validate-talos-kubernetes-compatibility,
	validate-ghcr-fanout-component-gate,
	]
	runs-on: ubuntu-latest
	environment: prod
	permissions:
	contents: read # checkout repository
	packages: write # push OCI artifacts to GHCR
	id-token: write # mint OIDC token for cosign keyless signing (Fulcio + Rekor)
	attestations: write # write SBOM + SLSA provenance attestations (see deploy-prod action)
	actions: read # read a prior run attempt status, to prove an orphaned fence holder is dead (deploy-prod recover-orphaned-fence)
	steps:
	- name: 📑 Checkout
	uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
	with:
	persist-credentials: false

	- name: 🚀 Deploy to Production
	# The deploy logic (push → sign → attest → reconcile → cluster update)
	# is shared with ci.yaml's merge-queue deploy via this composite action,
	# so the two paths can never drift. Secrets are passed as inputs because
	# composite actions cannot read `secrets` directly.
	uses: ./.github/actions/deploy-prod
	with:
	recovery-source-sha: ${{ github.sha }}
	sops-age-key: ${{ secrets.SOPS_AGE_KEY }}
	verify-wedding-backup-staging: ${{ inputs.verify-wedding-backup-staging }}
	kube-config: ${{ secrets.KUBE_CONFIG }}
	talos-config: ${{ secrets.TALOS_CONFIG }}
	ghcr-token: ${{ secrets.GHCR_TOKEN }}
	hcloud-token: ${{ secrets.HCLOUD_TOKEN }}
	wg-server-private-key: ${{ secrets.WG_SERVER_PRIVATE_KEY }}
	# Release a synchronization Lease that an already-dead deploy left
	# held, so one crashed run cannot wedge every later deploy and its own
	# heal. This relaxes no guard: the composite still refuses unless no
	# other fence is held, the heartbeat has stopped, and the holder's run
	# attempt is proven terminal via the API (#3343).
	recover-orphaned-fence: "true"

Navigation Menu

Repository navigation

CD #342

Workflow file

CD #342

Uh oh!

Workflow file for this run