[Tracking] YAML-DSL pivot — recipe-as-code platform (feat/yaml-dsl-pivot)

[yaroher]

[Tracking] YAML-DSL pivot — recipe-as-code platform (feat/yaml-dsl-pivot)

Goal: ship feat/yaml-dsl-pivot as the new main line of stroppy-cloud — the platform
where organizations author infra providers and benchmark workflows as code (YAML DSL
bundles in internal git, edited in an embedded IDE) and end users launch runs through a
form auto-generated from workflow.inputs ⊕ provider.params-schema — replacing the
hardcoded per-DB×topology Go recipes (~5.7 KLOC), enum providers, presets and the wizard.
Temporal stays the execution engine. The bar for merging is data-parity ≥ ref: the
run model changed completely, but everything drawn for a run (runs table, overview,
metrics, logs, grafana, compare, rating, quotas) must not regress below the ref branch.

Definition of done

• Data-parity ≥ ref — every fixed-UI surface (runs table, overview, metrics,
  logs, grafana, compare, rating, favorites, dashboard, quotas) at least as complete
  as ref (Go-side parity guards exist; web-side gaps listed below)
• Engine live end-to-end on the docker provider (real stand, real nomad alloc)
• Engine live end-to-end on a terraform provider (yandex) — never executed
• Authoring surface proven live: embedded IDE + LSP extension observed working in a
  real browser session (protocol-level proof only today)
• Recipe coverage: the DB set from ref re-authored as YAML recipes (postgres-ha is
  the only template today)
• Open gaps below closed or explicitly deferred with rationale
• Full suite green (currently ~790 Go tests / 81 pkgs, vitest, tsc — green)

Where the pivot intentionally drops a ref surface, it is called out under
Conscious non-parity, not silently omitted.

---

Engine (subproject 1 — compiler + flow wiring)

• internal/dsl/* compiler: YAML bundle (cluster.yaml + workflow.yaml +
  components/* + providers/*) → dslpb.CompiledPlan; include/CEL/contract/
  schemapb-schema derivation
• RunRecipeWorkflow + generic Temporal DAG interpreter
  (compile → reserve quotas → provision → execute → commit → teardown → release)
• Provider contract stroppy_nodes / stroppy_machines; builtin docker
  (nomad sidecar, per-node agent token/queue, systemd agent container) — live
• terraform provisioner path (actor bridge, per-run workdir, per-run
  apply/destroy log capture via LogSinkFn, per-tenant deploy creds)
• Quotas remodeled onto machine_groups (reserve-before-provision / commit /
  release-on-teardown)
• Live docker recipe-run GREEN e2e on the dev stand (runs b196f6c0,
  ffd89961, f2e264dc; 7 deploy-gated bugs fixed — agent token+queue wiring,
  systemd boot flags, nomad cgroup/placement/addr, docker runtime params injection)

SP-A — Schema layer (schemapb)

• Provider params schemapb.Schema derived from variables.tf + manifest.yaml
  (types, validation→CEL, secret, defaults)
• workflow.inputs as schemapb; Filled → Baked (hashable sealed snapshot)
• Form-schema composition workflow.inputs ⊕ provider.params — strict, fail-closed
  on undeclared keys (incl. recursive objects and typed-value maps, schemapb v1.6.0)

SP-B — Catalog + tenancy + RBAC

• Provider/workflow as first-class catalog entities; instance + org scopes
• Instance→org live link + implicit fork-on-edit (LINKED→FORKED lineage)
• RBAC gates for authoring / launching / viewing (existing role model)
• Catalog frontend (/admin/catalog, /t/:slug/catalog, lineage badges)
• Builtin catalog seeded at boot + backfilled into pre-existing orgs

SP-C — Internal git + embedded IDE

• Gitea sidecar; one full git repo per catalog entry (providers, workflows,
  recipes); commit-pinned source_ref (git:<owner>/<repo>@<sha>)
• GitBundleStore — LINKED reads through to the instance repo, first edit forks
• Gateway /ide/* → per-workspace code-server; fail-closed authorizer
• Browser SSO: 30s single-use scope-bound ticket → HttpOnly cookie (no second
  login, access JWT never enters a cookie) — verified live incl. forged-cookie 403s
• stroppy-yaml LSP delegating to DslService.Check / ComposedSchema / Preview;
  code-server image bakes stroppy-yaml + terraform-ls + hashicorp.terraform (Open VSX)
• Security hardening: Gitea token never on disk in worktrees (http.extraHeader),
  cross-repo isolation test, off-origin ticket targets rejected
• Catalog/recipe edit page is the embedded IDE (form editor replaced)

SP-D — Generated launch form

• ComposeLaunchFormSchema RPC; schemapb compiled to WASM validates in the browser
  identically to Go (real-hash parity test, schemapb.Hash == engine.hash)
• Launch form renderer (all field kinds, when, computed, enum options); 22 MB WASM
  code-split off the main bundle (verified)
• StartRun accepts Filled, server re-bakes from the recipe's own bundle
  (client-supplied Baked never trusted), field errors funnel back to the form
• Baked threaded → RunRecipeWorkflow → compile; persisted on the run
• Rerun = prefill the form from the stored Baked (owner decision — drift
  surfaces as field errors, nothing launches without an explicit press)
• Live-proven on the stand: form-driven docker run 7ee9e9da COMPLETED with typed
  user inputs reaching placement (on_group="db-b", reps=7)

SP-E — Run model rewrite

• First-class models.Run (Baked + CompiledPlan + topology + per-job status +
  observability refs + rating/compare) in run_records; TestRunRecord and old
  tables deleted
• All readers migrated (overview, compare, rating, dashboard, quota, favorite,
  share); data-parity guard tests over §6 A–F (59 assertions)
• Compiled plan persisted once after compile (Temporal history-bloat guard with
  call-count test)
• Runs table rebuilt (tanstack) with 18-field RunVM; RunDetail audited as a superset
  of ref; db_version on Run.Summary

SP-F — Execution shape-up

• Per-tenant provider deploy creds: Deps.EnvFn(ctx, tenantID) resolved inside
  activities from sealed identity_secrets namespace provider_deploy_cred — no
  secret crosses a boundary Temporal persists
• Per-run terraform stdout/stderr capture; per-run tf workdir (was a constant →
  state collision)
• service.health.timeout → nomad wait-for-healthy; docker teardown recovery after
  control-plane restart (ContainersByNetwork)

---

Open gaps (honest, none hidden)

Blocking-ish (needed before this branch is "the product")

• yandex/terraform never run live. deployments/terraform/yandex/*.tf conforms
  to the module contract now, but a real YC apply needs owner creds
  (YC_TOKEN/YC_CLOUD_ID/YC_FOLDER_ID) + a human watching
• Recipe coverage. Only examples/dsl/postgres-ha + docker builtin exist. The
  ref DB×topology matrix (postgres/patroni-HA, YDB, OrioleDB, …) must be
  re-authored as YAML recipes — deliberate greenfield (§9 migration of 11 DBs was
  cancelled; recipes are rewritten, not ported)
• LSP extension host never observed in a live browser session (protocol-level
  proof only); code-server extension that spawns the LSP binary unverified on a stand
• executeServiceJob is fire-and-forget (submits the nomad job, does not wait for
  alloc healthy) — health must be gated by a WaitStep in the dependent job, or the
  submit path should wait

Hardening / known limits

• IDE ticket single-use is tracked in-process — a restart or second replica
  weakens it to "once per process, within 30s"; needs a shared store for HA
• Ticket rides in the query string (one-shot, 30s) — strip from proxy logs or move
  the exchange to POST
• Docker path has no per-run subprocess log capture (Docker SDK gives no stream;
  documented, deliberately unwired)
• Nomad multi-node service placement impossible on the docker provider (one combined
  nomad node = one node_id) — cloud/tf concern, docker recipes pin services to the
  gateway group
• Scalar-valued tf maps (map(string)/map(number)) fall back to permissive Object
  typing in schemapb (no injection surface, just missing value typing) — upstream
  follow-up
• make protocols does rm -rf internal/proto and nukes hand-written files
  (run_test.go killed and restored 6×) — harden the Makefile
• tests/ integration suite does not compile under -tags integration
  (pre-existing, invisible to go test ./...)
• db_version has no facet/filter on the runs table; web RunConfigTab parity test
  still unwritten; ~44 web/src/lib/proto/**_pb.ts one-line generator-noise diffs

Deferred with rationale (recorded, not forgotten)

• Automatic workload sizing via CEL (needs a cluster-level CEL binding scope) —
  explicit YAML is acceptable for v1
• Marketplace / cross-instance recipe sharing — non-goal for v1

Conscious non-parity (intentional drops — called out so "parity" is honest)

• Wizard / presets / suites — replaced by YAML authoring + the generated launch
  form; "suite" = matrix in workflow.yaml
• Networks as a product entity — networking is the provider tf-module's
  responsibility (owner-confirmed)
• TestRunRecord and the old run flow (test.go stage machine,
  internal/domain/database/* ~5.7 KLOC, presets_seed) — deleted, replaced by
  models.Run + compiled plans
• Single-bundle CodeMirror editor — replaced by the embedded IDE
• No backward compatibility, no diff-tests against the old Go renderer, no coexistence
  behind a flag (owner decision). Invariant kept: the stack does not change (Go,
  connect, Temporal, komeet-style postgres, easyp proto codegen, React, hexagonal
  layout, diag discipline)

---

Background & sources

Branch: feat/yaml-dsl-pivot, forked from ref at 3c115b53; ~50 commits,
709 files changed (+187k / −326k). Live stand: dev.cloud.stroppy.io.

Read in this order:

1. docs/superpowers/specs/2026-07-08-product-vision.md —
   the product north star: domain model (Provider / Workflow / Catalog / Run / RBAC),
   the "only the launch form is generated" principle, SP-A…SP-F decomposition. Start here.
2. docs/superpowers/specs/2026-07-03-yaml-dsl-pivot-design.md —
   the engine spec: why drop hardcoded Go recipes, the DSL, provider contract, Nomad.
3. docs/superpowers/specs/2026-07-03-flow-wiring-design.md —
   wiring the compiler into Temporal, providers, quotas, run lifecycle.
4. Per-subproject specs + plans under
   docs/superpowers/specs/ /
   plans/
   (2026-07-08-sp-{a,b,c,d,e,f}-*.md).
5. .superpowers/sdd/progress.md (in-repo SDD ledger) — task-by-task execution
   record with review verdicts, live-stand proofs, security findings and every gap above
   traceable to its origin.