Piccolo.Specs Phase 1b — @problem_template parametric typing (templates become types)

[aarontrowbridge]

DRAFT — stacked on #258. Please review #258 first; this targets wip/piccolo-specs-phase1, not main. Rebase to main once #258 lands.

Phase 1b discharges the deferral contract in src/specs/CONTEXT.md: the @problem_template parametric-type rewrite. This is the poka-yoke core of the initiative — where compatibility rules stop being runtime checks and become facts of the type system.

Full Pkg.test(): 2717 pass (from 2498), same 6 pre-existing broken. See "Test accounting" below for the 1 fail / 8 errors — both are environmental and neither is code.

What is now true at the type level

• SmoothPulseProblem / SplinePulseProblem / BangBangPulseProblem are types (constrained aliases, the Vector{T} = Array{T,1} pattern) — isa Type == true, isa Function == false.
• SmoothPulseProblem{UnitaryTrajectory{LinearSplinePulse,…}} is a TypeError. The alias bound is the pulse × trajectory compatibility matrix. The wrong combination has no spelling.
• A Frozen state-dependent term reaching build is a MethodError by construction — no method of build_term accepts Frozen, for any tag or term name. @problem_template additionally refuses, at declaration time, a state_dependent term whose params field is typed wider than Live. This promotes a hand-written validation convention to a type; the frozen-drift burn class is gone rather than guarded.
• R_ddu on a bang-bang or spline template is an ArgumentError from the constructor and forbidden in the emitted JSON Schema, from the same declaration.
• QuantumControlProblem{T<:AbstractProblemTemplate, QT} <: AbstractQuantumControlProblem (1b owns that supertype, as CONTEXT.md reserved).

Call-surface neutrality held

Zero template tests were edited when the three base templates were redefined through the macro (543/0/0 on the template subset). The generated constructor passes user kwargs through verbatim to the existing builder, so the params struct is a validation-and-retention artifact, not a re-plumbing. Wrapper call sites are the sanctioned exception (spec success criterion 7) and did change — 25 isa assertions now pin the template tag too, which they could not before.

Two design calls worth a reviewer's attention

1. MinimumTimeProblem stays a function — deliberately NOT a wrapper type. Min-time is a recipe over the composition axes (a time objective + a final-fidelity constraint on one flat NLP). Spec success criterion 10 requires the recipe and its hand-factored form (goal_treatment="both" + free_dt + a time objective) to share a structure_hash. A MinimumTimeProblem{...} type would give two spellings of one NLP with the same structure_hash but different Julia types, breaking the "same structure_hash ⇒ same concrete types" invariant that the precompile workload and warm-worker routing rest on. Rationale is in a comment at the method. Only SamplingProblem became a parametric wrapper type.

2. Wrappers stay hand-registered (resolves the spec's open question). @problem_template earns its keep because a template's alias bound encodes a compatibility matrix emit_schema must mirror. A wrapper is polymorphic in what it wraps and has no such matrix, so a @problem_wrapper macro would generate ~4 lines and buy no anti-drift property. SamplingParams still gets its registry params through the same reflect_params.

Cross-repo impact

• problemspec.oss.schema.json changed additively: the control branch's allOf goes 4 → 7 (per-template conditionals + a free_phase/global_names branch). Verified strictly additive — every original branch present unaltered, everything outside allOf byte-identical. Only newly-invalid specs are those setting R_ddu/ddu_bound on a non-smooth template, which was never valid (silently ignored before, rejected now).
• Hash sidecars are byte-identical — hashes cover canonical spec data, not the schema — so amicode's and Prova's hash pins are unaffected. The vendored schemas do need re-staging downstream.

A latent flake this fixed

registries.jl's test item registered :MyTmpl into the process-global TEMPLATES, and TestItemRunner's reused workers carried it into the schema tests — putting a stray name into emit_schema's template enum. The drift gate was passing or failing on worker scheduling. The test item now cleans up, and _json_type_of gained a Symbol method plus an Any → nothing fallback (the registry API accepts a type name like :Float64, not only a Type), so uninterpretable entries are skipped rather than fatal.

Test accounting (the 9 non-passes)

• 8 errors — all ArgumentError: Package QuantumToolbox not found in ext/. QuantumToolbox is a test-only extra that cannot resolve here (needs LinearSolve 2.0–3.87; the manifest pins 5.1.0). No commit in this PR touches ext/.
• 1 fail — Aqua "stale dependencies", an artifact of a local Pkg.add of CairoMakie/Aqua/JET to make the numbers comparable; that was reverted, and the committed Project.toml adds only PrecompileTools.

Known follow-ups (not in this PR)

• Intonato.PulseTuningProblem.qcp must widen to AbstractQuantumControlProblem — it will break on a SamplingProblem. Blocked on this merging, since the symbol doesn't exist before it.
• RegistryEntry.factory widened Function → Union{Function,Type} (a template factory is now a type alias, and Type <: Function is false).
• Precompilation ~16s → ~65s one-time from the type-universe workload (steady-state load measured 6.5s). Trim type_universe() coverage if that becomes painful before the sysimage lands.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 91.51847% with 62 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/specs/extract.jl	82.64%	21 Missing ⚠️
src/control/problems.jl	77.58%	13 Missing ⚠️
src/control/templates/problem_template.jl	92.92%	8 Missing ⚠️
src/control/templates/spline_pulse_problem.jl	60.00%	8 Missing ⚠️
src/control/objectives.jl	96.72%	6 Missing ⚠️
src/specs/schema.jl	96.31%	6 Missing ⚠️

📢 Thoughts on this report? Let us know!

[jack-champagne]

Not landing this for the 07-29 Pasqal hackathon. Reasoning below, plus some things I verified rather than assumed — a couple of which correct my own earlier reads.

Why not for the hackathon

A breaking Julia release has a long tail. Registration goes through JuliaRegistrator → a General registry PR → ~25 min automerge → TagBot. That's not a same-day operation, and a breaking bump means every downstream consumer needs a compat decision. v1.21.0 (#252 + #262 + #258) is registering now and that's already the third Piccolo release today.

Attendees can't see it anyway. amicode's shipped packages/extension/julia/Project.toml now pins Piccolo = "~1.19" (amicode#223) and the manifest pins 1.19.0 exactly. Everything in the Pasqal golden path was vetted against that. So landing this buys the hackathon nothing and risks a resolver surprise.

The breaking surface is bigger than a loud minor

Four genuine API breaks, not one:

break	who it hits
Templates become types (isa Function == false)	anything dispatching on ::Function or testing isa Function
QuantumControlProblem gains a 2nd type parameter	any explicit QuantumControlProblem{QT} — one param becomes two
SamplingProblem stops being <: QuantumControlProblem	any ::QuantumControlProblem receiving a sampling problem
RegistryEntry.factory widens Function → Union{Function,Type}	registry consumers

Contrast #252, which also broke API but was defensibly minor because verify had existed for a matter of hours with zero callers across 11 repos. This one changes the central type of the package. Intonato hit exactly the ::QuantumControlProblem case — survivable only because Intonato.jl#19 parameterized PulseTuningProblem on the problem type first. A downstream package we don't control has no such fix waiting.

So I'd suggest 2.0.0 with a real ## Breaking changes section, rather than folding it into a routine minor.

What I verified

The R_ddu schema claim holds. I initially read additionalProperties and concluded the schema would not forbid R_ddu on a non-smooth template. Running a real Draft-7 validator against the emitted schema refutes that:

spec	result
SplinePulseProblem, no R_ddu	ACCEPTED
R_ddu on SplinePulseProblem	REJECTED
R_ddu on SmoothPulseProblem	ACCEPTED

The mechanism is allOf[5].then.properties.problem.allOf[0].not = {"required": ["R_ddu"]} — an explicit negation, not additionalProperties, which is the right idiom and gives a readable error. My mistake; the PR body was correct.

The schema growth is strictly additive, as claimed: control-branch allOf 4 → 7, every original branch present verbatim, everything outside allOf byte-identical, other oneOf branches untouched. The three new branches are per-template parameter constraints for BangBang / Smooth / Spline.

The lower testitem count is staleness, not thin coverage. 396 here vs 405 on main, because this branch predates the guard testitems added to #258, #262's weight tests, and #252's verify tests. Its own strongest signal stands: zero template tests edited while all three templates were redefined through the macro.

Where the value actually lands

For agents, #258 already shipped ~90% of it: wire format, parser, structured errors with field paths, hashes, registries, schema + drift gate, runner. #259's agent-facing delta is one class of error — per-template parameter validity — moving from a Julia round-trip into amicode's local ajv pass. Real, narrow.

For Julia callers, this is where the actual guarantee lives: compile-time impossibility instead of runtime rejection. That's the better engineering and the reason to land it properly rather than quickly.

Practical state

CONFLICTING/DIRTY across 10 paths, all semantic rather than textual:

• src/control/problems.jl — this branch's parametric rewrite vs Unify validate_solution into verify: one verification surface, tri-state verdict #252's verify rewrite, same file
• all of src/specs/* — this branch is built on pre-squash Piccolo.Specs Phase 1 — declarative ProblemSpec wire format + interpreter (call-surface neutral) #258, so it conflicts with the squashed result plus the omitted-block guard commit that went in with it
• the emitted schema JSON — both sides regenerate it

It also grew from +2813/−299 (29 files) to +7348/−224 (47 files) over the course of the evening, so it's still expanding. That's a rebase only you can adjudicate — I'm deliberately not attempting it.

One ordering constraint worth protecting

The Piccolissimo Specs registration half (register_all_piccolissimo! + the FULL schema emitter) was never pushed and is gone — no branch, no stash, no worktree, and problemspec.full.schema.json exists on no ref. amicode's vendored problemspec.schema.json is the only surviving copy, which makes it a byte-compare oracle for rebuilding it.

That oracle only works against #258's schema. This PR grows it 6932 → 9036 bytes, so the moment this lands the rebuild becomes unverifiable. Worth doing the Piccolissimo half first, then rebasing this onto it.

[jack-champagne]

Hey the ci is not passing, i'd like to get this in but this large of a breaking change should come with the tests to support and passing ci. ping me when its ready to rip and we can get this merged and this release cut

[jack-champagne]

🤖 Nightly Rebase Bot: Conflicts detected during rebase. An agent-attempted resolution is available on rebase/aarontrowbridge/wip/piccolo-specs-phase1b. Please review before merging.

[aarontrowbridge]

P1 rebase complete — the branch is rebased onto current main (c1448e7 + #312) and CI is green (all three Julia versions, Formatter, Documentation; the workflow_dispatch rerun at 4172f8a covers the dead pull_request event at that head).

What the rebase reconciled beyond pure replay:

• SamplingProblem structure preservation (SamplingProblem coverage: state-type bases, structure preservation, verification #270): ported the derivative-chain preservation (step 2b + DerivativeIntegrator replication) into the Phase-1b constructor — the branch had a TODO-then-silent-drop where main preserves :du/:ddu and the base regularizers.
• Wrapper dispatch widening: , , , , now dispatch on — Phase 1b's wrappers previously MethodError'd against every -typed entry point.
• ** restored** (dropped in the rebase conflict resolution; defined nowhere on the raw rebase).
• ** + template params:** two real gaps the Phase-1b loud-kwarg-validation surfaced — the fields existed as builder kwargs on main but not in the params structs, which now ARE the keyword truth. required porting Cubic spline templates: make bending-energy regularization the default smoothness path (port HermiteBendingEnergyRegularizer into Piccolo) #309's into (wholesale from main — pure addition) since the branch still imported it from Piccolissimo.
• ** round-trip:** params comparison now restricted to the spec-carried subset () — per-template extras like are dropped by design on extract, and 's injection is legit-but-non-canonical; the old comparison flagged both as violations. Round-trip test passes.
• Schema regen: the OSS schema JSON carries Phase 1b's per-template defaults.

The mergeable: CONFLICTING is a stale GitHub computation (base hasn't moved since the rebase); the diff is the six-commit Phase-1b arc on top of current main. Ready for review — this is the 'templates become types' foundation the extraction slices build on.