Add Vally-first Copilot on Rails evaluation system

[Alex Weininger (alexweininger)]

Summary

Adds a Vally-first evaluation and release-evidence system for Copilot on Rails.

• Defines a stratified 20-scenario corpus covering frontend, API, worker, data, auth, language, and framework combinations.
• Runs matched rails and baseline-controlled arms with the same model, scenario, endpoint, and attempt.
• Executes generated code only inside hardware-isolated ACA Sandboxes.
• Applies deterministic artifact, build, generated-test, integration, runtime, browser, accessibility, persistence, worker, debugger, deployment, security, provenance, and cleanup gates.
• Adds versioned release thresholds, paired reliability/cost reporting, and fail-closed evidence aggregation.
• Adds daily/weekly CI tiers with bounded concurrency, exact-owner cleanup, and durable artifacts.
• Adds a hand-authored grader-certification project plus one-fault mutations so the graders themselves are tested.
• Documents how teammates can run, extend, diagnose, and use the evaluation system to make product changes.

This is a stacked draft targeting feat/CoR and does not modify the existing PR #1615.

What the full suite runs

The standard suite has three layers:

1. Deterministic contracts: schemas, generated Vally specs, release policy, unit tests, offline graders, and one-fault mutations. No model calls or Azure resources.
2. ACA grader certification: a known-good project plus controlled failures exercising real build, test, runtime, browser, accessibility, persistence, debugger-readiness, and cleanup behavior. ACA usage, but no model calls.
3. Paired Vally E2E: normal Copilot on Rails and a controlled generic-Copilot baseline. Each pair uses the same model, scenario, endpoint, attempt, and validators.

The paired Vally experiment already runs both Rails and baseline. Do not run the standalone baseline command in addition unless debugging baseline behavior.

One-time teammate setup

Prerequisites

• Node.js 22
• npm 11.11.1
• Azure CLI
• Organizational Microsoft Entra account
• Access to the team's ACA Sandbox Group
• GitHub token authorized for the Copilot endpoint used by the evaluator

npm install --global npm@11.11.1
npm ci

Install and authenticate the ACA CLI

Linux/macOS:

curl -fsSL https://aka.ms/aca-cli-install | sh

Windows PowerShell:

irm https://aka.ms/aca-cli-install-ps | iex

This same install path is also used inside sandboxes and containers for agent-driven self-installs.

Only open interactive sign-in when cached authentication is absent:

aca --version
az account show -o none 2>/dev/null || az login
aca auth status >/dev/null 2>&1 || aca auth login
aca doctor

aca doctor must be green before running ACA tiers.

An administrator can grant a teammate access to the shared group:

aca sandboxgroup role create \
  --role "Container Apps SandboxGroup Data Owner" \
  --principal-id "$(az ad user show --id <teammate-upn> --query id -o tsv)"

To create a separate evaluation group:

az account show -o none 2>/dev/null || az login
aca auth status >/dev/null 2>&1 || aca auth login
aca sandboxgroup create --name <group> --location <region> --set-config
aca doctor

The group creator already receives Data Owner. --set-config is required so subsequent evaluator commands resolve the group.

The evaluator uses checked-in declarative manifests. For new manifests, the reproducible CI/CD flow is:

aca sandbox init
aca sandbox validate --file sandbox.yaml
aca sandbox apply --file sandbox.yaml

The manifest pattern is the recommended CI/CD and reproducibility path. Use aca sandbox schema for editor autocomplete.

Local environment

export GH_TOKEN="<Copilot-authorized GitHub token>"
export COR_EVAL_OWNER_ID="yourname-local"

COR_EVAL_OWNER_ID must be lowercase alphanumeric/hyphen text and at most 63 characters. Never commit tokens or generated result workspaces.

Run the complete local suite

1. Deterministic contracts

npm run build:check
npm run lint
npm run eval:cor:thresholds:validate
npm run eval:cor:spike:dry
npm run eval:cor:graders:certify
npm run eval:cor:vally:native:check
npm run eval:cor:vally:native:lint
npm run eval:cor:vally:native:oracle
npm run eval:cor:vally:native:test
npm run eval:cor:vally:native:pilot:dry

2. Real ACA grader certification

aca sandbox validate --file evals/sandbox.yaml
aca sandbox validate --file evals/sandbox-python.yaml
aca sandbox validate --file evals/sandbox-dotnet.yaml
npm run eval:cor:graders:certify:aca

Stop if grader certification fails; model/product results should not be interpreted while an oracle is uncertified.

To diagnose one case:

npm run eval:cor:graders:certify:aca -- --case golden-local-runtime

3. Paired Rails and baseline E2E

Inspect the exact plan before making paid calls:

npm run eval:cor:vally:native:pilot:dry

Run the primary-model compatibility pilot:

npm run eval:cor:vally:paid:pilot:gpt-5-6-sol

This command runs two scenarios x two arms x one attempt = four paid trials. Equivalent aliases exist for claude-sonnet-5 and gpt-5.4-mini.

Aggregate durable evidence:

npm run eval:cor:vally:native:report -- \
  --experiment-dir evals/results/vally-native/compatibility-pilot-gpt-5-6-sol \
  --output evals/results/vally-native-report

Review:

• report.md
• vally-native-report.json
• experiment-input-manifest.json
• Per-trial artifacts/native-summary.json
• Per-trial artifacts/cor-validation.json
• Per-trial artifacts/validation-manifest.json

candidate means all configured gates passed. hold means complete evidence contains a gate failure. insufficient_evidence means required coverage or proof is missing and is not a passing result.

Cleanup verification

aca sandbox list -l "owner-id=$COR_EVAL_OWNER_ID" -o json

The result must be empty. If diagnostic state must be preserved before destructive cleanup:

aca sandbox snapshot --id "$SANDBOX_ID" --name <diagnostic-snapshot>
aca sandbox delete --id "$SANDBOX_ID" --yes

Never use a broad selector or delete another run's sandboxes.

CI setup

Workflow: .github/workflows/copilot-on-rails-evals.yml

Repository variables:

• COR_EVAL_DAILY_ENABLED=true
• COR_EVAL_WEEKLY_ENABLED=true
• COR_EVAL_RESOURCE_GROUP
• COR_EVAL_SANDBOX_GROUP
• COR_EVAL_REGION

Repository secrets:

• COR_EVAL_AZURE_CLIENT_ID
• COR_EVAL_AZURE_TENANT_ID
• COR_EVAL_AZURE_SUBSCRIPTION_ID
• COR_EVAL_COPILOT_GITHUB_TOKEN

The Azure identity needs Container Apps SandboxGroup Data Owner on the configured group.

# Offline PR-equivalent checks
gh workflow run copilot-on-rails-evals.yml -f tier=contracts

# Paid primary-model pilot plus ACA grader certification
gh workflow run copilot-on-rails-evals.yml -f tier=daily

# Paid three-model representative suite plus ACA grader certification
gh workflow run copilot-on-rails-evals.yml -f tier=weekly

Daily is four paid trials. Weekly is 48 paid trials. A local release experiment is 120 paid trials. CI caps concurrency at two, retains evidence for 30 days, and runs exact-owner cleanup even after failures.

The release dispatch currently fails closed before paid calls because provenance-bound real VS Code breakpoint evidence and explicitly authorized live-deployment evidence are not yet wired into that workflow.

Extending and acting on the evaluations

See evals/CONTRIBUTING.md for:

• Adding or changing scenario contracts.
• Regenerating Vally specs.
• Adding graders, stable failure codes, golden cases, and one-fault mutations.
• Distinguishing product, harness, and infrastructure failures.
• Reproducing an exact model/scenario/arm/attempt.
• Turning a product failure into an agent/reference/workflow fix and regression test.
• Expanding from one paired rerun to pilot, representative, and release tiers.

Validation completed

• TypeScript build check
• ESLint
• Workflow YAML parse
• Versioned release-threshold validation
• 20-scenario dry-run validation
• Grader certification: 18/18 offline cases
• Grader certification: 6/6 ACA cases
• Focused evaluator contracts: 87 passing
• Vally-native suite: 67 passing
• Strict Vally lint
• Vally authoritative oracle
• Vally generation drift check
• Exact-owner cleanup check: zero remaining evaluation sandboxes

Initial evidence and limitations

The corrected three-model compatibility pilot produced six matched pairs:

• Rails: 1/6
• Controlled baseline: 0/6

The system therefore currently recommends insufficient_evidence, not release. The strongest recurring product gap is the React + Functions + PostgreSQL journey. Current real breakpoint evidence predates the latest provenance contract, and live deployment remains explicitly authorization-gated.