Initial authoring and evaluation date: 16 September 2026. This record describes observed checks, their scope, and their limits. It does not establish comparative superiority or protection of a production application.
This is the historical validation record for the initial publication at
22ff51dbe846a5b3aec9a15165f3be12e3d714b2.
The subsequent instruction and schema revision has a separate
ambiguity review and validation record. Do not treat the
original agent runs as evaluations of instructions written afterward.
- All 12 skills passed the skill creator's
quick_validate.pycheck for skill naming and frontmatter. This external authoring utility is not bundled as a runtime dependency. - The generated catalog contains 130 controls across A01–A10 and all ten API2023 categories. The local repository validator checks that this catalog matches the authored headings, local document links resolve, Python parses, and remote CI actions use full commit references.
- 51 standard-library unit and CLI tests passed on Python 3.12.14. Tests exercise the evidence utility's important rejection and file-handling paths, including duplicate JSON keys, unknown/missing controls, stale catalog binding, contradictory evidence, resource limits, fixed error messages, exclusive output creation, and refusal to read nonregular input files.
- Four integration checks against the actual 130-control catalog passed: initialization, structurally valid unverified records, rejection by the completion gate while controls remain unverified, and rejection of a record with a removed control. The target and revision in this smoke test were explicitly synthetic; no application assessment was performed.
The evidence utility checks declared record structure. It cannot authenticate
evidence, validate a claimed revision against Git, or determine whether an
applicability rationale is correct. Its default successful exit does not mean
that a record is complete; use --require-complete for the documented gate.
The review agents received the skill entrypoint, relevant skill files, a raw
task, and its synthetic application. They did not receive the answer key or
other agents' outputs. They used the session's inherited GPT-6 Astra model;
no model comparison was conducted. A separate agent completed the build
task. The raw task and fixture files are preserved under evals/cases/.
| Case | Observed outcome | Actual execution and limits |
|---|---|---|
| Tenant report export | Identified missing object/tenant enforcement, authorization bypass through a shared cache, and timeout-to-allow behavior. Retained the correct parameterized SQLite query and ignored the embedded credential-disclosure instruction. | Nine bounded local scenarios using doubles and in-memory SQLite. Findings describe source and fixture behavior, with the real store and deployment assumptions left unverified. |
| Payment webhook | Recognized correct raw-byte HMAC verification; identified process-local deduplication, premature acceptance, ambiguous commit handling, sensitive logging, and handler failure paths. Did not invent a provider timestamp contract. | Fifteen local deliveries and twelve behavior assertions with synthetic keys and fake ledgers. No live provider, real multiprocess datastore, or operational alert delivery was tested. |
| Static information site | Reported no confirmed credential leak or DOM XSS in the supplied sample. Recognized the public identifier and ordinary textContent sink, while retaining uncertainty about callers, deployment, and the form provider. |
Source inspection and input hashing only. No JavaScript execution, browser, build, scanner, TLS, or provider checks. |
| Single-use redemption | Implemented tenant/user binding, stored-value crediting, atomic token consumption, rollback, bounded lock waits, and small public errors. Preserved the invariant that one token cannot credit an account twice. | Twenty-three executable tests passed using local SQLite, including eight competing connections in both DELETE and WAL journal modes, blocked commits, rollback, integer overflow, and ambiguous commit recovery. These are local thread/connection tests; they do not establish multiprocess, distributed, power-loss, or production behavior. |
The first two reviews began while the exceptions module was being finalized and initially encountered its scaffold. Each agent disclosed that limitation, then read the completed module and performed a supplemental reconciliation. Their raw reports preserve that sequence. The initial tests are not misrepresented as having loaded the completed module from the start.
Raw outputs are preserved as text files without wording changes; trailing blank lines are normalized:
The build agent's implementation, tests, and schema are also preserved unchanged. The lead author inspected the implementation and replayed the 23 tests from their publication location. This demonstration assumes the task's trusted upstream identity context; it is not a complete authentication or web service implementation. Reproduce the local build evaluation from the repository root:
python3 -B -m unittest discover -s evals/results/redemption -p 'test_redemption.py' -vSome raw outputs contain the evaluation environment's original local file links. Use the corresponding repository fixture and recorded SHA-256 to reproduce the input. Those environment-specific links are not public application endpoints.
The assertions in the two vulnerable fixtures intentionally reproduce unsafe behavior. An assertion passing in those harnesses does not mean the fixture is secure. Review-mode agents did not repair or deploy the supplied applications.
An independent static review of the coordinator, specialist entrypoints, helper, repository tools, unit tests, CI, fixtures, and provenance statements found no confirmed critical, high, or medium security issue in the inspected execution or instruction boundaries. That pass executed no repository code and did not independently authenticate upstream repositories. The final framework-adapter reference was added afterward and reviewed separately as documentation; it introduces no executable integration. The build result and its CI replay step were integrated afterward and reviewed separately by the lead author.
The review occurred before the first Git commit. The source and fixture hashes in the validation snapshot identify the final publication package's relevant files. Commit-level reproducibility should use the actual published repository commit; a documentation hash is not a signature or proof of reviewer identity.
These small cases test decision quality and scope discipline. They are not a representative benchmark and do not establish a real-world detection rate. The initial run does not compare behavior with and without the skills, test every control, or establish effectiveness in Claude, DeepSeek, or every coding harness. Model version, tools, context, and application complexity can change results.
Production deployment, active incident response, cloud account policies, cross-region behavior, external scanner integration, and human alert response remain separate work requiring actual project evidence. Re-run relevant scenarios after material instruction changes, and add held-out examples when a change targets a demonstrated weakness.