Round-2 adversarial-mutation-test coverage: query, magic CLI, validators, CopyArtifacts

[thedavidmeister]

Round-2 AMT campaign over the units left uncovered by round 1: process_meta_query/get_authoring_meta (crates/cli/src/meta/query/mod.rs), the magic ls CLI surface, the common/v1 validator regexes, and script/CopyArtifacts.sol — probed with nix run github:rainlanguage/adversarial-mutation-test#mutation-probe over two suites: rust nix develop rainix#rust-shell -c cargo test (baseline 137 passed) and sol git checkout -- crates/bindings/abi && git clean -fdq crates/bindings/abi && forge build && forge script ./script/CopyArtifacts.sol --ffi && forge test (baseline 16 passed).

Sequence per the AMT protocol: all 49 mutants (40 rust + 9 sol) probed against the PRE-EXISTING suites first — rust 7 KILLED / 33 SURVIVED / 0 no-run, sol 4 KILLED / 4 SURVIVED / 1 NO-RUN (compile-enforced) — then every survivor either killed by a new or in-place-strengthened discriminating test and re-probed (--only pass 2: rust 21/21 KILLED, sol S1–S3 3/3 KILLED, both baselines green at 146/16), or recorded below as equivalent/dead/build-enforced with evidence, never forced.

Behaviour matrix

Verdict legend: PRE = killed by pre-existing test (validates existing coverage), NEW = killed by a test added here, STR = killed by a pre-existing test strengthened in place.

#	Behaviour	Mutation(s)	Verdict	Killing test(s)
1	process_meta_query: absent data member → NoRecordFound	R01	NEW	meta::query::tests::test_process_meta_query_missing_data_is_no_record_found
2	process_meta_query: null meta (true absence) → NoRecordFound	R02	NEW	test_process_meta_query_missing_meta_is_no_record_found
3	process_meta_query: undecodable rawBytes hex → NoRecordFound, not CorruptMeta, not swallowed to empty	R03, R04	NEW	test_process_meta_query_bad_hex_is_no_record_found
4	process_meta_query: POST with the JSON query body to the given url	R05, R06	NEW	test_process_meta_query_success_exact_bytes, test_process_meta_query_non_json_is_reqwest_error
5	process_meta_query: transport error → ReqwestError	R07	NEW	test_process_meta_query_send_error_is_reqwest_error
6	process_meta_query: body decode error → ReqwestError	R08	NEW	test_process_meta_query_non_json_is_reqwest_error
7	process_meta_query: returned bytes are exactly the decoded rawBytes	R09	NEW	test_process_meta_query_success_exact_bytes
8	get_authoring_meta: unpack failure skips the item and keeps scanning	A01	NEW	test_get_authoring_meta_skips_unpack_failure
9	get_authoring_meta: scan continues beyond the first item	A02	NEW	test_get_authoring_meta_scans_beyond_first_item
10	magic ls prints every KnownMagic, in order, correct columns, 0x-prefixed exact values	G01–G05	NEW	tests/magic_cli.rs::test_magic_ls_exact_output (e2e via CARGO_BIN_EXE_rain-metadata)
11	RainSymbol: lowercase-only tail; end anchor	C11 (tail widened to uppercase)	STR	test_rain_symbol_validate (added "aA" invalid)
		C12 (end anchor dropped)	PRE	test_rain_symbol_validate
12	SolidityIdentifier: end anchor; no hyphen in tail	C13, C14	PRE	test_solidity_identifier_validate
13	RainString: excludes DEL; whitespace class not narrowed to space	C15 (DEL)	STR	test_rain_string_validate (added "\u{7f}" invalid)
		C22 (whitespace narrowed)	PRE	test_rain_string_validate
14	RainTitle: first char printable non-space, excludes DEL; single-char alternative	C16 (DEL first char)	STR	test_rain_title_validate (added DEL-leading invalid)
		C17 (single-char alternative dropped)	PRE	test_rain_title_validate
15	HASH_PATTERN: exactly 64 chars; hex class; lowercase 0x prefix	C18, C19 (63/65 length)	PRE	test_hash_pattern
		C20 (class widened to g), C21 (0X accepted)	STR	test_hash_pattern (added 64×g and "0X"+64-hex invalids)
16	CopyArtifacts.run(): every contract copied; loop body does the copy; content byte-exact	S1 (first contract only), S2 (body no-op)	STR	testArtifactsCommitted (corrupt both committed files → run() → byte-exact restore against pre-corruption on-disk state)
17	CopyArtifacts.run(): missing dst is recreated, exists guard not inverted	S3	STR	testArtifactsCommitted (delete one committed file → run() → recreated byte-exact)
18	CopyArtifacts: writeFile content/target-path/stable-subset/presence	S5, S6, S7, S8	PRE	testArtifactsCommitted
19	IMetaBoardV1_2.emitMeta interface name	S9	NO-RUN	compile-enforced: rename fails forge build at LibDescribedByMeta.sol:37

Equivalent / dead / build-enforced dispositions (no test, never forced)

• L01 equivalent — serde_cbor 0.11.2 SliceRead raises every EOF at offset == slice.len(), and cbor_decode compares against exactly that slice length, so == vs <= is unobservable. Empirical truncation sweep (>10 EOF cases, all offset == len).
• L02 equivalent — track/metas are pushed in the same iteration; the length check can never fire.
• L03 equivalent — dotrain create() uri-gate half removal unobservable.
• L04/L05 dead branches — panic-armed mutants survive, proving the branches unreached.
• P03/P06/P07 equivalent — ValidationErrors::merge_all drops all child results (children are keyed by their own field names / Field kind and never pass the err.remove(field) + Struct filter; Operand carries zero field rules). Behaviour already on file as BitIntegerRange/OperandArgRange/Output validation drops all sub-validation errors (ValidationErrors::merge_all misuse) #173/OpMeta validation silently discards nested BitInteger/Output errors (merge_all misused) #214 — no new issue.
• V02/V03 equivalent — zero #[validate] attrs in solidity_abi/v2.rs make the validate gates vacuous; already on file as SolidityAbiMeta validation is vacuous: Validate derives carry no rules, so the TryFrom validate gate and per-index annotation can never reject #153.
• MB03 equivalent — cynic rejects data-absent + errors-absent responses ("Either data or errors must be present"), so the Empty arm is unreachable; already on file as CynicClientError::Empty is unreachable from CynicClient::query: missing/null data surfaces as a Request decode error #178. Empirical: {"data":null}, {}, {"data":null,"errors":null} all surface as Request decode errors.
• B01 pass-1 verdict unsound (stale artifact); actually build-enforced — with a clean OUT_DIR, mutating the metaboard build.rs schema registration name fails compilation at lib.rs:5 #[cynic::schema("metaboard")] ("Could not find a schema with this name"), i.e. NO-RUN class like S9. The pass-1 SURVIVED verdict came from the probe's incremental build reusing stale OUT_DIR cynic-schemas/metaboard.{graphql,rs} residue from the unmutated baseline; demonstrated both ways (clean OUT_DIR → exit 101; stale artifacts restored → mutated crate builds green). No runtime test can pin a build-registry name.
• S4 equivalent — vm.writeFile truncates before writing, so the remove-before-write is redundant for an existing dst.
• U1-M03 (no syntactic mutant) — #[serde] Option fields decode absent keys to None identically with or without an explicit default; analytic serde equivalence recorded.

Adjudication A-1: disproof

The round-1 adjudication claim that generate source emits empty meta was DISPROVEN: fresh cargo build at bba50a7, printf '#main\n_: int-add(1 2);' | rain-metadata generate source yields selector 0x37480e2a, meta length 0x46, calldata meta bytes identical to the independently CBOR-encoded document, ff0a89c674ee7874 document prefix present. The "empty meta" observation was a stale-binary artifact.

Deviation: forge runs same-contract tests in parallel

The plan assumed test functions within one contract are serialized; forge 1.7.2 disproved this (new functions appended to CopyArtifactsTest raced testArtifactsCommitted on the shared on-disk committed files, flaking the suite). Since every step shares the same files, testArtifactsCommitted was instead strengthened IN PLACE into one serialized sequence: original assertions first (unweakened) → corrupt both committed files → run() → assert byte-exact restore (kills S1/S2) → delete one committed file → run() → assert recreated byte-exact (kills S3).

Issues filed (adversarial findings, labels audit+adversarial)

#231 (solc artifact prints null, exit 0, for a missing component — adjudication A-2), #232 (Store::merge deployer_hash_map overwrite asymmetry — A-3), #233 (str_to_bytes32/bytes32_to_str NUL round-trip asymmetry — A-4), #234 (process_meta_query collapses corrupt payloads / bad hex / GraphQL errors into NoRecordFound; meta-side sibling of #213), #235 (RainSymbol regex accepts trailing/double hyphens vs kebab-case contract), #236 (RainTitle validator message contradicts its regex and doc). Duplicates deliberately not re-filed: merge_all child-dropping (#173/#214), cynic Empty unreachability (#178), vacuous SolidityAbiMeta validation (#153).

QA

• Discriminating tests: meta::query::tests::* (6 httpmock process_meta_query tests + 2 multi-item get_authoring_meta tests), tests/magic_cli.rs::test_magic_ls_exact_output (e2e against the built binary), strengthened in place test_rain_symbol_validate / test_rain_string_validate / test_rain_title_validate / test_hash_pattern (new discriminating invalids only — no existing case weakened), sol testArtifactsCommitted (corrupt-both-restore + delete-one-recreate appended after the original unweakened assertions). Every pass-1 survivor selected for pass 2 flips to KILLED with the killer named by the probe fail-pattern (probe-rust-pass2: 21/21; probe-sol-pass2: 3/3; baselines green 146 / 16).
• Mutations applied: 49 exact-string mutants — 40 rust (R01–R09, A01–A02, G01–G05, C11–C22, L01–L05, P03/P06/P07, V02/V03, MB03, B01) + 9 sol (S1–S9); per-mutant line → mutation → verdict in the matrix above (e.g. .ok_or(Error::NoRecordFound) → CorruptMeta → test_process_meta_query_missing_data_is_no_record_found; CopyArtifacts exists-guard inverted → delete-one-recreate sequence).
• Oracle: doc comments of the mutated units (process_meta_query's "resolving if a record was found" contract, common/v1 regex doc contracts, CopyArtifacts committed-artifact invariant), the KnownMagic table for the exact magic ls pin, httpmock request-shape assertions (method, path, body), and independently constructed expected values (expected meta bytes hex-encoded in the test, not read back from the implementation; sol byte-exact oracle = pre-corruption on-disk state captured before mutating the files).
• Category check: round-2 worklist covers 49 mutants across 4 unit groups; all 49 dispositioned — rust 28 KILLED (7 PRE + 21 NEW/STR) + 12 equivalent/dead/build-enforced, sol 7 KILLED (4 PRE + 3 STR) + 1 equivalent (S4) + 1 compile-enforced (S9); adversarial candidates filed as CLI solc artifact prints null with exit 0 when the requested component is missing from the artifact #231–RainTitle validator message claims 'alphanumeric ASCII letters and spaces' but the regex accepts all printable ASCII and rejects edge spaces #236.

Probe tool: nix run github:rainlanguage/adversarial-mutation-test#mutation-probe with the two suite commands above, proof (\d+) passed; (\d+) failed, fail-pattern naming killer tests. Mutant files kept out of the repo.

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

Next included review available in 32 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 31470f3b-f40f-4b32-8862-8a7c0de138f3

📥 Commits

Reviewing files that changed from the base of the PR and between 9c65a23 and 7d620d8.

📒 Files selected for processing (4)

• crates/cli/src/meta/query/mod.rs
• crates/cli/src/meta/types/common/v1.rs
• crates/cli/tests/magic_cli.rs
• test/script/CopyArtifacts.t.sol

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch 2026-08-22-amt-round2

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

@coderabbitai assess this PR size classification for the totality of the PR with the following criterias and report it in your comment:

S/M/L PR Classification Guidelines:

This guide helps classify merged pull requests by effort and complexity rather than just line count. The goal is to assess the difficulty and scope of changes after they have been completed.

Small (S)

Characteristics:

• Simple bug fixes, typos, or minor refactoring
• Single-purpose changes affecting 1-2 files
• Documentation updates
• Configuration tweaks
• Changes that require minimal context to review

Review Effort: Would have taken 5-10 minutes

Examples:

• Fix typo in variable name
• Update README with new instructions
• Adjust configuration values
• Simple one-line bug fixes
• Import statement cleanup

Medium (M)

Characteristics:

• Feature additions or enhancements
• Refactoring that touches multiple files but maintains existing behavior
• Breaking changes with backward compatibility
• Changes requiring some domain knowledge to review

Review Effort: Would have taken 15-30 minutes

Examples:

• Add new feature or component
• Refactor common utility functions
• Update dependencies with minor breaking changes
• Add new component with tests
• Performance optimizations
• More complex bug fixes

Large (L)

Characteristics:

• Major feature implementations
• Breaking changes or API redesigns
• Complex refactoring across multiple modules
• New architectural patterns or significant design changes
• Changes requiring deep context and multiple review rounds

Review Effort: Would have taken 45+ minutes

Examples:

• Complete new feature with frontend/backend changes
• Protocol upgrades or breaking changes
• Major architectural refactoring
• Framework or technology upgrades

Additional Factors to Consider

When deciding between sizes, also consider:

• Test coverage impact: More comprehensive test changes lean toward larger classification
• Risk level: Changes to critical systems bump up a size category
• Team familiarity: Novel patterns or technologies increase complexity

Notes:

• the assessment must be for the totality of the PR, that means comparing the base branch to the last commit of the PR
• the assessment output must be exactly one of: S, M or L (single-line comment) in format of: SIZE={S/M/L}
• do not include any additional text, only the size classification
• your assessment comment must not include tips or additional sections
• do NOT tag me or anyone else on your comment