VM: adopt sha256tree at 0x3f (divergence D9) - #16
Merged
Merged
Conversation
Decision by Evan, 2026-07-29, ratified in VM.md section 8 entry 7. The operator returns the 32-byte tree hash of its single argument: an atom hashes as SHA-256 of a 0x01 tag then the atom's bytes, a pair as SHA-256 of a 0x02 tag then the two child hashes. Any node is a legal argument, arity is the only check, and it precedes every charge. Semantics, opcode, and cost constants (270 base, 460 per pair, 2 per byte counting the leaf tag byte, flat 320 malloc) are the pinned consensus wheel's own sha256tree, carried behind its release flag and scheduled for consensus activation upstream (CHIP-0049, in review). Verified against the released binary by probe 2026-07-29: tree hashes, totals at the exact budget boundary, arity errors, and flags-0 unknown acceptance all match the constants and ordering specified here. Upstream source (clvm_rs treehash.rs) informed the charge interleaving, and the binary confirmed it. The per-node charge rule is normative for soundness, not only for pricing: (c 1 1) doubles the environment reachable from its result for one cons, and k nested applies of that shape reach 2^k visited nodes for build cost linear in k, sharing the witness never spells. Charging each visited node as the walk reaches it bounds the work a budget can buy, recorded in COSTS.md section 8 alongside the substr copy-on-slice note. New divergence row D9: deployed consensus at flags 0 treats 0x3f as an unknown operator, BitLisp dispatches sha256tree. Oracle strategy recorded in VM.md section 7: the flag-enabled wheel, the wheel's tree_hash puzzle-hash utility, and an in-language tree-hash program over the intersection, three released-binary legs.
The walk mirrors the machine's explicit-stack rule so argument depth is never limited by the Python recursion limit, charges each visited node as it reaches it with the base cost riding on the first node's charge, and never deduplicates sharing: a node reachable twice is charged and hashed twice. Constants come from the consensus oracle's flag-gated operator, verified by probe against the released wheel (results, boundary totals, arity errors) before this commit. Upstream source (clvm_rs treehash.rs) informed the traversal and charge interleaving, and the released binary confirmed every constant and ordering, per the reference-material guardrails. Spec: VM.md section 4 (operator table, crypto family), section 6 row D9, section 8 entry 7, COSTS.md sections 1 and 8.
Twenty-one cases in vectors/vm/sha256tree.json: the two COSTS.md worked examples, leaf-tag and actual-bytes-as-given rules (redundant and minimal integer spellings hash differently), pair-tag child order, an evaluated pair argument from the environment, hash-of-hash composition, the shared-subtree double charge under (c 1 1), the inclusive budget boundary at the exact total and one below, all three arity shapes including arity beating the budget at max_cost 1, the zero-budget dispatch burst (D7 interaction), a raising argument beating the operator, and the exponential shared-environment DAG (forty nested applies doubling the environment per level) reporting cost_exceeded under a 500000 budget, the fail-fast rule of COSTS.md section 8. Every success case and every oracle-expressible error case was cross-checked against the flag-enabled consensus wheel (result and cost) and the tree_hash utility at generation time. Spec: VM.md sections 4 and 6 (D9), COSTS.md sections 1 and 8.
tools/diff_sha256tree.py pins the operator on three released-binary legs: (cost, result) against the pinned wheel's flag-enabled dispatch including budgets within a few units of the measured cost, results against the wheel's tree_hash puzzle-hash utility, and results against an in-language tree-hash program built from intersection operators and run through bitlisp and both pinned oracles at flags 0. The generator draws atom sizes across the serialization forms, redundant integer spellings, and (c X X) shared-structure chains where the walk charges per visit. Every run also probes arity defects and an exponential shared-environment DAG (25 to 50 nested doublings under a small budget), where both sides must report cost_exceeded in bounded time: an implementation that hashed before charging would hang the harness instead. CI runs 400 cases per push with the run id as seed, printed for local reproduction, alongside the existing diff legs. The unit suite gains a fixed-seed 150-case smoke slice so pytest alone stays a complete local gate. Spec: VM.md section 7 (oracle strategy for D9).
Findings from the five-reviewer pass on this PR, each verified before fixing. The section 5 error-taxonomy row scoped arg_not_atom to the whole crypto family, contradicting the section 4 statement that sha256tree has no arg_not_atom path: the row now excepts it, the reverse of the edit that widened the row when secp_verify landed. The D2 record's family enumeration (sha256 plus secp_verify, nothing else) is now explicitly amended by entry 7 rather than left silently overturned, per the section 8 preamble's amendment rule. The rewritten boolean-family sentence claimed only booleans and sha256tree accept pair arguments outside the tree ops, false given apply's environment argument and raise's arguments: the exclusivity claim is dropped. Section 7 no longer calls the three legs independent, since two are views of the one pinned wheel and only one checks cost, and it now describes the fourth leg added in this PR: every generated program runs through both oracles at flags 0, pinning the D9 oracle column continuously instead of by a one-time probe. The D9 row's vector cell now says which column the vector file pins.
The docstring's two-way split (consensus-oracle order for everyone, secp_verify its own normative choice) missed the third category this PR introduced: op_sha256tree's order matches the same released wheel behind its release flag, while flags-0 consensus treats the opcode as unknown, a recorded divergence. Spec: VM.md sections 6 (D9) and 7.
The envelope's spec field now spells divergence D9 the way sibling files spell their rows, and the corpus README's divergence-case enumeration now includes vm/sha256tree.json with the flag-enabled cross-check that produced it. Spec: VM.md section 6 (D9).
The divergence row's oracle column (deployed consensus accepts 0x3f as an unknown operator returning nil) was verified by a one-time probe but re-verified by nothing. The harness now runs every generated program through both oracles at flags 0 and requires unknown-op acceptance with a nil result, guarded on success so an erroring argument is never misread as rejection. Costs are not compared on that leg, the unknown-op charge has nothing to do with the operator's constants. Also from review: the docstring no longer claims the budget-boundary leg observes charge interleaving (only the order-independent total decides this operator's outcome), it now states which legs share the one wheel and that only the first checks cost, and the exponential stat counts like its siblings. Spec: VM.md sections 6 (D9) and 7.
The Phase 1 checklist line defined the operator set as CLVM core minus BLS plus secp_verify, stale once sha256tree joined. The line now records the post-close amendment with its date and divergence row, following the plan's precedent of correcting frozen facts that drifted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
sha256treejoins the v0 operator table at opcode0x3f(decision by Evan, 2026-07-29). It returns the 32-byte tree hash of its single argument: atoms hash as SHA-256 of a0x01tag then the atom's bytes, pairs as SHA-256 of a0x02tag then the two child hashes. Semantics, opcode, and cost constants are the pinned consensus wheel's own flag-gated sha256tree, which upstream has scheduled for consensus activation in Chia's 3.0 hard fork (CHIP-0049, in review, cited by name deliberately without a link).Spec sections that authorize it
Read the commits in this order
(a (q . inner) (c 1 1))level, reaching 2^k visited nodes in linear program bytes, so an implementation must charge as it walks or a small budget buys unbounded hashing.How to verify independently
.venv/bin/pytest python/tests(77 tests, includes the new fixed-seed smoke slice).venv/bin/python tools/run_vectors.py(481 cases).venv/bin/python tools/run_upstream.py(832 vendored cases, unaffected: the upstream unknown-op corpus uses multi-byte opcodes, not0x3f).venv/bin/python tools/diff_clvm.py --count 10000 --seed <fresh>(intersection unaffected).venv/bin/python tools/diff_sha256tree.py --count 400 --seed <fresh>(the new operator's three oracle legs, boundary budgets, arity, exponential DAG)chia_rs.run_chia_program(prog, env, cost, chia_rs.ENABLE_SHA256_TREE)reproduces every success vector byte for byte, andchia_rs.tree_hashreproduces every digest.All of the above ran clean on this branch before pushing (fresh seeds 8151623, plus 424242 and 20260729 during development).
Notes for review
sha256tree, review finding), and the D2 record's family enumeration (now explicitly amended by entry 7, review finding).docs/opcode-comparison.md(PR Docs: CLVM/bllsh/BitLisp opcode comparison #13, open) is updated separately on that PR's branch.Post-review fixes
A five-reviewer pass (adversarial, including mutation testing of the implementation and hand-verification of every vector digest and cost) found no implementation, vector, or constant defects. It found spec and doc consistency issues, fixed in the five commits after the original four: the arg_not_atom taxonomy row, the D2 family-enumeration amendment, an over-broad pair-argument claim, the overclaimed leg independence, an unpinned oracle-column claim (now pinned by a flags-0 harness leg, 400 of 400 per run), the operators module docstring, the corpus README's divergence enumeration, and the plan's Phase 1 operator-set line.