From c08e98ea7de5644f23af7d0a8a5578da9e7a7b3e Mon Sep 17 00:00:00 2001 From: Evan Date: Wed, 29 Jul 2026 16:17:07 -0700 Subject: [PATCH 1/9] Spec: adopt sha256tree at 0x3f (divergence D9) Decision by Evan, 2026-07-29, ratified in VM.md section 8 entry 7. The operator returns the 32-byte tree hash of its single argument: an atom hashes as SHA-256 of a 0x01 tag then the atom's bytes, a pair as SHA-256 of a 0x02 tag then the two child hashes. Any node is a legal argument, arity is the only check, and it precedes every charge. Semantics, opcode, and cost constants (270 base, 460 per pair, 2 per byte counting the leaf tag byte, flat 320 malloc) are the pinned consensus wheel's own sha256tree, carried behind its release flag and scheduled for consensus activation upstream (CHIP-0049, in review). Verified against the released binary by probe 2026-07-29: tree hashes, totals at the exact budget boundary, arity errors, and flags-0 unknown acceptance all match the constants and ordering specified here. Upstream source (clvm_rs treehash.rs) informed the charge interleaving, and the binary confirmed it. The per-node charge rule is normative for soundness, not only for pricing: (c 1 1) doubles the environment reachable from its result for one cons, and k nested applies of that shape reach 2^k visited nodes for build cost linear in k, sharing the witness never spells. Charging each visited node as the walk reaches it bounds the work a budget can buy, recorded in COSTS.md section 8 alongside the substr copy-on-slice note. New divergence row D9: deployed consensus at flags 0 treats 0x3f as an unknown operator, BitLisp dispatches sha256tree. Oracle strategy recorded in VM.md section 7: the flag-enabled wheel, the wheel's tree_hash puzzle-hash utility, and an in-language tree-hash program over the intersection, three released-binary legs. --- spec/COSTS.md | 42 +++++++++++++++++++ spec/VM.md | 111 +++++++++++++++++++++++++++++++++++++++++++++----- 2 files changed, 143 insertions(+), 10 deletions(-) diff --git a/spec/COSTS.md b/spec/COSTS.md index efb64af..0af51db 100644 --- a/spec/COSTS.md +++ b/spec/COSTS.md @@ -122,6 +122,18 @@ with Phase 3 measurements. neither the empty-signature nil nor a verification outcome is ever reported when the budget cannot cover the charge, pinned by boundary vectors. + - `sha256tree`: the arity check precedes every charge. The base + cost accrues without a budget check and rides on the first + visited node's charge (every tree has at least one node). Then + one checked charge per visited node: a pair charges the pair + cost when the walk reaches it, an atom charges the per-byte cost + on its length plus one, the leaf tag byte. The traversal order + among nodes is not consensus-visible: the only error a walk + charge can raise is `cost_exceeded` and the total is + order-independent, so an implementation may walk in any order + provided it charges each node as the walk reaches it (the + soundness rule in section 8). The result's malloc is one final + checked charge after the walk. - Evaluation cost does not include deserialization. A per-byte cost on the serialized program belongs to the weight mapping (section 9). @@ -246,6 +258,7 @@ Worked example, pinned by vectors: `(any (q . 1) (q . 2))` costs | --- | --- | | `sha256` | `87 + 134 * n_args + 2 * total_arg_bytes + malloc(result)` | | `secp_verify` | `1300000`, flat, no malloc (PROVISIONAL) | +| `sha256tree` | `270 + 460 * n_pairs + 2 * (n_atoms + total_atom_bytes) + malloc(result)`, counting every visited node: a node the walk reaches twice counts twice | The sha256 result atom is always exactly 32 bytes, so its malloc is a flat 320 charged after the argument loop. Per-byte terms count @@ -262,6 +275,29 @@ recorded in VM.md section 8. The empty-signature branch charges the same flat cost in v0, with a cheaper price explicitly left as a Phase 3 question there. +`sha256tree`'s per-byte term prices each visited atom's actual bytes +plus one, the leaf tag byte, at `sha256`'s 2 per byte. The pair +constant 460 covers a pair node's own SHA-256 invocation, whose input +is always the tag byte and two 32-byte child hashes. The result atom +is always exactly 32 bytes, so its malloc is a flat 320 charged after +the walk. The constants are the consensus oracle's own, carried +behind its release flag (divergence D9), and every visited node +counts: the walk follows structure without deduplicating sharing, so +a subtree reachable twice is charged twice. + +`sha256tree`'s per-node charging is load-bearing for more than the +error class at the budget boundary. Evaluation builds shared +structure cheaply: `(c 1 1)` doubles the environment reachable from +its result for one 50-cost cons, and k nested applies of that shape, +a few hundred cost units each, reach 2^k visited nodes for build +cost linear in k. The sharing is built by evaluation, never spelled +in the witness, so canonical serialization is no defense. An +implementation must charge each node as the walk reaches it and stop +at `cost_exceeded`, doing exactly the budget's worth of hashing. +Hashing the whole tree before charging does unbounded work under a +small budget, a validation denial of service, the same hazard class +as the copy-on-slice note in section 5. + Worked examples, pinned by vectors: `(sha256 (q . "ab") (q . "cd"))` costs `20 + 20 + 1 + 87 + 134 * 2 + 2 * 4 + 320 = 724`, and a `secp_verify` application on three quoted arguments costs @@ -269,6 +305,12 @@ costs `20 + 20 + 1 + 87 + 134 * 2 + 2 * 4 + 320 = 724`, and a a valid signature and for an empty one. The failing path charges identically before it raises `secp_verify_failed`. +`sha256tree` worked examples, pinned by vectors: `(sha256tree (q))` +hashes nil at `20 + 1 + 270 + 2 * 1 + 320 = 613`, and +`(sha256tree (q "ab" "cd"))` walks two pairs and three atoms holding +four bytes for +`20 + 1 + 270 + 460 * 2 + 2 * (3 + 4) + 320 = 1545`. + ## 9. Weight mapping TODO (Phase 3): mapping from VM cost units to Bitcoin transaction diff --git a/spec/VM.md b/spec/VM.md index e23264b..f74fbc8 100644 --- a/spec/VM.md +++ b/spec/VM.md @@ -182,8 +182,8 @@ failing closed. The operator table is complete for v0: the core specials, the tree ops family, the arithmetic family, the bytes and strings family, the -bitwise family, the boolean family, and the crypto family (`sha256` -and `secp_verify`, divergence D2). +bitwise family, the boolean family, and the crypto family (`sha256`, +`secp_verify`, and `sha256tree`, divergences D2 and D9). | Opcode | Name | Arity | Semantics | | --- | --- | --- | --- | @@ -217,6 +217,7 @@ and `secp_verify`, divergence D2). | `0x20` | `not` | 1 | TRUE if the argument is nil, nil otherwise. | | `0x21` | `any` | 0 or more | TRUE if at least one argument is not nil. No arguments gives nil. | | `0x22` | `all` | 0 or more | TRUE if every argument is not nil. No arguments gives TRUE. | +| `0x3f` | `sha256tree` | 1 | The tree hash of its argument, atom or pair: an atom hashes as SHA-256 of `0x01` then the atom's bytes, a pair as SHA-256 of `0x02` then the first child's hash then the rest child's hash. Divergence D9. | The tree ops select, build, and compare nodes without interpreting them as integers: @@ -362,10 +363,10 @@ byte, so the section 2 threshold for building an atom the wire format cannot encode applies unchanged. The boolean ops test nil-ness and return the shared TRUE and nil -constants. They are the one non-tree family that accepts pair -arguments: any node is legal in any position, and nil is the only -false value (the one-byte atom `0x00` and every pair are true), the -same rule `i` applies to its selector. +constants. Outside the tree ops, only they and `sha256tree` accept +pair arguments: any node is legal in any position, and nil is the +only false value (the one-byte atom `0x00` and every pair are true), +the same rule `i` applies to its selector. - `not` takes exactly one argument: TRUE if it is nil, nil otherwise. - `any` returns TRUE if at least one argument is not nil, and nil @@ -376,10 +377,11 @@ same rule `i` applies to its selector. per-byte term: a large atom argument costs the same as a one-byte one. -The crypto family holds `sha256` and `secp_verify`. Like the bytes -and strings ops both treat atoms as raw byte strings: every argument -must be an atom, and a pair raises `arg_not_atom`. `sha256` has no -operand size limit. +The crypto family holds `sha256`, `secp_verify`, and `sha256tree`. +The first two treat atoms as raw byte strings, like the bytes and +strings ops: every argument must be an atom, and a pair raises +`arg_not_atom`. `sha256tree` hashes structure, so its single argument +may be any node. Neither hashing operator has an operand size limit. - `sha256` returns the SHA-256 digest of the concatenation of its arguments in argument order, a freshly built 32-byte atom charging @@ -393,6 +395,33 @@ operand size limit. - The result is always exactly 32 bytes, so `sha256` can never build an atom the wire format cannot encode. +`(sha256tree X)` returns the 32-byte **tree hash** of the node X +(divergence D9), a content address for a whole tree: + +- An atom's tree hash is the SHA-256 digest of the byte `0x01` + followed by the atom's bytes, so nil's tree hash digests the single + byte `0x01`. A pair's tree hash is the SHA-256 digest of the byte + `0x02` followed by the first child's tree hash then the rest + child's tree hash, 65 bytes in all. +- The two tag bytes separate leaves from pairs, and the fixed 32-byte + child-hash width makes the pair form unambiguous, so distinct trees + have distinct hashes up to SHA-256 collision resistance. +- Exactly one argument, which may be any node. There is no + `arg_not_atom` path, and no operand size or depth limit, the cost + budget is the only bound. The arity check precedes every charge, so + `wrong_arg_count` wins over `cost_exceeded` when both would fire. +- Atoms hash their actual bytes as given, redundant encoding bytes + included, like `sha256`: the tree hashes of `0x0002` and `0x02` + differ. +- Cost accrues per visited node during the walk, checked at every + charge, and the walk follows structure without deduplicating + sharing: a pair built as `(c X X)` reaches X twice, and both visits + charge. Charging as the walk proceeds is load-bearing for + soundness, not only for cost accounting. COSTS.md sections 1 and 8 + state the rule and the work-amplification hazard it closes. +- The result is a freshly built 32-byte atom charging malloc, so + `sha256tree` can never build an atom the wire format cannot encode. + `secp_verify` is the one operator with no CLVM counterpart in any form (divergence D2). `(secp_verify pubkey msg sig)` verifies a BIP340 Schnorr signature over secp256k1. @@ -478,6 +507,7 @@ pin it. No divergence exists outside this table. "Both oracles" means | D6 | `/` with negative operands | Consensus (`chia-rs`): floor division. The `clvm` package injects a policy error ("deprecated") that is not consensus | Floor division, matching consensus | Intersection parity targets the consensus oracle. The Python package's rejection is library policy, the diff harness treats it as an expected divergence. Ratified, see section 8. | `vm/arith.json`, upstream corpus D6 bucket | | D7 | Zero cost budget | Both oracles treat `max_cost = 0` as unlimited | A zero budget is a real budget, no program succeeds under it (section 3.3) | A zero sentinel meaning unlimited is a library convenience, not consensus behavior. In the Bitcoin context the budget derives from transaction weight and is never legitimately zero, and an accidental zero must fail closed rather than open. Ratified, see section 8. | `vm/dispatch.json` | | D8 | Resource limits outside the cost model | The consensus oracle enforces caps the cost model never sees: at most 62,500,000 atoms and as many pairs per run (deserialization spends one count per atom and two per cons, probed at the boundary: a 62.7 million node budget fails "too many pairs" before evaluation, 62.4 million deserializes), a 4 GiB atom-byte heap, 20,000,000-entry value and environment stacks, and a two-argument `substr` whose default end index passes through a signed 32-bit cast, rejecting data atoms of 2^31 bytes or more | No equivalent limits: BitLisp is bounded by the cost budget, and its deserializer by the input's size alone | Every cap sits far outside the reachable regime. The cheapest evaluation-time trigger costs about 5.6e10 against the harness budget of 1.1e10, and the deserialization trigger needs roughly 42 MB of input against Bitcoin's 4 MB witness ceiling. PROVISIONAL, see section 8: the Phase 3 budget and input-size bounds must be recorded against these thresholds, or the caps mirrored fail-closed. | none, unreachable (section 8) | +| D9 | `sha256tree` | Deployed consensus (flags 0) treats opcode `0x3f` as an unknown operator under the D3 acceptance rule: arguments evaluate, cost derives from the opcode byte, result nil. The pinned oracle wheel carries a `sha256tree` operator at the same opcode behind its release flag, scheduled for consensus activation in Chia's next hard fork (CHIP-0049, in review) | `sha256tree` is a table operator with the wheel's semantics, cost constants, and opcode | Covenant recursion computes program commitments in-program, the pattern behind upstream's own promotion of the operator. Adopting the upstream opcode, semantics, and constants keeps the operator inside the diffable intersection once upstream activates. Decision by Evan, ratified, see section 8. | `vm/sha256tree.json` | ## 7. Oracle provenance @@ -526,6 +556,19 @@ official vectors and votes on every generated triple. The leg is opportunistic on developer machines, required in CI, and the pinned oracles never depend on it. +`sha256tree` is tested against the pinned consensus wheel itself. +The released artifact carries the operator behind its +`ENABLE_SHA256_TREE` flag and separately exports the `tree_hash` +puzzle-hash utility, the algorithm Chia consensus has applied to +puzzle commitments since genesis. `tools/diff_sha256tree.py` diffs +the operator's (result, cost) against the flag-enabled wheel at the +exact budget boundary, the result against the utility, and the +result against an in-language tree-hash program built from +intersection operators and run through both pinned oracles at flags +0. The operator is therefore pinned by released-binary evidence on +three independent legs even though no deployed VM dispatches the +opcode yet. + ## 8. Design decision record Decisions taken during Phase 1, each ratified or explicitly left @@ -731,3 +774,51 @@ phase that owes the answer. Mirroring the deserializer's node budget is the strongest candidate since its trigger is input size, not cost, and the input-size bound belongs to the embedding rather than this spec. +7. **D9 (sha256tree adoption).** RATIFIED (decision by Evan, + 2026-07-29): `sha256tree` joins the v0 table at `0x3f` with the + pinned wheel's semantics and cost constants. + + - In-language tree hashing (recursive `sha256` over the leaf and + pair tags) is expressible with the v0 intersection, so the + operator adds cost efficiency and witness compactness, not + capability. It was adopted anyway because the demand is + structural: covenant recursion computes a child program's + commitment in-program, the dominant pattern in Chia's deployed + puzzles, and upstream is promoting the same operator to + consensus (CHIP-0049) on that evidence. Carrying the recursive + program in every witness that needs it spends bytes exactly + where the witness-size obligation is tightest. + - Opcode `0x3f` matches the upstream assignment, so the operator + joins the diffable intersection when upstream activates. The + probe record in the D2 entry above lists `0x3f` among the bytes + unassigned in both oracles: that statement described flags-0 + dispatch and stands, the wheel carries the operator behind a + release flag. + - Semantics and constants were verified against the pinned wheel + by probe on 2026-07-29 (tree hashes, totals at the exact budget + boundary, arity errors, flags-0 unknown acceptance) and are + pinned continuously by `tools/diff_sha256tree.py` (section 7). + CHIP-0049 is still in review upstream, so a constants change + before activation is possible: that lands as an ordinary + pin-bump triage, adopt or keep, one reviewed commit, and Phase 3 + re-measures every inherited constant regardless. + - The algorithm is Chia's puzzle-hash tree hash unchanged. A + tagged-hash variant with a protocol-specific prefix was + considered and declined: it would forfeit the released-binary + oracle and the future intersection to buy cross-protocol domain + separation the commitment context already provides. Recorded + consequence: a BitLisp node and a Chia node with equal trees + share a tree hash. + - The per-node charge rule exists to close a work-amplification + hazard, not only to price the hashing. Evaluation builds shared + structure cheaply: `(c 1 1)` doubles the environment reachable + from its result for one 50-cost cons, and k nested applies of + that shape, a few hundred cost units each, reach 2^k visited + nodes for build cost linear in k. The wire format's lack of + back-references is no defense, because the sharing is built by + evaluation, never spelled in the witness. The walk charges + every visited node as it is reached and stops at + `cost_exceeded`, bounding the work a budget can buy. An + implementation that hashes first and charges after is open to + unbounded work under a small budget, the same hazard class as + the substr copy-on-slice note in COSTS.md section 5. From aae65a74854e194ff1f71d879de0368acf10ad2a Mon Sep 17 00:00:00 2001 From: Evan Date: Wed, 29 Jul 2026 16:17:07 -0700 Subject: [PATCH 2/9] VM: implement sha256tree The walk mirrors the machine's explicit-stack rule so argument depth is never limited by the Python recursion limit, charges each visited node as it reaches it with the base cost riding on the first node's charge, and never deduplicates sharing: a node reachable twice is charged and hashed twice. Constants come from the consensus oracle's flag-gated operator, verified by probe against the released wheel (results, boundary totals, arity errors) before this commit. Upstream source (clvm_rs treehash.rs) informed the traversal and charge interleaving, and the released binary confirmed every constant and ordering, per the reference-material guardrails. Spec: VM.md section 4 (operator table, crypto family), section 6 row D9, section 8 entry 7, COSTS.md sections 1 and 8. --- python/bitlisp/costs.py | 11 +++++++++++ python/bitlisp/operators.py | 38 +++++++++++++++++++++++++++++++++++++ 2 files changed, 49 insertions(+) diff --git a/python/bitlisp/costs.py b/python/bitlisp/costs.py index 1c70197..c0d7866 100644 --- a/python/bitlisp/costs.py +++ b/python/bitlisp/costs.py @@ -57,6 +57,17 @@ # fixed by the operator's shape checks. SECP_VERIFY_COST = 1_300_000 +# sha256tree charges per visited node during its walk: the base cost +# rides on the first node's charge, each visited pair charges the +# pair cost, each visited atom charges the per-byte cost on its +# length plus one for the leaf tag byte, and the 32-byte result +# charges plain malloc. The constants are the consensus oracle's own +# sha256tree, carried behind its release flag (a recorded +# divergence: at flags 0 the oracle treats the opcode as unknown). +SHA256TREE_BASE_COST = 270 +SHA256TREE_PAIR_COST = 460 +SHA256TREE_COST_PER_BYTE = 2 + GRS_BASE_COST = 117 GRS_COST_PER_BYTE = 1 SUBSTR_COST = 1 diff --git a/python/bitlisp/operators.py b/python/bitlisp/operators.py index 76a97e9..22c785a 100644 --- a/python/bitlisp/operators.py +++ b/python/bitlisp/operators.py @@ -552,6 +552,43 @@ def op_secp_verify(args, charge): return TRUE +def op_sha256tree(args, charge): + # The arity check is the only check and precedes every charge: + # any node is a legal argument, hashing structure is the + # operator's purpose, so there is no arg_not_atom path. The walk + # charges each node as it reaches it, the base cost riding on + # the first node's charge, because evaluation builds shared + # structure cheaply (one cons of a node to itself doubles the + # reachable tree) and only walk-time charging keeps the hashing + # work bounded by the budget. Sharing is never deduplicated: a + # node the walk reaches twice is charged and hashed twice. The + # stack mirrors the machine's explicit-stack rule, so argument + # depth is not limited by the Python recursion limit. + _exactly(args, 1, "sha256tree") + pending = costs.SHA256TREE_BASE_COST + hashes = [] + stack = [(False, args[0])] + while stack: + combine, node = stack.pop() + if combine: + first = hashes.pop() + rest = hashes.pop() + hashes.append(hashlib.sha256(b"\x02" + first + rest).digest()) + elif is_pair(node): + charge(pending + costs.SHA256TREE_PAIR_COST) + pending = 0 + stack.append((True, None)) + stack.append((False, node[0])) + stack.append((False, node[1])) + else: + charge(pending + costs.SHA256TREE_COST_PER_BYTE * (len(node) + 1)) + pending = 0 + hashes.append(hashlib.sha256(b"\x01" + node).digest()) + result = hashes[0] + _malloc(charge, result) + return result + + def op_raise(args, charge): raise BitLispError("user_raise", "clvm raise") @@ -585,4 +622,5 @@ def op_raise(args, charge): b"\x20": op_not, b"\x21": op_any, b"\x22": op_all, + b"\x3f": op_sha256tree, } From 0f4c7e81380aead51c0a76eddd417a4623cec9a5 Mon Sep 17 00:00:00 2001 From: Evan Date: Wed, 29 Jul 2026 16:17:07 -0700 Subject: [PATCH 3/9] Vectors: pin sha256tree Twenty-one cases in vectors/vm/sha256tree.json: the two COSTS.md worked examples, leaf-tag and actual-bytes-as-given rules (redundant and minimal integer spellings hash differently), pair-tag child order, an evaluated pair argument from the environment, hash-of-hash composition, the shared-subtree double charge under (c 1 1), the inclusive budget boundary at the exact total and one below, all three arity shapes including arity beating the budget at max_cost 1, the zero-budget dispatch burst (D7 interaction), a raising argument beating the operator, and the exponential shared-environment DAG (forty nested applies doubling the environment per level) reporting cost_exceeded under a 500000 budget, the fail-fast rule of COSTS.md section 8. Every success case and every oracle-expressible error case was cross-checked against the flag-enabled consensus wheel (result and cost) and the tree_hash utility at generation time. Spec: VM.md sections 4 and 6 (D9), COSTS.md sections 1 and 8. --- vectors/vm/sha256tree.json | 194 +++++++++++++++++++++++++++++++++++++ 1 file changed, 194 insertions(+) create mode 100644 vectors/vm/sha256tree.json diff --git a/vectors/vm/sha256tree.json b/vectors/vm/sha256tree.json new file mode 100644 index 0000000..30cd140 --- /dev/null +++ b/vectors/vm/sha256tree.json @@ -0,0 +1,194 @@ +{ + "schema": "bitlisp-vector-v0", + "suite": "vm", + "spec": "VM.md sections 4 and 6 (D9) and COSTS.md sections 1 and 8 (sha256tree)", + "cases": [ + { + "name": "nil_leaf_worked_example", + "program": "ff3fffff018080", + "env": "80", + "expect": { + "result": "a04bf5122f344554c53bde2ebb8cd2b7e3d1600ad631c385a5d7cce23c7785459a", + "cost": 613 + } + }, + { + "name": "list_worked_example", + "program": "ff3fffff01ff826162ff8263648080", + "env": "80", + "expect": { + "result": "a0cc4dfd790f10aa8891ba1f939ad9749f67216b0da1392bf005faa5d0d3f7c765", + "cost": 1545 + } + }, + { + "name": "one_byte_atom", + "program": "ff3fffff010780", + "env": "80", + "expect": { + "result": "a0ca6c6588fa01171b200740344d354e8548b7470061fb32a34f4feee470ec281f", + "cost": 615 + } + }, + { + "name": "thirty_three_byte_atom", + "program": "ff3fffff01a1000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f2080", + "env": "80", + "expect": { + "result": "a0f3bda57ba54ba36b0e594d62929eabbb4ebff19ed7b7ac59ea5eb45b8eeda2de", + "cost": 679 + } + }, + { + "name": "redundant_encoding_two_bytes", + "program": "ff3fffff0182000280", + "env": "80", + "expect": { + "result": "a016305a6b2292e931665a34089864ee3fcc24171c81551f152cf4d1db0dd6122d", + "cost": 617 + } + }, + { + "name": "minimal_encoding_one_byte", + "program": "ff3fffff010280", + "env": "80", + "expect": { + "result": "a0a12871fee210fb8619291eaea194581cbd2531e4b23759d225f6806923f63222", + "cost": 615 + } + }, + { + "name": "simple_pair", + "program": "ff3fffff01ff616280", + "env": "80", + "expect": { + "result": "a0ddf7d5e743e693e9a9bde3c22082fc8776c215616943488c9ae75affcd91dbca", + "cost": 1079 + } + }, + { + "name": "swapped_pair_differs", + "program": "ff3fffff01ff626180", + "env": "80", + "expect": { + "result": "a021c95f199fc30b38419f654eea4e6d87a59f774e46657c7ecd9757560eb7ba7f", + "cost": 1079 + } + }, + { + "name": "nested_left", + "program": "ff3fffff01ffff61626380", + "env": "80", + "expect": { + "result": "a0890c749cefaeb80d8fd17954bff3b34ce217f8a30af1a8b035e015ecbf3a78b1", + "cost": 1543 + } + }, + { + "name": "nested_right", + "program": "ff3fffff01ff61ff626380", + "env": "80", + "expect": { + "result": "a0d08f3f683951f120e10a4e6c021ebd5e5cd3e3855db2b7208d87c9992771affb", + "cost": 1543 + } + }, + { + "name": "argument_from_env_path", + "program": "ff3fff0180", + "env": "ff826162ff82636480", + "expect": { + "result": "a0cc4dfd790f10aa8891ba1f939ad9749f67216b0da1392bf005faa5d0d3f7c765", + "cost": 1569 + } + }, + { + "name": "composed_hash_of_hash", + "program": "ff3fffff3fffff01836162638080", + "env": "80", + "expect": { + "result": "a06c669f95b6ad7dc40e773854d5f017bd5e870f739a49c2ac2f743026dfa4a6c0", + "cost": 1276 + } + }, + { + "name": "shared_subtree_charged_twice", + "program": "ff3fffff04ff01ff018080", + "env": "ff826162826364", + "expect": { + "result": "a0085cc3e6b2f9dd204ce8845a59420d8874b86b3ff4bb28a65f0c4d503656852e", + "cost": 2134 + } + }, + { + "name": "budget_exact_succeeds", + "program": "ff3fffff01ff826162ff8263648080", + "env": "80", + "max_cost": 1545, + "expect": { + "result": "a0cc4dfd790f10aa8891ba1f939ad9749f67216b0da1392bf005faa5d0d3f7c765", + "cost": 1545 + } + }, + { + "name": "budget_one_short_cost_exceeded", + "program": "ff3fffff01ff826162ff8263648080", + "env": "80", + "max_cost": 1544, + "expect": { + "error": "cost_exceeded" + } + }, + { + "name": "zero_args_wrong_arg_count", + "program": "ff3f80", + "env": "80", + "expect": { + "error": "wrong_arg_count" + } + }, + { + "name": "two_args_wrong_arg_count", + "program": "ff3fffff0161ffff016280", + "env": "80", + "expect": { + "error": "wrong_arg_count" + } + }, + { + "name": "arity_check_wins_over_budget", + "program": "ff3f80", + "env": "80", + "max_cost": 1, + "expect": { + "error": "wrong_arg_count" + } + }, + { + "name": "zero_budget_dispatch_bursts", + "program": "ff3f80", + "env": "80", + "max_cost": 0, + "expect": { + "error": "cost_exceeded" + } + }, + { + "name": "raising_argument_wins", + "program": "ff3fffff088080", + "env": "80", + "expect": { + "error": "user_raise" + } + }, + { + "name": "exponential_shared_env_fails_fast", + "program": "ff3fffff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff01ff02ffff0101ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff018080ffff04ff01ff01808080", + "env": "8473656564", + "max_cost": 500000, + "expect": { + "error": "cost_exceeded" + } + } + ] +} From fc050cb50171af1de28d1b126a7b1ca41709bdce Mon Sep 17 00:00:00 2001 From: Evan Date: Wed, 29 Jul 2026 16:17:07 -0700 Subject: [PATCH 4/9] Tools: sha256tree differential harness, CI leg, smoke test tools/diff_sha256tree.py pins the operator on three released-binary legs: (cost, result) against the pinned wheel's flag-enabled dispatch including budgets within a few units of the measured cost, results against the wheel's tree_hash puzzle-hash utility, and results against an in-language tree-hash program built from intersection operators and run through bitlisp and both pinned oracles at flags 0. The generator draws atom sizes across the serialization forms, redundant integer spellings, and (c X X) shared-structure chains where the walk charges per visit. Every run also probes arity defects and an exponential shared-environment DAG (25 to 50 nested doublings under a small budget), where both sides must report cost_exceeded in bounded time: an implementation that hashed before charging would hang the harness instead. CI runs 400 cases per push with the run id as seed, printed for local reproduction, alongside the existing diff legs. The unit suite gains a fixed-seed 150-case smoke slice so pytest alone stays a complete local gate. Spec: VM.md section 7 (oracle strategy for D9). --- .github/workflows/ci.yml | 13 ++ python/tests/test_differential.py | 25 +++ tools/diff_sha256tree.py | 311 ++++++++++++++++++++++++++++++ 3 files changed, 349 insertions(+) create mode 100644 tools/diff_sha256tree.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9e5d151..a9ebf56 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -91,3 +91,16 @@ jobs: echo "diff_secp seed: ${DIFF_SEED}" "${RUNNER_TEMP}/venv/bin/python" tools/diff_secp.py \ --count 40 --seed "${DIFF_SEED}" --require-libsecp + - name: sha256tree differential, flag-gated oracle + # sha256tree sits outside the flags-0 intersection (divergence + # D9): the differential runs against the pinned wheel's + # flag-enabled operator, its tree_hash utility, and an + # in-language tree-hash program on both oracles at flags 0. + # Reproduce locally with + # tools/diff_sha256tree.py --count 400 --seed . + env: + DIFF_SEED: ${{ github.run_id }} + run: | + echo "diff_sha256tree seed: ${DIFF_SEED}" + "${RUNNER_TEMP}/venv/bin/python" tools/diff_sha256tree.py \ + --count 400 --seed "${DIFF_SEED}" diff --git a/python/tests/test_differential.py b/python/tests/test_differential.py index 68ea6c9..39d6e4c 100644 --- a/python/tests/test_differential.py +++ b/python/tests/test_differential.py @@ -19,9 +19,34 @@ # instead of failing collection when only `dev` is installed. pytest.importorskip("chia_rs") import diff_clvm # noqa: E402 +import diff_sha256tree # noqa: E402 from diff_clvm import Generator, run_bitlisp, run_rs # noqa: E402 +def test_fixed_seed_sha256tree_agrees_with_flag_enabled_oracle(): + # A fast fixed slice of tools/diff_sha256tree.py: trees through + # the operator versus the wheel's flag-enabled dispatch, and + # results versus its tree_hash puzzle-hash utility. + import chia_rs + + flag = chia_rs.ENABLE_SHA256_TREE + rng = random.Random(7777) + gen = diff_sha256tree.TreeGenerator(rng, max_depth=4) + mismatches = [] + for i in range(150): + tree = gen.tree(rng.randint(0, 4)) + program = diff_sha256tree.serialize( + diff_sha256tree.lst(b"\x3f", (b"\x01", tree)) + ) + env = diff_sha256tree.serialize(b"") + bl = diff_sha256tree.run_bitlisp(program, env, 11_000_000_000) + rs = diff_sha256tree.run_rs(program, env, 11_000_000_000, flag) + digest = bytes(chia_rs.tree_hash(diff_sha256tree.serialize(tree))).hex() + if bl != rs or bl[0] != "ok" or bl[2] != "a0" + digest: + mismatches.append((i, program.hex(), bl, rs, digest)) + assert not mismatches, mismatches[:3] + + def test_fixed_seed_corpus_agrees_with_consensus_oracle(): rng = random.Random(7777) gen = Generator(rng, max_depth=5) diff --git a/tools/diff_sha256tree.py b/tools/diff_sha256tree.py new file mode 100644 index 0000000..4d3c767 --- /dev/null +++ b/tools/diff_sha256tree.py @@ -0,0 +1,311 @@ +#!/usr/bin/env python3 +"""Differential harness for sha256tree. + +Deployed consensus at flags 0 treats opcode 0x3f as unknown (a +recorded divergence), so the operator cannot ride the intersection +harness. It is pinned against the released oracle artifacts on three +legs instead: + +1. The pinned consensus wheel dispatches the same operator behind its + ENABLE_SHA256_TREE release flag. Every generated program runs + through bitlisp and the flag-enabled wheel, and the (cost, result) + or error class must match exactly, including at budgets within a + few units of the measured cost, where the charge interleaving + decides the outcome. +2. The wheel separately exports the tree_hash puzzle-hash utility, + the algorithm Chia consensus has applied to puzzle commitments + since genesis. Every hashed tree's result must equal the utility's + digest. +3. An in-language tree-hash program built from intersection operators + (a, i, l, c, sha256, paths) runs through bitlisp and both pinned + oracles at flags 0, and every implementation's result must equal + the operator's. This leg ties the operator to semantics every + deployed binary can confirm without the flag. + +The generator draws leaf-heavy and pair-heavy trees, atom sizes that +cross the one-byte and length-prefixed serialization forms, redundant +integer encodings, and cons-built shared structure ((c X X) chains), +where the walk must charge a node once per visit. Arity defects run +against the flag-enabled wheel each round. Any disagreement fails the +run and prints the case with the seed, so a failure reproduces with: + + tools/diff_sha256tree.py --count 400 --seed +""" + +import argparse +import io +import random +import sys +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parent.parent +sys.path.insert(0, str(REPO_ROOT / "python")) + +import chia_rs # noqa: E402 +from bitlisp import BitLispError, run_serialized, serialize # noqa: E402 +from clvm import SExp # noqa: E402 +from clvm import run_program as clvm_run_program # noqa: E402 +from clvm.operators import OPERATOR_LOOKUP # noqa: E402 +from clvm.serialize import sexp_from_stream, sexp_to_stream # noqa: E402 + +MAX_COST = 11_000_000_000 +OP = b"\x3f" +NIL = b"" +Q = b"\x01" + + +def q(node): + return (Q, node) + + +def lst(*items): + node = NIL + for item in reversed(items): + node = (item, node) + return node + + +# The in-language tree hash, leg 3. Env shape is (self tree): path 2 +# is self, path 5 the tree, paths 9 and 13 the tree's children. An +# atom hashes as (sha256 (q . 1) 5), a pair recurses through apply on +# both children and hashes the tag byte 2 with the two child digests. +ATOM_BRANCH = lst(b"\x0b", q(b"\x01"), b"\x05") +PAIR_BRANCH = lst( + b"\x0b", + q(b"\x02"), + lst(b"\x02", b"\x02", lst(b"\x04", b"\x02", lst(b"\x04", b"\x09", q(NIL)))), + lst(b"\x02", b"\x02", lst(b"\x04", b"\x02", lst(b"\x04", b"\x0d", q(NIL)))), +) +SELF_PROG = lst( + b"\x02", + lst(b"\x03", lst(b"\x07", b"\x05"), q(PAIR_BRANCH), q(ATOM_BRANCH)), + b"\x01", +) + + +def in_language_program(tree): + return lst( + b"\x02", + q(SELF_PROG), + lst(b"\x04", q(SELF_PROG), lst(b"\x04", q(tree), q(NIL))), + ) + + +def run_bitlisp(program, env, max_cost): + try: + cost, result = run_serialized(program, env, max_cost) + return ("ok", cost, result.hex()) + except BitLispError as exc: + return ("err", exc.code) + + +def run_rs(program, env, max_cost, flags): + try: + cost, node = chia_rs.run_chia_program(program, env, max_cost, flags) + except Exception as exc: # chia_rs raises ValueError + message = str(exc) + if "cost exceeded" in message: + return ("err", "cost_exceeded") + if "takes exactly" in message: + return ("err", "wrong_arg_count") + return ("err", message) + out = bytearray() + stack = [node] + while stack: + n = stack.pop() + if n.atom is not None: + buf = io.BytesIO() + sexp_to_stream(SExp.to(n.atom), buf) + out += buf.getvalue() + else: + out.append(0xFF) + left, right = n.pair + stack.append(right) + stack.append(left) + return ("ok", cost, bytes(out).hex()) + + +def run_py(program, env, max_cost): + try: + prog = sexp_from_stream(io.BytesIO(program), SExp.to) + env_node = sexp_from_stream(io.BytesIO(env), SExp.to) + cost, result = clvm_run_program( + prog, env_node, OPERATOR_LOOKUP, max_cost=max_cost + ) + except Exception as exc: + return ("err", str(exc)) + buf = io.BytesIO() + sexp_to_stream(result, buf) + return ("ok", cost, buf.getvalue().hex()) + + +class TreeGenerator: + def __init__(self, rng, max_depth): + self.rng = rng + self.max_depth = max_depth + + def atom(self): + r = self.rng + roll = r.random() + if roll < 0.3: + return b"" if r.random() < 0.3 else bytes([r.randint(0, 255)]) + if roll < 0.75: + size = r.randint(2, 40) + elif roll < 0.95: + # Crosses the one-byte serialization form's 0x40 ceiling. + size = r.randint(41, 300) + else: + size = r.randint(301, 3000) + data = r.randbytes(size) + if r.random() < 0.15: + # Redundant integer spellings are legal leaves and must + # hash as given. + data = (b"\x00" if r.random() < 0.5 else b"\xff") * r.randint(1, 3) + data + return data + + def tree(self, depth): + if depth <= 0 or self.rng.random() < 0.4: + return self.atom() + return (self.tree(depth - 1), self.tree(depth - 1)) + + def shared_program(self): + # (c X X) chains reach 2^k leaf-tree visits for k conses: the + # walk must charge per visit, and the budget must bound the + # work. Half the chains hash a quoted tree, half the + # environment. + r = self.rng + k = r.randint(1, 8) + node = q(self.tree(2)) if r.random() < 0.5 else b"\x01" + for _ in range(k): + node = lst(b"\x04", node, node) + return lst(OP, node) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--count", type=int, default=400) + parser.add_argument("--seed", type=int, default=1) + parser.add_argument("--max-depth", type=int, default=6) + args = parser.parse_args() + + flag = chia_rs.ENABLE_SHA256_TREE + rng = random.Random(args.seed) + gen = TreeGenerator(rng, args.max_depth) + failures = 0 + stats = { + "full": 0, + "boundary": 0, + "utility": 0, + "in_language": 0, + "arity": 0, + "exponential": 0, + } + + def fail(kind, detail): + nonlocal failures + failures += 1 + print(f"MISMATCH {kind} seed={args.seed}: {detail}") + + for i in range(args.count): + shared = rng.random() < 0.25 + if shared: + program_node = gen.shared_program() + env_node = gen.tree(2) + else: + tree = gen.tree(rng.randint(0, args.max_depth)) + program_node = lst(OP, q(tree)) + env_node = NIL + program = serialize(program_node) + env = serialize(env_node) + + bl = run_bitlisp(program, env, MAX_COST) + rs = run_rs(program, env, MAX_COST, flag) + if bl != rs: + fail("full", f"#{i} prog={program.hex()} env={env.hex()} {bl} vs {rs}") + continue + stats["full"] += 1 + + if bl[0] == "ok": + budget = bl[1] + rng.choice((-2, -1, 0)) + bl_b = run_bitlisp(program, env, budget) + rs_b = run_rs(program, env, budget, flag) + if bl_b != rs_b: + fail( + "boundary", + f"#{i} prog={program.hex()} env={env.hex()} " + f"budget={budget} {bl_b} vs {rs_b}", + ) + else: + stats["boundary"] += 1 + + if not shared and bl[0] == "ok": + digest = bytes(chia_rs.tree_hash(serialize(tree))).hex() + if bl[2] != "a0" + digest: + fail( + "utility", f"#{i} tree={serialize(tree).hex()} {bl[2]} vs {digest}" + ) + else: + stats["utility"] += 1 + + if not shared and bl[0] == "ok" and rng.random() < 0.25: + ref_program = serialize(in_language_program(tree)) + ref_bl = run_bitlisp(ref_program, env, MAX_COST) + ref_rs = run_rs(ref_program, env, MAX_COST, 0) + ref_py = run_py(ref_program, env, MAX_COST) + outcomes = { + "bitlisp": ref_bl[2] if ref_bl[0] == "ok" else ref_bl, + "chia_rs": ref_rs[2] if ref_rs[0] == "ok" else ref_rs, + "clvm": ref_py[2] if ref_py[0] == "ok" else ref_py, + } + if any(value != bl[2] for value in outcomes.values()): + fail( + "in_language", + f"#{i} tree={serialize(tree).hex()} op={bl[2]} {outcomes}", + ) + else: + stats["in_language"] += 1 + + for argc in (0, 2, 3): + node = NIL + for _ in range(argc): + node = (q(gen.atom()), node) + program = serialize((OP, node)) + env = serialize(NIL) + bl = run_bitlisp(program, env, MAX_COST) + rs = run_rs(program, env, MAX_COST, flag) + if bl != rs or bl != ("err", "wrong_arg_count"): + fail("arity", f"argc={argc} {bl} vs {rs}") + else: + stats["arity"] += 1 + + # Exponential shared-environment DAG: k nested applies of + # (a (q . inner) (c 1 1)) double the reachable environment per + # level in linear program bytes, so the walk faces 2^k visits + # under a budget that covers a few thousand. Both sides must + # report cost_exceeded, and must do so in bounded time: an + # implementation that hashes before charging would hang here. + k = rng.randint(25, 50) + node = b"\x01" + for _ in range(k): + node = lst(b"\x02", q(node), lst(b"\x04", b"\x01", b"\x01")) + program = serialize(lst(OP, node)) + env = serialize(gen.atom()) + budget = rng.randint(50_000, 2_000_000) + bl = run_bitlisp(program, env, budget) + rs = run_rs(program, env, budget, flag) + if bl != rs or bl != ("err", "cost_exceeded"): + fail("exponential", f"k={k} budget={budget} {bl} vs {rs}") + else: + stats["exponential"] = 1 + + print( + f"diff_sha256tree: seed={args.seed}, {stats['full']} full, " + f"{stats['boundary']} boundary, {stats['utility']} utility, " + f"{stats['in_language']} in-language, {stats['arity']} arity, " + f"{stats['exponential']} exponential, {failures} failures" + ) + return 1 if failures else 0 + + +if __name__ == "__main__": + sys.exit(main()) From c46a8158c366965d589dd2ba99d4da1181de71c8 Mon Sep 17 00:00:00 2001 From: Evan Date: Wed, 29 Jul 2026 16:57:07 -0700 Subject: [PATCH 5/9] Spec: post-review corrections to the sha256tree text Findings from the five-reviewer pass on this PR, each verified before fixing. The section 5 error-taxonomy row scoped arg_not_atom to the whole crypto family, contradicting the section 4 statement that sha256tree has no arg_not_atom path: the row now excepts it, the reverse of the edit that widened the row when secp_verify landed. The D2 record's family enumeration (sha256 plus secp_verify, nothing else) is now explicitly amended by entry 7 rather than left silently overturned, per the section 8 preamble's amendment rule. The rewritten boolean-family sentence claimed only booleans and sha256tree accept pair arguments outside the tree ops, false given apply's environment argument and raise's arguments: the exclusivity claim is dropped. Section 7 no longer calls the three legs independent, since two are views of the one pinned wheel and only one checks cost, and it now describes the fourth leg added in this PR: every generated program runs through both oracles at flags 0, pinning the D9 oracle column continuously instead of by a one-time probe. The D9 row's vector cell now says which column the vector file pins. --- spec/VM.md | 35 ++++++++++++++++++++--------------- 1 file changed, 20 insertions(+), 15 deletions(-) diff --git a/spec/VM.md b/spec/VM.md index f74fbc8..5b52b28 100644 --- a/spec/VM.md +++ b/spec/VM.md @@ -363,10 +363,9 @@ byte, so the section 2 threshold for building an atom the wire format cannot encode applies unchanged. The boolean ops test nil-ness and return the shared TRUE and nil -constants. Outside the tree ops, only they and `sha256tree` accept -pair arguments: any node is legal in any position, and nil is the -only false value (the one-byte atom `0x00` and every pair are true), -the same rule `i` applies to its selector. +constants. Any node is legal in any position, and nil is the only +false value (the one-byte atom `0x00` and every pair are true), the +same rule `i` applies to its selector. - `not` takes exactly one argument: TRUE if it is nil, nil otherwise. - `any` returns TRUE if at least one argument is not nil, and nil @@ -469,7 +468,7 @@ informative, not normative. | `unknown_operator` | Operator atom not in the table and not reserved | (accepted, D3) | (accepted, D3) | | `bad_arg_list` | Operator arguments are not a proper list | (varies) | (varies) | | `wrong_arg_count` | Operator arity violated | InvalidOperatorArg | (per-op message) | -| `arg_not_atom` | An atom-only operator (`=`, the integer family, the bytes family outside `substr`'s index positions, the bitwise family outside shift count positions, or the crypto family) got a pair | InvalidOperatorArg | (per-op message) | +| `arg_not_atom` | An atom-only operator (`=`, the integer family, the bytes family outside `substr`'s index positions, the bitwise family outside shift count positions, or the crypto family outside `sha256tree`) got a pair | InvalidOperatorArg | (per-op message) | | `arg_not_pair` | `f` or `r` applied to an atom | InvalidOperatorArg: first/rest of non-cons | first/rest of non-cons | | `arg_too_long` | Operand exceeds a section 4 size limit | InvalidOperatorArg | (absent, the `clvm` package has no operand limits) | | `bad_index` | A `substr` index or a shift count argument is a pair or an atom longer than four bytes | (per-op) requires int32 args (with no leading zeros) | (per-op) requires int32 args | @@ -507,7 +506,7 @@ pin it. No divergence exists outside this table. "Both oracles" means | D6 | `/` with negative operands | Consensus (`chia-rs`): floor division. The `clvm` package injects a policy error ("deprecated") that is not consensus | Floor division, matching consensus | Intersection parity targets the consensus oracle. The Python package's rejection is library policy, the diff harness treats it as an expected divergence. Ratified, see section 8. | `vm/arith.json`, upstream corpus D6 bucket | | D7 | Zero cost budget | Both oracles treat `max_cost = 0` as unlimited | A zero budget is a real budget, no program succeeds under it (section 3.3) | A zero sentinel meaning unlimited is a library convenience, not consensus behavior. In the Bitcoin context the budget derives from transaction weight and is never legitimately zero, and an accidental zero must fail closed rather than open. Ratified, see section 8. | `vm/dispatch.json` | | D8 | Resource limits outside the cost model | The consensus oracle enforces caps the cost model never sees: at most 62,500,000 atoms and as many pairs per run (deserialization spends one count per atom and two per cons, probed at the boundary: a 62.7 million node budget fails "too many pairs" before evaluation, 62.4 million deserializes), a 4 GiB atom-byte heap, 20,000,000-entry value and environment stacks, and a two-argument `substr` whose default end index passes through a signed 32-bit cast, rejecting data atoms of 2^31 bytes or more | No equivalent limits: BitLisp is bounded by the cost budget, and its deserializer by the input's size alone | Every cap sits far outside the reachable regime. The cheapest evaluation-time trigger costs about 5.6e10 against the harness budget of 1.1e10, and the deserialization trigger needs roughly 42 MB of input against Bitcoin's 4 MB witness ceiling. PROVISIONAL, see section 8: the Phase 3 budget and input-size bounds must be recorded against these thresholds, or the caps mirrored fail-closed. | none, unreachable (section 8) | -| D9 | `sha256tree` | Deployed consensus (flags 0) treats opcode `0x3f` as an unknown operator under the D3 acceptance rule: arguments evaluate, cost derives from the opcode byte, result nil. The pinned oracle wheel carries a `sha256tree` operator at the same opcode behind its release flag, scheduled for consensus activation in Chia's next hard fork (CHIP-0049, in review) | `sha256tree` is a table operator with the wheel's semantics, cost constants, and opcode | Covenant recursion computes program commitments in-program, the pattern behind upstream's own promotion of the operator. Adopting the upstream opcode, semantics, and constants keeps the operator inside the diffable intersection once upstream activates. Decision by Evan, ratified, see section 8. | `vm/sha256tree.json` | +| D9 | `sha256tree` | Deployed consensus (flags 0) treats opcode `0x3f` as an unknown operator under the D3 acceptance rule: arguments evaluate, cost derives from the opcode byte, result nil. The pinned oracle wheel carries a `sha256tree` operator at the same opcode behind its release flag, scheduled for consensus activation in Chia's next hard fork (CHIP-0049, in review) | `sha256tree` is a table operator with the wheel's semantics, cost constants, and opcode | Covenant recursion computes program commitments in-program, the pattern behind upstream's own promotion of the operator. Adopting the upstream opcode, semantics, and constants keeps the operator inside the diffable intersection once upstream activates. Decision by Evan, ratified, see section 8. | `vm/sha256tree.json` (BitLisp column), the oracle column pinned per run by the flags-0 leg of `tools/diff_sha256tree.py` | ## 7. Oracle provenance @@ -560,14 +559,18 @@ pinned oracles never depend on it. The released artifact carries the operator behind its `ENABLE_SHA256_TREE` flag and separately exports the `tree_hash` puzzle-hash utility, the algorithm Chia consensus has applied to -puzzle commitments since genesis. `tools/diff_sha256tree.py` diffs -the operator's (result, cost) against the flag-enabled wheel at the -exact budget boundary, the result against the utility, and the -result against an in-language tree-hash program built from -intersection operators and run through both pinned oracles at flags -0. The operator is therefore pinned by released-binary evidence on -three independent legs even though no deployed VM dispatches the -opcode yet. +puzzle commitments since genesis. `tools/diff_sha256tree.py` runs +four legs per corpus: the operator's (result, cost) against the +flag-enabled wheel at the exact budget boundary, the result against +the utility, the result against an in-language tree-hash program +built from intersection operators and run through both pinned +oracles at flags 0, and the D9 oracle column itself, every generated +program run through both oracles at flags 0, which must accept the +opcode as unknown and return nil. The first two legs are two views +of the one pinned wheel and only the first checks cost, so the cost +constants rest on the flag-enabled wheel alone. The other two legs +cross both oracles. No deployed VM dispatches the opcode yet, and +every leg is released-binary evidence. ## 8. Design decision record @@ -653,7 +656,9 @@ phase that owes the answer. necessary, while a shipped guard could never be removed. 2. **D2 (crypto family curation).** RATIFIED (decisions by Evan, 2026-07-28). The crypto family is `sha256` plus `secp_verify`, - nothing else, and `secp_verify` is BIP340 only. + nothing else, and `secp_verify` is BIP340 only. The family + enumeration was amended by entry 7 below: `sha256tree` joined on + 2026-07-29, and the rest of this entry stands. - **ECDSA declined.** The consensus oracle's `secp256k1_verify` and `secp256r1_verify` were probed 2026-07-28: raise-style From fa6702879a93a69388e46c8a2a001e29850c8a67 Mon Sep 17 00:00:00 2001 From: Evan Date: Wed, 29 Jul 2026 16:57:07 -0700 Subject: [PATCH 6/9] VM: correct the operators module docstring's oracle claim The docstring's two-way split (consensus-oracle order for everyone, secp_verify its own normative choice) missed the third category this PR introduced: op_sha256tree's order matches the same released wheel behind its release flag, while flags-0 consensus treats the opcode as unknown, a recorded divergence. Spec: VM.md sections 6 (D9) and 7. --- python/bitlisp/operators.py | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/python/bitlisp/operators.py b/python/bitlisp/operators.py index 22c785a..e14ce01 100644 --- a/python/bitlisp/operators.py +++ b/python/bitlisp/operators.py @@ -11,9 +11,12 @@ cost_exceeded, wrong_arg_count, arg_not_atom, arg_not_pair, arg_too_long, bad_index, index_out_of_range, shift_too_large, div_by_zero, and secp_verify_failed is reported. Every function below -performs them in the consensus oracle's order, except op_secp_verify, -which has no oracle and whose order is its own normative choice. The -boundary cases are pinned by vectors. +performs them in the consensus oracle's order, with two exceptions: +op_secp_verify has no oracle and its order is its own normative +choice, and op_sha256tree matches the same released wheel's operator +behind its release flag, since at flags 0 the opcode is unknown to +the oracle (a recorded divergence). The boundary cases are pinned by +vectors. """ import hashlib From 941a54ed1f3a42f8e0da9b761a856ab9e909de8b Mon Sep 17 00:00:00 2001 From: Evan Date: Wed, 29 Jul 2026 16:57:07 -0700 Subject: [PATCH 7/9] Vectors: spell the D9 citation, record the new file's cross-check The envelope's spec field now spells divergence D9 the way sibling files spell their rows, and the corpus README's divergence-case enumeration now includes vm/sha256tree.json with the flag-enabled cross-check that produced it. Spec: VM.md section 6 (D9). --- vectors/README.md | 6 ++++-- vectors/vm/sha256tree.json | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/vectors/README.md b/vectors/README.md index 81e29dd..32c9c48 100644 --- a/vectors/README.md +++ b/vectors/README.md @@ -52,8 +52,10 @@ interleaving rules they pin are in `spec/COSTS.md` section 1. Every intersection case was cross-checked against the consensus oracle (`chia-rs`, flags 0) when it was written. Divergence cases (`vm/serialize.json` strictness, unknown and pair operators in -`vm/dispatch.json`) pin BitLisp behavior that intentionally differs, -each cites its divergence row. +`vm/dispatch.json`, and all of `vm/sha256tree.json`, whose success +cases were cross-checked against the same wheel with its sha256tree +release flag enabled) pin BitLisp behavior that intentionally +differs, each cites its divergence row. Run the corpus with `python3 tools/run_vectors.py`. A vector file whose suite has no runner yet fails loudly rather than being skipped. diff --git a/vectors/vm/sha256tree.json b/vectors/vm/sha256tree.json index 30cd140..7848288 100644 --- a/vectors/vm/sha256tree.json +++ b/vectors/vm/sha256tree.json @@ -1,7 +1,7 @@ { "schema": "bitlisp-vector-v0", "suite": "vm", - "spec": "VM.md sections 4 and 6 (D9) and COSTS.md sections 1 and 8 (sha256tree)", + "spec": "VM.md sections 4 and 6 (divergence D9) and COSTS.md sections 1 and 8 (sha256tree)", "cases": [ { "name": "nil_leaf_worked_example", From 37461322734a5f622712348fdf44e2e045d41de7 Mon Sep 17 00:00:00 2001 From: Evan Date: Wed, 29 Jul 2026 16:57:07 -0700 Subject: [PATCH 8/9] Tools: pin the D9 oracle column with a flags-0 leg The divergence row's oracle column (deployed consensus accepts 0x3f as an unknown operator returning nil) was verified by a one-time probe but re-verified by nothing. The harness now runs every generated program through both oracles at flags 0 and requires unknown-op acceptance with a nil result, guarded on success so an erroring argument is never misread as rejection. Costs are not compared on that leg, the unknown-op charge has nothing to do with the operator's constants. Also from review: the docstring no longer claims the budget-boundary leg observes charge interleaving (only the order-independent total decides this operator's outcome), it now states which legs share the one wheel and that only the first checks cost, and the exponential stat counts like its siblings. Spec: VM.md sections 6 (D9) and 7. --- tools/diff_sha256tree.py | 40 ++++++++++++++++++++++++++++++++-------- 1 file changed, 32 insertions(+), 8 deletions(-) diff --git a/tools/diff_sha256tree.py b/tools/diff_sha256tree.py index 4d3c767..c8b5d2a 100644 --- a/tools/diff_sha256tree.py +++ b/tools/diff_sha256tree.py @@ -3,24 +3,29 @@ Deployed consensus at flags 0 treats opcode 0x3f as unknown (a recorded divergence), so the operator cannot ride the intersection -harness. It is pinned against the released oracle artifacts on three +harness. It is pinned against the released oracle artifacts on four legs instead: 1. The pinned consensus wheel dispatches the same operator behind its ENABLE_SHA256_TREE release flag. Every generated program runs through bitlisp and the flag-enabled wheel, and the (cost, result) or error class must match exactly, including at budgets within a - few units of the measured cost, where the charge interleaving - decides the outcome. + few units of the measured cost, where the budget boundary decides + the outcome. This is the only leg that checks cost, so the cost + constants rest on the flag-enabled wheel alone. 2. The wheel separately exports the tree_hash puzzle-hash utility, the algorithm Chia consensus has applied to puzzle commitments since genesis. Every hashed tree's result must equal the utility's - digest. + digest. Legs 1 and 2 are two views of the one pinned wheel. 3. An in-language tree-hash program built from intersection operators (a, i, l, c, sha256, paths) runs through bitlisp and both pinned oracles at flags 0, and every implementation's result must equal the operator's. This leg ties the operator to semantics every deployed binary can confirm without the flag. +4. The divergence itself: every generated program also runs through + both oracles at flags 0, which must accept the opcode as unknown + and return nil. This pins the recorded oracle-side behavior, that + deployed consensus does not dispatch 0x3f, on every run. The generator draws leaf-heavy and pair-heavy trees, atom sizes that cross the one-byte and length-prefixed serialization forms, redundant @@ -199,6 +204,7 @@ def main(): "in_language": 0, "arity": 0, "exponential": 0, + "flags0": 0, } def fail(kind, detail): @@ -225,6 +231,23 @@ def fail(kind, detail): continue stats["full"] += 1 + # Leg 4: at flags 0 both oracles must treat 0x3f as an + # unknown operator, accepted with result nil, the oracle side + # of the recorded divergence. Costs are not compared, the + # unknown-op charge has nothing to do with the operator's. + # Guarded on success so an erroring argument's failure is + # never misread as unknown-op rejection. + if bl[0] == "ok": + rs0 = run_rs(program, env, MAX_COST, 0) + py0 = run_py(program, env, MAX_COST) + if rs0[0] != "ok" or rs0[2] != "80" or py0[0] != "ok" or py0[2] != "80": + fail( + "flags0", + f"#{i} prog={program.hex()} env={env.hex()} rs={rs0} py={py0}", + ) + else: + stats["flags0"] += 1 + if bl[0] == "ok": budget = bl[1] + rng.choice((-2, -1, 0)) bl_b = run_bitlisp(program, env, budget) @@ -296,13 +319,14 @@ def fail(kind, detail): if bl != rs or bl != ("err", "cost_exceeded"): fail("exponential", f"k={k} budget={budget} {bl} vs {rs}") else: - stats["exponential"] = 1 + stats["exponential"] += 1 print( f"diff_sha256tree: seed={args.seed}, {stats['full']} full, " - f"{stats['boundary']} boundary, {stats['utility']} utility, " - f"{stats['in_language']} in-language, {stats['arity']} arity, " - f"{stats['exponential']} exponential, {failures} failures" + f"{stats['flags0']} flags0, {stats['boundary']} boundary, " + f"{stats['utility']} utility, {stats['in_language']} in-language, " + f"{stats['arity']} arity, {stats['exponential']} exponential, " + f"{failures} failures" ) return 1 if failures else 0 From 13a05410ddeaa903ce82f9edfdcf855377de2e22 Mon Sep 17 00:00:00 2001 From: Evan Date: Wed, 29 Jul 2026 16:57:07 -0700 Subject: [PATCH 9/9] Plan: record the post-close operator-set amendment The Phase 1 checklist line defined the operator set as CLVM core minus BLS plus secp_verify, stale once sha256tree joined. The line now records the post-close amendment with its date and divergence row, following the plan's precedent of correcting frozen facts that drifted. --- docs/execution-plan.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/execution-plan.md b/docs/execution-plan.md index 3c999ef..33a7207 100644 --- a/docs/execution-plan.md +++ b/docs/execution-plan.md @@ -67,7 +67,7 @@ **Goal:** a minimal Python evaluator whose shared core is bit-for-bit CLVM-equivalent, with divergences enumerated. - [x] Implement the evaluator in `python/bitlisp/` — own code, not a wrapper (it is the spec artifact), small and boring: cons cells, serialization, operator dispatch, cost accounting. -- [x] Define operator set in `VM.md`: CLVM core **minus** BLS operators, **plus** `secp_verify` (BIP340, assertive semantics deferred to condition layer). Divergence table with rationale per row. +- [x] Define operator set in `VM.md`: CLVM core **minus** BLS operators, **plus** `secp_verify` (BIP340, assertive semantics deferred to condition layer). Divergence table with rationale per row. Amended after the Phase 1 close: `sha256tree` adopted 2026-07-29 (decision by Evan, VM.md divergence D9). - [x] Inherit the CLVM cost table (`COSTS.md`); weight-mapping section stubbed for Phase 3 data. - [x] **Differential harness v1** (`tools/diff_clvm.py`): run every intersection program through bitlisp-python AND `clvm`/`chia_rs`; assert identical (result, cost) or identical error class. - [x] Import Chia's official CLVM test vectors for the intersection; generate randomized program corpus (Claude Code task: corpus generator with size/depth knobs).