VM: adopt sha256tree at 0x3f (divergence D9)

.github/workflows/ci.yml

@@ -91,3 +91,16 @@ jobs:
           echo "diff_secp seed: ${DIFF_SEED}"
           "${RUNNER_TEMP}/venv/bin/python" tools/diff_secp.py \
             --count 40 --seed "${DIFF_SEED}" --require-libsecp
+      - name: sha256tree differential, flag-gated oracle
+        # sha256tree sits outside the flags-0 intersection (divergence
+        # D9): the differential runs against the pinned wheel's
+        # flag-enabled operator, its tree_hash utility, and an
+        # in-language tree-hash program on both oracles at flags 0.
+        # Reproduce locally with
+        # tools/diff_sha256tree.py --count 400 --seed <printed seed>.
+        env:
+          DIFF_SEED: ${{ github.run_id }}
+        run: |
+          echo "diff_sha256tree seed: ${DIFF_SEED}"
+          "${RUNNER_TEMP}/venv/bin/python" tools/diff_sha256tree.py \
+            --count 400 --seed "${DIFF_SEED}"

docs/execution-plan.md

@@ -67,7 +67,7 @@
 **Goal:** a minimal Python evaluator whose shared core is bit-for-bit CLVM-equivalent, with divergences enumerated.
 
 - [x] Implement the evaluator in `python/bitlisp/` — own code, not a wrapper (it is the spec artifact), small and boring: cons cells, serialization, operator dispatch, cost accounting.
-- [x] Define operator set in `VM.md`: CLVM core **minus** BLS operators, **plus** `secp_verify` (BIP340, assertive semantics deferred to condition layer). Divergence table with rationale per row.
+- [x] Define operator set in `VM.md`: CLVM core **minus** BLS operators, **plus** `secp_verify` (BIP340, assertive semantics deferred to condition layer). Divergence table with rationale per row. Amended after the Phase 1 close: `sha256tree` adopted 2026-07-29 (decision by Evan, VM.md divergence D9).
 - [x] Inherit the CLVM cost table (`COSTS.md`); weight-mapping section stubbed for Phase 3 data.
 - [x] **Differential harness v1** (`tools/diff_clvm.py`): run every intersection program through bitlisp-python AND `clvm`/`chia_rs`; assert identical (result, cost) or identical error class.
 - [x] Import Chia's official CLVM test vectors for the intersection; generate randomized program corpus (Claude Code task: corpus generator with size/depth knobs).

python/bitlisp/costs.py

@@ -57,6 +57,17 @@
 # fixed by the operator's shape checks.
 SECP_VERIFY_COST = 1_300_000
 
+# sha256tree charges per visited node during its walk: the base cost
+# rides on the first node's charge, each visited pair charges the
+# pair cost, each visited atom charges the per-byte cost on its
+# length plus one for the leaf tag byte, and the 32-byte result
+# charges plain malloc. The constants are the consensus oracle's own
+# sha256tree, carried behind its release flag (a recorded
+# divergence: at flags 0 the oracle treats the opcode as unknown).
+SHA256TREE_BASE_COST = 270
+SHA256TREE_PAIR_COST = 460
+SHA256TREE_COST_PER_BYTE = 2
+
 GRS_BASE_COST = 117
 GRS_COST_PER_BYTE = 1
 SUBSTR_COST = 1

python/bitlisp/operators.py

@@ -11,9 +11,12 @@
 cost_exceeded, wrong_arg_count, arg_not_atom, arg_not_pair,
 arg_too_long, bad_index, index_out_of_range, shift_too_large,
 div_by_zero, and secp_verify_failed is reported. Every function below
-performs them in the consensus oracle's order, except op_secp_verify,
-which has no oracle and whose order is its own normative choice. The
-boundary cases are pinned by vectors.
+performs them in the consensus oracle's order, with two exceptions:
+op_secp_verify has no oracle and its order is its own normative
+choice, and op_sha256tree matches the same released wheel's operator
+behind its release flag, since at flags 0 the opcode is unknown to
+the oracle (a recorded divergence). The boundary cases are pinned by
+vectors.
 """
 
 import hashlib
@@ -552,6 +555,43 @@ def op_secp_verify(args, charge):
     return TRUE
 
 
+def op_sha256tree(args, charge):
+    # The arity check is the only check and precedes every charge:
+    # any node is a legal argument, hashing structure is the
+    # operator's purpose, so there is no arg_not_atom path. The walk
+    # charges each node as it reaches it, the base cost riding on
+    # the first node's charge, because evaluation builds shared
+    # structure cheaply (one cons of a node to itself doubles the
+    # reachable tree) and only walk-time charging keeps the hashing
+    # work bounded by the budget. Sharing is never deduplicated: a
+    # node the walk reaches twice is charged and hashed twice. The
+    # stack mirrors the machine's explicit-stack rule, so argument
+    # depth is not limited by the Python recursion limit.
+    _exactly(args, 1, "sha256tree")
+    pending = costs.SHA256TREE_BASE_COST
+    hashes = []
+    stack = [(False, args[0])]
+    while stack:
+        combine, node = stack.pop()
+        if combine:
+            first = hashes.pop()
+            rest = hashes.pop()
+            hashes.append(hashlib.sha256(b"\x02" + first + rest).digest())
+        elif is_pair(node):
+            charge(pending + costs.SHA256TREE_PAIR_COST)
+            pending = 0
+            stack.append((True, None))
+            stack.append((False, node[0]))
+            stack.append((False, node[1]))
+        else:
+            charge(pending + costs.SHA256TREE_COST_PER_BYTE * (len(node) + 1))
+            pending = 0
+            hashes.append(hashlib.sha256(b"\x01" + node).digest())
+    result = hashes[0]
+    _malloc(charge, result)
+    return result
+
+
 def op_raise(args, charge):
     raise BitLispError("user_raise", "clvm raise")
 
@@ -585,4 +625,5 @@ def op_raise(args, charge):
     b"\x20": op_not,
     b"\x21": op_any,
     b"\x22": op_all,
+    b"\x3f": op_sha256tree,
 }

python/tests/test_differential.py

@@ -19,9 +19,34 @@
 # instead of failing collection when only `dev` is installed.
 pytest.importorskip("chia_rs")
 import diff_clvm  # noqa: E402
+import diff_sha256tree  # noqa: E402
 from diff_clvm import Generator, run_bitlisp, run_rs  # noqa: E402
 
 
+def test_fixed_seed_sha256tree_agrees_with_flag_enabled_oracle():
+    # A fast fixed slice of tools/diff_sha256tree.py: trees through
+    # the operator versus the wheel's flag-enabled dispatch, and
+    # results versus its tree_hash puzzle-hash utility.
+    import chia_rs
+
+    flag = chia_rs.ENABLE_SHA256_TREE
+    rng = random.Random(7777)
+    gen = diff_sha256tree.TreeGenerator(rng, max_depth=4)
+    mismatches = []
+    for i in range(150):
+        tree = gen.tree(rng.randint(0, 4))
+        program = diff_sha256tree.serialize(
+            diff_sha256tree.lst(b"\x3f", (b"\x01", tree))
+        )
+        env = diff_sha256tree.serialize(b"")
+        bl = diff_sha256tree.run_bitlisp(program, env, 11_000_000_000)
+        rs = diff_sha256tree.run_rs(program, env, 11_000_000_000, flag)
+        digest = bytes(chia_rs.tree_hash(diff_sha256tree.serialize(tree))).hex()
+        if bl != rs or bl[0] != "ok" or bl[2] != "a0" + digest:
+            mismatches.append((i, program.hex(), bl, rs, digest))
+    assert not mismatches, mismatches[:3]
+
+
 def test_fixed_seed_corpus_agrees_with_consensus_oracle():
     rng = random.Random(7777)
     gen = Generator(rng, max_depth=5)

spec/COSTS.md

@@ -122,6 +122,18 @@ with Phase 3 measurements.
     neither the empty-signature nil nor a verification outcome is
     ever reported when the budget cannot cover the charge, pinned by
     boundary vectors.
+  - `sha256tree`: the arity check precedes every charge. The base
+    cost accrues without a budget check and rides on the first
+    visited node's charge (every tree has at least one node). Then
+    one checked charge per visited node: a pair charges the pair
+    cost when the walk reaches it, an atom charges the per-byte cost
+    on its length plus one, the leaf tag byte. The traversal order
+    among nodes is not consensus-visible: the only error a walk
+    charge can raise is `cost_exceeded` and the total is
+    order-independent, so an implementation may walk in any order
+    provided it charges each node as the walk reaches it (the
+    soundness rule in section 8). The result's malloc is one final
+    checked charge after the walk.
 - Evaluation cost does not include deserialization. A per-byte cost on
   the serialized program belongs to the weight mapping (section 9).
 
@@ -246,6 +258,7 @@ Worked example, pinned by vectors: `(any (q . 1) (q . 2))` costs
 | --- | --- |
 | `sha256` | `87 + 134 * n_args + 2 * total_arg_bytes + malloc(result)` |
 | `secp_verify` | `1300000`, flat, no malloc (PROVISIONAL) |
+| `sha256tree` | `270 + 460 * n_pairs + 2 * (n_atoms + total_atom_bytes) + malloc(result)`, counting every visited node: a node the walk reaches twice counts twice |
 
 The sha256 result atom is always exactly 32 bytes, so its malloc is a
 flat 320 charged after the argument loop. Per-byte terms count
@@ -262,13 +275,42 @@ recorded in VM.md section 8. The empty-signature branch charges the
 same flat cost in v0, with a cheaper price explicitly left as a
 Phase 3 question there.
 
+`sha256tree`'s per-byte term prices each visited atom's actual bytes
+plus one, the leaf tag byte, at `sha256`'s 2 per byte. The pair
+constant 460 covers a pair node's own SHA-256 invocation, whose input
+is always the tag byte and two 32-byte child hashes. The result atom
+is always exactly 32 bytes, so its malloc is a flat 320 charged after
+the walk. The constants are the consensus oracle's own, carried
+behind its release flag (divergence D9), and every visited node
+counts: the walk follows structure without deduplicating sharing, so
+a subtree reachable twice is charged twice.
+
+`sha256tree`'s per-node charging is load-bearing for more than the
+error class at the budget boundary. Evaluation builds shared
+structure cheaply: `(c 1 1)` doubles the environment reachable from
+its result for one 50-cost cons, and k nested applies of that shape,
+a few hundred cost units each, reach 2^k visited nodes for build
+cost linear in k. The sharing is built by evaluation, never spelled
+in the witness, so canonical serialization is no defense. An
+implementation must charge each node as the walk reaches it and stop
+at `cost_exceeded`, doing exactly the budget's worth of hashing.
+Hashing the whole tree before charging does unbounded work under a
+small budget, a validation denial of service, the same hazard class
+as the copy-on-slice note in section 5.
+
 Worked examples, pinned by vectors: `(sha256 (q . "ab") (q . "cd"))`
 costs `20 + 20 + 1 + 87 + 134 * 2 + 2 * 4 + 320 = 724`, and a
 `secp_verify` application on three quoted arguments costs
 `20 * 3 + 1 + 1300000 = 1300061` when it returns, the same total for
 a valid signature and for an empty one. The failing path charges
 identically before it raises `secp_verify_failed`.
 
+`sha256tree` worked examples, pinned by vectors: `(sha256tree (q))`
+hashes nil at `20 + 1 + 270 + 2 * 1 + 320 = 613`, and
+`(sha256tree (q "ab" "cd"))` walks two pairs and three atoms holding
+four bytes for
+`20 + 1 + 270 + 460 * 2 + 2 * (3 + 4) + 320 = 1545`.
+
 ## 9. Weight mapping
 
 TODO (Phase 3): mapping from VM cost units to Bitcoin transaction