Skip to content

VM: the bitwise and boolean operator families - #7

Merged
EvanWinget merged 7 commits into
mainfrom
bitwise-boolean-families
Jul 28, 2026
Merged

EvanWinget merged 7 commits into
mainfrom
bitwise-boolean-families

Conversation

@EvanWinget

Copy link
Copy Markdown
Owner

What changed

The bitwise family (ash, lsh, logand, logior, logxor, lognot) and the
boolean family (not, any, all) enter the spec, the reference
implementation, the vector corpus, and the diff harness. The two
families share one PR by decision (Evan, 2026-07-27): boolean is three
trivial flat-cost operators. After this PR, crypto is the only Phase 1
family left.

One new error class, shift_too_large, for a shift count whose
magnitude exceeds 65535. Shift count shape violations reuse bad_index
with substr's indices, matching the consensus oracle's identical
reporting of both.

Spec sections

VM.md section 4 (operator rows and the two prose blocks), section 5
(taxonomy rows for shift_too_large and the broadened bad_index, plus
the misleading-message caveat extending to shift counts). COSTS.md
section 1 (charge order bullets), new sections 6 and 7 (weight mapping
and condition costs renumbered to 8 and 9).

Read the commits in this order

  1. Spec: the bitwise and boolean families, probed against both oracles. The semantics and cost claims, all probe-established.
    The subtle ones: the shifts' per-byte term counts the result's
    magnitude bytes rather than its encoded bytes (disambiguated by a
    result of 128, one magnitude byte but two encoding bytes), lsh
    reads its value atom unsigned so it can change a value's sign and
    magnitude, and not costs a flat 200 with no per-argument term.
  2. Implement the bitwise and boolean families. Nine operators
    following the established charge-order style. The log fold reuses
    op_add's accrue-then-check loop shape. The shifts run every check
    before any charge.
  3. Pin the bitwise and boolean families: 98 vectors. Every
    expectation was generated against chia-rs 0.46.0 flags 0, with ok
    cases also cross-checked against clvm 0.9.15. Budget-boundary
    vectors pin check-versus-charge order at one-unit margins.
  4. Reach the bitwise and boolean families in the diff harness. New
    count distribution mirroring the substr index treatment, three new
    message mappings, and a seventh tolerated Python-oracle
    disagreement (its pair-in-count message is indistinguishable from
    pair-in-value, where consensus distinguishes them).

Verify independently

.venv/bin/pytest python/tests
.venv/bin/python tools/run_vectors.py
.venv/bin/python tools/diff_clvm.py --count 10000 --seed 20260727
.venv/bin/python tools/diff_clvm.py --count 10000 --seed 1123581321
ci/lint/lint.sh

Both 10k seeds are fresh to this PR and finish with zero failures,
with the new shift-count-conflation tolerance counter nonzero on
both. Any fresh seed should behave the same, and an unexplained
divergence fails the run loudly.

VM.md section 4 gains nine operator rows and two prose blocks. The
bitwise ops read atoms as signed two's complement with no operand
size limits. Shift counts follow the substr index shape (at most
four bytes, leading zeros legal within the cap, bad_index otherwise)
plus a magnitude cap of 65535 with a new shift_too_large error class
beyond it, in either direction. lsh reads the value's bytes as
unsigned, so a shift can change sign and value, pinned by the
(lsh -1 0) = 255 example. The boolean ops are the one non-tree
family accepting pair arguments, nil-only falsehood as with i.

The error taxonomy broadens bad_index to shift counts, records the
oracles' misleading no-leading-zeros message extending to shift
counts, and records the Python oracle's conflation of a pair count
with a pair value, which the diff harness will tolerate.

COSTS.md gains sections 6 and 7 (weight mapping and condition costs
renumber to 8 and 9). The shifts' per-byte term counts the result's
magnitude bytes, one less than the minimal encoding when the top
magnitude bit is set, the same rule as the multiplication
accumulator, disambiguated by the (ash 1 7) probe where the result
128 has one magnitude byte but two encoding bytes. The log ops
charge in the same loop shape as +, pinned at the exact boundary by
a budget sweep (pair in the second argument reported only from
budget 408 on the worked example). Every shift check precedes every
shift charge. not costs a flat 200 with no per-argument term.

All statements established by probes against chia-rs 0.46.0 flags 0
and cross-checked against clvm 0.9.15, to be pinned by vectors in
this PR.
Nine operators per the new VM.md section 4 rows: ash, lsh, logand,
logior, logxor, lognot, not, any, all. Cost constants and charge
interleavings per COSTS.md sections 1, 6, and 7.

The log fold reuses the accrue-then-check loop shape op_add
established. The shifts run every check before any charge, with the
value's atom check ahead of the count's shape and range checks, and
their per-byte term counts the result's magnitude bytes through the
same ceil(bit_length / 8) expression op_mul uses for its
accumulator, which Python's sign-agnostic int.bit_length makes
correct for negative results too. lsh reads its value atom unsigned
via int.from_bytes, so its shifted value is never negative. The
boolean ops accept any node and return only the shared constants.

shift_too_large joins the error taxonomy. The count shape shares
bad_index with substr's indices, matching the consensus oracle's
identical reporting of both.

Verified against chia-rs 0.46.0 flags 0 by an ad hoc differential
run before the harness commit in this PR reaches the families: 119
targeted cases plus full budget sweeps over 15 boundary-sensitive
programs, every budget value from 1 to each program's full cost,
zero mismatches in result, cost, and error class. Upstream clvm_rs
source was not consulted for this change, the formulas come from
probe fits recorded in the spec commit.
74 bitwise and 24 boolean cases covering the COSTS.md worked
examples, the fold identities, sign extension across widths, both
shift directions with floor behavior, lsh's unsigned reading and
sign flips, the magnitude-byte costing cases (result 128 in one
magnitude byte, result -256 in two), count encodings through the
four-byte cap and past it, the 65535 range cap in both directions,
the no-size-limit cases, every arity and pair-argument error, the
boolean truth tables with pair and 0x00 truthiness, flat large-atom
costs, and budget boundaries pinning each family's check-versus-
charge order at exact one-unit margins.

Every expectation was established against chia-rs 0.46.0 flags 0 by
the generating script, cost and result byte-exact for ok cases and
error class for failures, with ok cases additionally cross-checked
against clvm 0.9.15. Spec citations: VM.md sections 4 and 5,
COSTS.md sections 6 and 7.
The generator gains the nine opcodes. Shift programs mostly draw
their counts near the legal range through a dedicated count
distribution, mirroring the substr index treatment: uncorrelated
counts would almost never land under the 65535 cap, and the
distribution crosses the cap in both directions, pads non-negative
counts with zero bytes and negative counts with 0xff bytes, inside
and past the four-byte shape cap. The remaining shift programs go
through the generic path for pair values, pair counts, and wild
counts. Boolean ops need no special shape: pairs are legal
arguments.

Error maps gain Shift too large, the lsh-specific used-on-list
message, and the generic int32 fragment that covers the shift
counts, with substr's now-redundant specific spelling folded into
it. A seventh tolerated Python-oracle disagreement is recorded: its
generic requires-int-args message for a pair in a shift count
position, indistinguishable from a pair value, where consensus and
bitlisp report bad_index versus arg_not_atom. A stale inline count
of the tolerances (four, where the docstring already said six) now
defers to the module docstring.

Clean on 10000 programs at fresh seeds 20260727 and 1123581321,
zero failures, the new tolerance counter nonzero on both seeds.
@EvanWinget

Copy link
Copy Markdown
Owner Author

Code review

Found 1 issue:

  1. Off-by-one in the new spec text: the left-shift growth bound says "at most 8191 bytes over its input", but the true maximum is 8192. A count of 65535 bits adds up to 8192 magnitude bytes, and this PR's own vector ash_max_left_count in vectors/vm/bitwise.json shifts a 1-byte value left by 65535 and pins an 8193-byte result, a growth of 8192 bytes. The sentence backs the section 2 encodability argument, whose conclusion still holds, but the constant is falsified by the corpus in the same PR.

BitLisp/spec/VM.md

Lines 338 to 342 in c84dd93

A left shift can grow an atom by at most 8191 bytes over its input,
and the result charges plain malloc per byte, so the section 2
threshold for building an atom the wire format cannot encode applies
unchanged.

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

The bitwise prose claimed a left shift grows an atom by at most 8191
bytes. The true bound is 8192: a count of 65535 bits adds up to 8192
magnitude bytes, and the ash_max_left_count vector already pins
(ash (q . 1) (q . 65535)) growing one byte to 8193, sign byte
included. The encodability conclusion the sentence supports is
unchanged.

Also state the count redundancy rule in full: leading zeros on a
non-negative count and 0xff bytes on a negative one are both legal
within the four-byte cap, the section 1 reading. The harness already
generates the 0xff spellings, the vectors pin them in the companion
vector commit.

Found in review of PR 7.
The _shift comment claimed every check ran before the call, but the
count shape and range checks run inside _shift, before its single
charge. The lognot comment claimed the result is never longer than
the argument's minimal encoding, false for nil, whose complement -1
takes one byte, pinned by the lognot_nil_is_minus_one vector. The
_shift_count comment now states the full count redundancy rule, 0xff
padding on negatives included. No behavior changes.

Found in review of PR 7.
- all_full_cost_short_one pins op_all's cost_exceeded side at one
  unit under, completing the exact/one-under pair every other
  boundary in the corpus has.
- lsh_value_check_beats_charge and lsh_value_check_below_dispatch
  pin lsh's value atom check winning over cost_exceeded. op_lsh
  checks its value with its own inline code rather than op_ash's
  _int_arg path, so ash's boundary vectors never exercised it.
- lsh_count_minus_65536_too_large pins the negative direction of
  lsh's range cap, previously pinned only for ash.
- ash_count_ff_padded_negative_within_cap and
  ash_count_ff_padded_past_cap_bad_index pin the 0xff-padded
  negative count spellings the harness generates, legal within the
  four-byte cap and bad_index past it, with cost identical to the
  minimal spelling since count length never enters the cost.

Every expectation established against chia-rs 0.46.0 flags 0.
Spec citations: VM.md sections 4 and 5, COSTS.md sections 6 and 7.

Found in review of PR 7.
@EvanWinget

Copy link
Copy Markdown
Owner Author

Review disposition: the growth-bound issue is fixed in f2d5ce8. Five lower-confidence findings from the same review pass were also addressed: two inaccurate comments corrected in 3989b2a, and six vectors added in cd31a47 (the all short-one boundary, lsh's own value-check boundary pair, lsh's negative range cap, and the 0xff-padded count spellings, all established against the consensus oracle). One finding was declined: a lognot below-boundary partner at max_cost 20, which would pin only the dispatch-charge boundary already covered by dispatch.json and the logand and ash pairs.

@EvanWinget

Copy link
Copy Markdown
Owner Author

Upstream cross-check, performed after implementation as review evidence (reading policy guardrail: probes established the spec, source read only confirms). Against clvm_rs 0.18.0 in references/, every probe-derived claim matches the source: all ten cost constants, the shift per-byte term as value length plus result magnitude bytes (limbs_for_int is bits div_ceil 8, sign-agnostic), lsh's unsigned BigUint reading of the raw value atom, the check order (arity, value atom, count shape, count range, then cost), the inclusive 65535 cap in both directions, the count reader accepting redundant encodings within four bytes while its error text claims otherwise (the misleading message is hardcoded in i32_atom regardless of input, confirming the VM.md section 5 caveat at the source level), the log fold's per-argument check points and fold identities, lognot's formula, not's flat 200 with no per-argument term, and any/all charging per argument with no node checks. Two structural differences with no observable effect: clvm_rs folds log arguments through separate positive and negative accumulators combined at the end, and clvm_rs ops return one machine-checked total where bitlisp charges stepwise, equivalent at every budget the sweeps covered. Zero meaningful differences found.

@EvanWinget
EvanWinget merged commit 977de08 into main Jul 28, 2026
2 checks passed
@EvanWinget
EvanWinget deleted the bitwise-boolean-families branch July 28, 2026 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant