Count the offsets that decide a result in code points

[azrollin]

Counts the windows, steps and cuts that decide a result in code points, the way the scanner counts them for ordinary text. Two remaining limits over surrogate input are listed under What remains different. Follows the list in #35 (review item 4). The rule set is untouched.

Cause

The scanner measures how far back a negation cue or a framing phrase may sit, how far a hit must read the same in the raw input, where the next search resumes, which character sits beside a keyword, the co-occurrence window, the 50 character match text and the context around a keyword in code points. An index in JavaScript counts UTF-16 units, and a character outside the Basic Multilingual Plane (an emoji, a mathematical letter, a regional indicator, a rare CJK character) takes two. Over such text a window was half as wide and a step could land inside a pair. Both directions followed: a text the scanner blocks was allowed here, a text it allows was blocked, and a reported match could end on half of a pair (a lone surrogate in matched_text).

Change

• src/negation.js: cpBack and cpFwd step by code point. The negation and framing windows, the raw-read span, the layered-decoding probe (64 code points), the neighbour step in samePlace and the cue start use them.
• src/engine.js: the resume position after a match, the character beside a keyword, the co-occurrence window and stride, the 50 character match text and the keyword context (excerpt) count code points. A rule searched around a rare token reads the whole document when it holds any surrogate code unit, paired or isolated (the span is a number of code points and its windows are ranges of units), as it already did when folding changed the length. The cut of an over-long folded view no longer splits a pair.
• src/preprocessor.js and the README: the matching entry is rewritten to what remains. The first commit is the README wording for the base64 padding entry that the review of Decode a base64 run only where the scanner decodes it #36 asked for.

Test, red on main and green on this head

tests/code_point_offsets.test.mjs with tests/fixtures/scan_corpus_code_point_offsets.json (76 rows: 60 decision rows from the scanner's own engine at 0.6.8, in families: negation cue then a wide character, a cue glued to its payload, a negated payload with a wide character, framing then a wide character, plain controls; and 16 separate rows that also check the reported match text) plus four generated rows for the anchored window.

node --test tests/code_point_offsets.test.mjs
main (538cef9):  pass 1, fail 6, skipped 1
this head:       pass 8, fail 0

Differential against the scanner

12,870 texts per seed: the scanner-recorded corpus and 3,000 generated rows. A generated row takes a source text of at most 3,000 code points and inserts a run of wide characters (1 to 121 long) at one of four places: the front, the middle, one third in, or the end. Each source text gets three such draws, made at random with replacement. Only new differences count: the row without the run already matched the scanner and the row with it does not.

seed	main, new differences	this head, new differences
11	10 (5 looser, 5 stricter)	4 (0 looser, 4 stricter)
12	4 (2 looser, 2 stricter)	2 (0 looser, 2 stricter)

matched_text on the seed 11 set, compared to the scanner: 1,814 rows differ on main, 789 of them with a lone surrogate; on this head 5 differ (a different rule id in the pair, not the cut) and none carries a lone surrogate.

Replays unchanged on this head: scan_corpus_068.json 1 differing row, the generated cue x payload set 16, the negation fuzz set 16, the unicode length set 0.

What remains different

• The 6 rows left in the table are not offsets. They have a mapped look-alike letter with an &, %, \ or a tag character within 64 characters after it, and the Worker's layered-decoding check answers "yes" for any such start where the scanner decodes and compares. The direction seen is stricter (block where the scanner gives allow_redacted). Tracked separately.
• Word boundaries next to a letter outside the ASCII range are the existing, separately listed difference.
• Isolated surrogate halves that become adjacent when an invisible character between them is removed: the Worker joins them into a pair and steps over them as one character, the scanner keeps two code points. An independent review of this head measured it on 8,476 channel-expanded rows as 57 stricter decisions, none looser, and 165 changed reports, 96 of them in the excerpt. Not fixed here.
• The anchored search: any surrogate code unit, including an isolated one, makes the Worker search the whole document, where the scanner keeps the anchor window. The same review measured additional findings with unchanged decisions on an anchored-rule family (189 rows, 54 of them with no astral character). Not fixed here.
• Indexes inside a view are still UTF-16 units and are only compared with each other.

This PR makes no claim that the hosted demo matches the scanner beyond the rows and checks listed here.

Gates on this head

Private checkout of the scanner at 0a3b4686. All exit 0 on head 4870dc0 (r1): disclosure gate; policy parity (14 cases, 0 mismatches); engine parity (137 cases, 0 verdict, 0 finding-set deltas); channel parity (2,488 cases, 0, 0); wide parity (1,555 pairs, 0, 0, no known entries); counterexample parity (0 deltas, 0 slower than the bound); parity_test.py (890 regexes, 0 misses; 13,024 benign checks, 0 disagreements; seed 3720106481, replay with PARITY_SEED); v-flag disclosure; metadata contract. node --test tests/*.test.mjs: 172 passed, 0 failed (164 on main, plus the 8 new). python3 -m pytest on the dotless-i tests: 62 passed.

r2 head a041cde: wording only. It narrows the README paragraph and the matching LIMITATIONS string in src/preprocessor.js (one string literal) to name the two surrogate limits; no other file differs from 4870dc0. On r2 the disclosure gate, the v-flag disclosure gate and the metadata contract exit 0 and node --test tests/*.test.mjs is 172 passed, 0 failed. The parity gates above were not re-run, since no code path changed.