Skip to content

Port the 0.6.8 engine reads: folded regex view, negation walk, record hook - #35

Merged
azrollin merged 5 commits into
mainfrom
t10/port-068-engine
Oct 13, 2026
Merged

azrollin merged 5 commits into
mainfrom
t10/port-068-engine

Conversation

@azrollin

@azrollin azrollin commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

Ports three scanner engine changes from 0.6.8 into the Worker engine, on top of the recompile in #34 (now merged). The rule set is untouched; this changes how the engine reads text.

Commits, in order:

  1. Allow the one known wide-parity difference, and fail when it goes away. Recompile from v0.6.8 (1569 of 1569 rules) #34 merged with wide-parity red on one pair (test_supply_chain_skill_gym.py#41, channel api_response: scanner blocks, Worker allows, only GLS-MECH-011 on the scanner side). This commit lists that one pair in KNOWN_DELTAS with its exact verdicts and ids, so main's check is green before the port lands. Any other difference still fails, and so does an entry that no longer shows up.
  2. Port the 0.6.8 engine reads: folded regex view, negation walk, preprocessor record hook. Closes the pair above and removes its entry.
  3. Replay recorded scanner scans through the Worker engine. The corpus replay and the two scripts that build it.
  4. Read whole code points beside a mark or a quote, and measure the folded length, and 5. a follow-up that takes the whole code point at the left edge of the framing window. Both are fixes found on review, below.

Merge text: the record hook is normalizeWithLength() and decodeBase64Segments() taking an optional record of replaced segments. Commit 2's message says normalize() reports too; it does not, and normalize() is unchanged for callers.

The three ports

  • Folded regex view (scanner #342). The keyword scan reads a view with invisible characters removed, NFKC applied and look-alike letters mapped to ASCII; the regex scan did not. The engine now gives the regex scan that view as a further subject, capped the way the scanner caps it (the cap counts UTF-16 units here where the scanner counts code points). normalizeWithLength() returns the folded length too.
  • Negation and defensive framing by clause (scanner #345, #347, #361). The old check read a fixed window before a hit. src/negation.js walks the raw text the way the scanner does, so a negation or a framing phrase governs only its own clause and a later hit is judged on its own. The regex scan also resumes after a hit that was judged, so a later live hit is not lost behind an earlier covered one.
  • Preprocessor record hook. normalizeWithLength() and decodeBase64Segments() take an optional record of what was decoded; the walk uses it. normalize() is unchanged for callers.

The step 3.5 length gate now measures the folded text, in code points, as the scanner does. The first version of this PR kept the raw-length test; a padded, encoded payload showed the two differ (109 rows in the reviewer's set), and the enrichment gate counts the same way.

Two fixes after review

  • A neighbour test reads a character, and a character outside the BMP is two UTF-16 units. The combining-mark test before a framing cue (15 rows), and three similar single-unit reads (apostrophe between letters, the space test in the raw-input check, the word-fragment test at the negation window edge), now read the whole code point. The one character of left context that the framing check reads at the edge of its window is now a whole code point too (2 rows in the review set, plus 4 kept as controls).
  • The corroboration length gate, above.

tests/unicode_neighbour_and_length_gate.test.mjs replays 35 scanner-backed rows (15 mark rows, 6 window-edge rows, 14 padded rows).

One test per port, red on the previous engine, green on this head

Same command on both (node --test tests/<file>, previous engine swapped in through SG_WORKER_ENGINE / SG_WORKER_PREPROCESSOR):

tests/folded_regex_view.test.mjs            #34 head: pass 1, fail 38     this head: pass 39, fail 0
tests/negation_governs_clause.test.mjs      #34 head: pass 3, fail 14     this head: pass 17, fail 0   (3 controls pass on both)
tests/defensive_framing_clause.test.mjs     #34 head: pass 2, fail 6      this head: pass 8, fail 0
tests/preprocessor_record_hook.test.mjs     #34 head: fail 6 (hook missing) this head: pass 6, fail 0
tests/scan_corpus_parity.test.mjs           #34 head: pass 2, fail 2      this head: pass 4, fail 0
tests/unicode_neighbour_and_length_gate.test.mjs  first head of this PR: pass 0, fail 4  this head: pass 4, fail 0

Differential against the scanner (0.6.8, commit 0a3b4686)

Decision and sorted finding ids, one scan per row, stock rules:

corpus rows differing on #34 head differing now
default-engine scans recorded from the scanner's own folded-view, negation and framing tests (tests/fixtures/scan_corpus_068.json) 1,307 681 1
generated: cue x payload x change x context (tools/gen_fuzz_corpus.py) 12,614 6,711 16

Of the 6,711, 4,942 were the Worker being less strict than the scanner. Now none are: all 17 remaining rows have the Worker stricter. The 685-row subset kept in tests/fixtures/scan_corpus_negation_fuzz.json is every row the scanner downgrades, the 16 remaining rows, and 600 sampled.

What remains different

  • Inherited, not fixed here: one pair of GLS-PI-007 rows (one padded, one not) where the scanner blocks and the Worker allows. The same pair differs on main before this PR. The cause is a difference in how the Worker's preprocessor decodes base64 (decodeBase64Segments), outside the three ports. Tracked separately.
  • Offsets and windows count UTF-16 units in the Worker where the scanner counts code points. The neighbour reads above take the whole code point; helpers that compare positions or size windows still use units: samePlace, the layered-decode lookahead in the walk, the negation window in checkNegation, resumeAfter and wordBounded in the engine. No stock-rule miss was found from any of them in the review sets.
  • The walk stops vouching after a decoding step. The Worker does not vouch for a cue that comes after text it decoded (an entity, a percent or hex escape, shadow characters, a base64 run). There the Worker reports the hit at its plain severity where the scanner downgrades it, so it blocks more, never less. These are the 17 rows above, they are flagged residual in the fixtures, and scan_corpus_parity.test.mjs requires them to be at least as strict as the scanner and caps their number. This is tracked separately; this PR makes no claim that the hosted demo matches the scanner beyond the rows and checks listed here.

Gates on this head

Private checkout of the scanner at 0a3b4686. All exit 0: disclosure gate; policy parity (14 cases, 0 mismatches); engine parity (137 cases, 0 verdict, 0 finding-set deltas); channel parity (2,488 cases, 0, 0); wide parity (1,555 pairs, 0, 0, no known entries left); counterexample parity (0 deltas, 0 slower than the bound); parity_test.py (883 regexes this run, 0 misses; 13,024 benign checks, 0 disagreements; seed 2460659979, replay with PARITY_SEED; it now takes the scanner from SUNGLASSES_SRC like the other gates and prints where it imported from); v-flag disclosure; metadata contract. node --test tests/*.test.mjs: 156 passed, 0 failed (78 on the #34 head). python3 -m pytest on the dotless-i tests: 62 passed. The newline-run and whitespace-run scan-time tests from #34 still pass.

Not done here: no deploy. After merge a deploy is a separate step.

The wide differential stops on the 0.6.8 recompile at one case-channel pair
where the scanner blocks and the Worker allows. That difference is tracked
separately.

KNOWN_DELTAS names that one pair with the exact verdicts and the ids only the
scanner reports. Any other difference still fails the gate, and an entry that no
longer shows up fails it too, so the change that closes the gap has to delete
the entry.
…cessor record hook

Three scanner engine changes that the 0.6.8 recompile does not carry on its own.

Folded regex view. The keyword lane reads a view with invisible characters
removed, NFKC applied and look-alike letters mapped to ASCII. The regex lane read
only the raw, shadow and normalized views, so a rule that lives on its regex could
be missed after one invisible character, a Cyrillic letter or a full-width letter.
The engine now gives the regex lane that folded view as a further subject, capped
the way the scanner caps it.

Negation and defensive framing read by clause. The old check looked at a fixed
window of the substring before a hit. The new one walks the raw text the way the
scanner does (src/negation.js), so a negation or a framing phrase governs only the
clause it sits in, and a hit later in the text is judged on its own. The walk
stops vouching for text after a step the Worker's own decoders also take
(entities, percent and hex escapes, shadow characters, base64), which can only
make it stricter than the scanner, never looser.

Preprocessor record hook. normalize() and decodeBase64Segments() can report what
they decoded and where, and normalizeWithLength() returns the folded length too.

The wide differential's known entry is removed here: the pair it allowed now
agrees, and the gate fails if a stale entry is left behind.

One test per port, each red on the previous engine and green here:
folded_regex_view, negation_governs_clause, defensive_framing_clause,
preprocessor_record_hook.
tools/capture_scan_corpus.py records every default-engine scan() call that a set
of scanner test files makes (1,307 distinct text and channel pairs for the folded
view, negation and framing tests), with the scanner's decision and finding ids.
tools/gen_fuzz_corpus.py builds a second corpus around a cue, a payload and a
change that must not let the cue govern; 685 of its rows are kept.

tests/scan_corpus_parity.test.mjs replays both: each row must reach the scanner's
decision and finding ids. Rows marked residual are the ones where the Worker is
stricter than the scanner (it does not vouch for text after a decoding step); the
test requires them to be at least as strict, never looser, and caps how many
there may be.
…ed length

A neighbour test reads a character, and a character outside the BMP is two UTF-16
units. The combining-mark test in front of a framing cue, the apostrophe test in
a quote search, the space test in the raw-input check and the word-fragment test
at the edge of the negation window each read one unit, so they saw half a
surrogate pair. They now read the whole code point, as the scanner does. A
supplementary combining mark in front of a framing cue no longer lets the cue
downgrade a hit the scanner keeps at full severity.

Step 3.5 runs only on short inputs, and the scanner measures the folded text,
after the normalizer has removed padding. The Worker measured the raw text, so an
encoded payload padded past the limit skipped the pass. It now uses the folded
length, counted in code points; the enrichment gate counts the same way.

The character cache that decides whether a character folds to an escape start
clears at 4,096 entries rather than 4,097.

parity_test.py now takes the scanner from SUNGLASSES_SRC like the other gates,
asserts it imported from there, and prints the path.

tests/unicode_neighbour_and_length_gate.test.mjs replays 29 scanner-backed rows,
all red on the previous head.
The framing check reads one character to the left of its window so that a phrase
starting on the window's first character is judged against the text in front of
it. It took one UTF-16 unit, so a character outside the BMP there lost its high
half. It now takes the whole code point.

The record hook is normalizeWithLength() and decodeBase64Segments() taking an
optional record of replaced segments; normalize() itself does not report, and the
record holds replaced segments, not positions. The earlier commit message in this
branch says otherwise.

tests/unicode_neighbour_and_length_gate.test.mjs gains 6 scanner-backed rows for
the window edge; they fail without this change.
@azrollin
azrollin merged commit 8f5b870 into main Oct 13, 2026
2 checks passed
@azrollin
azrollin deleted the t10/port-068-engine branch October 13, 2026 19:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant