Repository navigation
Port the 0.6.8 engine reads: folded regex view, negation walk, record hook - #35
Merged
Merged
Conversation
The wide differential stops on the 0.6.8 recompile at one case-channel pair where the scanner blocks and the Worker allows. That difference is tracked separately. KNOWN_DELTAS names that one pair with the exact verdicts and the ids only the scanner reports. Any other difference still fails the gate, and an entry that no longer shows up fails it too, so the change that closes the gap has to delete the entry.
…cessor record hook Three scanner engine changes that the 0.6.8 recompile does not carry on its own. Folded regex view. The keyword lane reads a view with invisible characters removed, NFKC applied and look-alike letters mapped to ASCII. The regex lane read only the raw, shadow and normalized views, so a rule that lives on its regex could be missed after one invisible character, a Cyrillic letter or a full-width letter. The engine now gives the regex lane that folded view as a further subject, capped the way the scanner caps it. Negation and defensive framing read by clause. The old check looked at a fixed window of the substring before a hit. The new one walks the raw text the way the scanner does (src/negation.js), so a negation or a framing phrase governs only the clause it sits in, and a hit later in the text is judged on its own. The walk stops vouching for text after a step the Worker's own decoders also take (entities, percent and hex escapes, shadow characters, base64), which can only make it stricter than the scanner, never looser. Preprocessor record hook. normalize() and decodeBase64Segments() can report what they decoded and where, and normalizeWithLength() returns the folded length too. The wide differential's known entry is removed here: the pair it allowed now agrees, and the gate fails if a stale entry is left behind. One test per port, each red on the previous engine and green here: folded_regex_view, negation_governs_clause, defensive_framing_clause, preprocessor_record_hook.
tools/capture_scan_corpus.py records every default-engine scan() call that a set of scanner test files makes (1,307 distinct text and channel pairs for the folded view, negation and framing tests), with the scanner's decision and finding ids. tools/gen_fuzz_corpus.py builds a second corpus around a cue, a payload and a change that must not let the cue govern; 685 of its rows are kept. tests/scan_corpus_parity.test.mjs replays both: each row must reach the scanner's decision and finding ids. Rows marked residual are the ones where the Worker is stricter than the scanner (it does not vouch for text after a decoding step); the test requires them to be at least as strict, never looser, and caps how many there may be.
…ed length A neighbour test reads a character, and a character outside the BMP is two UTF-16 units. The combining-mark test in front of a framing cue, the apostrophe test in a quote search, the space test in the raw-input check and the word-fragment test at the edge of the negation window each read one unit, so they saw half a surrogate pair. They now read the whole code point, as the scanner does. A supplementary combining mark in front of a framing cue no longer lets the cue downgrade a hit the scanner keeps at full severity. Step 3.5 runs only on short inputs, and the scanner measures the folded text, after the normalizer has removed padding. The Worker measured the raw text, so an encoded payload padded past the limit skipped the pass. It now uses the folded length, counted in code points; the enrichment gate counts the same way. The character cache that decides whether a character folds to an escape start clears at 4,096 entries rather than 4,097. parity_test.py now takes the scanner from SUNGLASSES_SRC like the other gates, asserts it imported from there, and prints the path. tests/unicode_neighbour_and_length_gate.test.mjs replays 29 scanner-backed rows, all red on the previous head.
The framing check reads one character to the left of its window so that a phrase starting on the window's first character is judged against the text in front of it. It took one UTF-16 unit, so a character outside the BMP there lost its high half. It now takes the whole code point. The record hook is normalizeWithLength() and decodeBase64Segments() taking an optional record of replaced segments; normalize() itself does not report, and the record holds replaced segments, not positions. The earlier commit message in this branch says otherwise. tests/unicode_neighbour_and_length_gate.test.mjs gains 6 scanner-backed rows for the window edge; they fail without this change.
This was referenced Oct 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports three scanner engine changes from 0.6.8 into the Worker engine, on top of the recompile in #34 (now merged). The rule set is untouched; this changes how the engine reads text.
Commits, in order:
Allow the one known wide-parity difference, and fail when it goes away. Recompile from v0.6.8 (1569 of 1569 rules) #34 merged withwide-parityred on one pair (test_supply_chain_skill_gym.py#41, channelapi_response: scanner blocks, Worker allows, only GLS-MECH-011 on the scanner side). This commit lists that one pair inKNOWN_DELTASwith its exact verdicts and ids, so main's check is green before the port lands. Any other difference still fails, and so does an entry that no longer shows up.Port the 0.6.8 engine reads: folded regex view, negation walk, preprocessor record hook. Closes the pair above and removes its entry.Replay recorded scanner scans through the Worker engine. The corpus replay and the two scripts that build it.Read whole code points beside a mark or a quote, and measure the folded length, and 5. a follow-up that takes the whole code point at the left edge of the framing window. Both are fixes found on review, below.Merge text: the record hook is
normalizeWithLength()anddecodeBase64Segments()taking an optional record of replaced segments. Commit 2's message saysnormalize()reports too; it does not, andnormalize()is unchanged for callers.The three ports
normalizeWithLength()returns the folded length too.src/negation.jswalks the raw text the way the scanner does, so a negation or a framing phrase governs only its own clause and a later hit is judged on its own. The regex scan also resumes after a hit that was judged, so a later live hit is not lost behind an earlier covered one.normalizeWithLength()anddecodeBase64Segments()take an optional record of what was decoded; the walk uses it.normalize()is unchanged for callers.The step 3.5 length gate now measures the folded text, in code points, as the scanner does. The first version of this PR kept the raw-length test; a padded, encoded payload showed the two differ (109 rows in the reviewer's set), and the enrichment gate counts the same way.
Two fixes after review
tests/unicode_neighbour_and_length_gate.test.mjsreplays 35 scanner-backed rows (15 mark rows, 6 window-edge rows, 14 padded rows).One test per port, red on the previous engine, green on this head
Same command on both (
node --test tests/<file>, previous engine swapped in throughSG_WORKER_ENGINE/SG_WORKER_PREPROCESSOR):Differential against the scanner (0.6.8, commit 0a3b4686)
Decision and sorted finding ids, one scan per row, stock rules:
tests/fixtures/scan_corpus_068.json)tools/gen_fuzz_corpus.py)Of the 6,711, 4,942 were the Worker being less strict than the scanner. Now none are: all 17 remaining rows have the Worker stricter. The 685-row subset kept in
tests/fixtures/scan_corpus_negation_fuzz.jsonis every row the scanner downgrades, the 16 remaining rows, and 600 sampled.What remains different
decodeBase64Segments), outside the three ports. Tracked separately.samePlace, the layered-decode lookahead in the walk, the negation window incheckNegation,resumeAfterandwordBoundedin the engine. No stock-rule miss was found from any of them in the review sets.residualin the fixtures, andscan_corpus_parity.test.mjsrequires them to be at least as strict as the scanner and caps their number. This is tracked separately; this PR makes no claim that the hosted demo matches the scanner beyond the rows and checks listed here.Gates on this head
Private checkout of the scanner at 0a3b4686. All exit 0: disclosure gate; policy parity (14 cases, 0 mismatches); engine parity (137 cases, 0 verdict, 0 finding-set deltas); channel parity (2,488 cases, 0, 0); wide parity (1,555 pairs, 0, 0, no known entries left); counterexample parity (0 deltas, 0 slower than the bound);
parity_test.py(883 regexes this run, 0 misses; 13,024 benign checks, 0 disagreements; seed 2460659979, replay withPARITY_SEED; it now takes the scanner fromSUNGLASSES_SRClike the other gates and prints where it imported from); v-flag disclosure; metadata contract.node --test tests/*.test.mjs: 156 passed, 0 failed (78 on the #34 head).python3 -m pyteston the dotless-i tests: 62 passed. The newline-run and whitespace-run scan-time tests from #34 still pass.Not done here: no deploy. After merge a deploy is a separate step.