Keep an encoded replacement character when invalid bytes are dropped

[azrollin]

Keeps a genuine replacement character when a base64 segment also holds invalid bytes, as the scanner does. Follows the review of #36 (the 256 rows it left open). The rule set is untouched. Stacked on #37; the diff here is this change only.

Cause

The scanner decodes a base64 run with errors="ignore": every invalid byte is dropped and a replacement character that was encoded (the bytes EF BF BD) stays where it was written. The Worker tried a strict decode first, which is right when every byte is valid. When any byte was invalid it decoded with replacement and then deleted every U+FFFD, the ones the decoder had written and the encoded one alike. A decoder that replaces cannot tell the two apart. Deleting the encoded one joined the text on either side of it, so a phrase that the scanner reads as two words was one word here and the rule that needs the boundary did not fire. Example: "ignore all previous instructions", U+FFFD, "x", then a stray byte. The scanner blocks it, this runtime allowed it.

Change

• src/preprocessor.js: when the strict decode fails, the bytes are walked against the well-formed UTF-8 table (lead byte, range of the second byte, continuation count) and whatever is not well-formed is skipped. A valid sequence, including EF BF BD, is decoded where it stands. The strict decode is still tried first.
• The README entry and the matching LIMITATIONS string no longer list this as open. The other difference in that entry (the printable table) is unchanged.

Test, red on the base and green on this head

tests/invalid_bytes_beside_replacement_char.test.mjs with tests/fixtures/scan_corpus_invalid_bytes.json: 256 rows (32 texts across 8 channels) that the scanner blocks and the base allowed, with decision and finding ids from the scanner's own engine at 0.6.8, plus 7 direct decode cases.

node --test tests/invalid_bytes_beside_replacement_char.test.mjs
base (a041cde):  pass 2, fail 6
this head:       pass 8, fail 0

Differential

• The new decode against the scanner's bytes.decode("utf-8", "ignore") on 60,001 byte strings (mostly lead, continuation and boundary bytes, plus a case with an encoded replacement character): 0 differences.
• The 4,472 independent rows of the review (344 inputs across the 13 channels), base against this head: this head changes exactly the 256 rows, lowers none, and none is weaker than the scanner now (256 before). The 273 rows that still differ are stricter than the scanner and are the same on the base.

What remains different

• str.isprintable() is read from this runtime's Unicode tables, so a character assigned in one and not the other can decide a segment differently. Not changed here.
• The layered-decoding rows are tracked separately.

This PR makes no claim that the hosted demo matches the scanner beyond the rows and checks listed here.

Gates on this head

Private checkout of the scanner at 0a3b4686. Every gate exited 0 on the code of this change (the last commit then changed only the README paragraph and the matching LIMITATIONS string; the disclosure gate, v-flag disclosure, metadata contract, engine parity and node --test were run again on head 1521b4f): disclosure gate; policy parity (14 cases, 0 mismatches); engine parity (137 cases, 0 verdict, 0 finding-set deltas); channel parity (2,488 cases, 0, 0); wide parity (1,555 pairs, 0, 0, no known entries); counterexample parity (0 deltas, 0 slower than the bound); parity_test.py (886 regexes, 0 misses; 13,024 benign checks, 0 disagreements; seed 3010664057, replay with PARITY_SEED); v-flag disclosure; metadata contract. node --test tests/*.test.mjs: 180 passed, 0 failed (172 on #37, plus the 8 new). python3 -m pytest on the dotless-i tests: 62 passed.