JSON deserializer can't read escaped surrogate pairs, including Serde's own output for emoji #411

Serde's JSON serializer writes characters outside the Basic Multilingual Plane, like emoji, as escaped surrogate pairs, but the deserializer can't read an escaped surrogate pair. So Serde can't read back its own output for any string with an emoji in it. Unescaped emoji read fine.

Repro (Serde 0.18.2)

using System;
using Serde;
using Serde.Json;

string written = JsonSerializer.Serialize(new Note { Text = "🤖" });
Console.WriteLine(written);                                                // {"text":"🤖"}
Console.WriteLine(JsonSerializer.Deserialize<Note>("{\"text\":\"🤖\"}").Text); // 🤖
Console.WriteLine(JsonSerializer.Deserialize<Note>(written).Text);         // throws

[GenerateSerde]
partial record Note
{
    public required string Text { get; init; }
}

System.InvalidOperationException: Cannot read incomplete UTF-16 JSON text as string with missing low surrogate.
   at Serde.Json.ThrowHelper.ThrowInvalidOperationException_ReadIncompleteUTF16() in src/serde/json/reader/ThrowHelper.cs:line 608
   at Serde.Json.JsonReaderHelper.DecodeUnicodeEscape(...) in src/serde/json/reader/JsonReaderHelper.Unescaping.cs:line 664
   at Serde.IO.MemoryReader.LexEscape(Boolean skipOnly, ScratchBuffer scratch) in src/serde/json/newreader/ArrayReader.cs:line 169
   ...

The same happens with JsonValue.

Likely cause

DecodeUnicodeEscape comes from System.Text.Json, where idx starts on the u of \u. MemoryReader.LexEscape calls it with idx on the first hex digit, and the first half was adapted to match (source.Slice(idx, 4)), so after idx += 4, idx is on the byte after the four digits, the next \. The surrogate-pair branch still uses System.Text.Json's offsets, which assume idx is on the last hex digit:

if (source.Length < idx + 7 || source[idx + 1] != '\\' || source[idx + 2] != 'u')
...
source.Slice(idx + 3, 4)
...
idx += 6;

source[idx + 1] is the u, not the \, so every escaped surrogate pair throws. With idx past the first escape, the branch should check source[idx] and source[idx + 1], read source.Slice(idx + 2, 4), and advance by 6.

Found in forj, which worked around it by not round-tripping records through JsonValue.