fix: Text indices are character-based; add Doc(offset_kind=...)

[xrl]

Summary

• Add Doc(offset_kind="utf8" | "utf16") (default "utf8", the yrs default) controlling how yrs counts text positions internally
• Convert Python character (code point) indices to the doc's offset units in all Text mutation methods — insert, insert_embed, __setitem__, __delitem__, format, __iadd__
• len(text) now returns the Python character count
• Export get_utf8_index / get_utf16_index conversion helpers
• 30+ new Unicode tests (emoji, CJK, combining marks, ZWJ, surrogate pairs), parametrized over both offset kinds

The bug

pycrdt passed Python character indices straight to yrs, which interprets them in the doc's offset units (UTF-8 bytes by default). Any index past a multi-byte character landed in the wrong place — or mid-character:

doc = Doc()
doc['t'] = t = Text()
t += 'A📊B'
t.insert(2, 'X')   # before: 'A📊BX' (or a yrs panic) — after: 'A📊XB'

On main, the new test cases produce wrong results and pyo3_runtime.PanicException panics from yrs (see @dlqqq's run earlier in this thread). With the conversion in place, all operations are character-based in both offset kinds.

What offset_kind does (and doesn't do)

Earlier revisions of this description claimed UTF-16 was required for yjs compatibility — @jbdyn was right to push on that, and it's now corrected. The update wire format is identical for both kinds (struct clocks are UTF-16 code units either way); the tests now sync docs across mismatched kinds to pin that down, and I've verified content round-trips against yjs 13.6.31 in both modes. offset_kind controls how indices passed to yrs APIs are interpreted. "utf16" matches JS yjs index semantics, which matters when raw yrs offsets cross runtimes — sticky indices (#382) and event delta units (discussed above).

Heads-up for release planning

Making indices character-based is a behavior change for callers that pre-converted to byte offsets as a #308 workaround — jupyter_ydoc's YUnicode.set() (jupyter-server/jupyter_ydoc#370) is a released example and would corrupt against this change. Since jupyter_ydoc pins pycrdt >=0.14,<0.15, shipping this as 0.15.0 with a coordinated jupyter_ydoc update (drop the byte-diff workaround when pycrdt >= 0.15) avoids any window where both fixes are active. Happy to open that jupyter_ydoc PR.

Index semantics

Indices follow Python conventions in both offset kinds: out-of-range slice bounds clamp (del t[1:100] ≡ del t[1:]), negative indices count from the end. Note negative int keys previously raised OverflowError on main; they now resolve Python-style, matching __getitem__. If you'd rather keep rejecting them I can do that instead — the important property is that both offset kinds behave identically, which is now tested.

Follow-ups already split out: #381 (XmlText, stacked on this), #382 (StickyIndex), TextEvent.delta units (flagged in-thread for discussion).

Fixes #308
Related: jupyter-ai-contrib/jupyter-server-documents#197

[krassowski]

Thanks for the PR. I think this should include comprehensive tests.

See jupyter-server/jupyter_ydoc#370 for some prior art (using a different implementation, but you could also pull test cases out of there, though they will need adopting)

[xrl]

@krassowski please take a look now 👌

[davidbrochart]

@xrl It looks like _utf16_to_char is not tested.

[xrl]

@davidbrochart take a look now, thanks!

[davidbrochart]

Hmm I'm a bit confused that _utf16_to_char is actually not used. Should we remove it and the corresponding tests?

[davidbrochart]

@jbdyn Since you looked at that in #129, what do you think about this PR?

[jbdyn]

@davidbrochart I am currently looking into this, but I need a bit of time.

[jbdyn]

The __iadd__ of Text does not behave:

from pycrdt import Doc, Text

d = Doc()
t = d["t"] = Text()
t += "A📊B"
t += "X"
print(str(t))

which prints

A📊XB  # contrary to expected 'A📊BX'

Internally, Text.integrated.insert(txn, index, chars) needs the UTF-16 index, but Text.__len__ returns the Python character counts.

[xrl]

Further review of the surface area turned up three more spots touched by the OffsetKind::Utf16 change. Rather than balloon this PR, I've split them out:

• fix: use UTF-16 offsets for XmlText operations #381 — XmlText has the same bug as Text (unconverted offsets in insert/insert_embed/format/__delitem__/__iadd__, plus __len__ returning UTF-16 count). Opened as a draft PR stacked on this one; will mark ready once this lands.
• StickyIndex: UTF-16 offsets not converted for Text sequences #382 — StickyIndex doesn't convert in either direction for Text/XmlText. Filed as an issue rather than a PR because the fix has design questions around serialized indices and their source sequence.
• TextEvent.delta / XmlEvent.delta — retain/delete counts bubble out of yrs as UTF-16 code units. This might actually be the right contract for yjs wire compat — flagging here for discussion rather than changing unilaterally. Let me know if you'd prefer a separate issue for it.

[jbdyn]

I cannot reproduce the garbling example

doc['source'] += '# Analysis 📊\n'           # 15 chars, but yrs sees 18 bytes
doc['source'].insert(len(text), 'model()\n')  # Python says 15, yrs reads 15
                                               # → offset 15 in bytes lands INSIDE 📊

given in the PR description on the main branch.

When running

from pycrdt import Doc, Text

d = Doc()
t = d["t"] = Text()
text = "# Analysis 📊\n"
t += text
t.insert(len(text), 'model()\n')
print(str(t))

I get

# Analysis 📊\nmodel()\n

and not # Analysis 📊\nmod📊el()\n.

Is yjs still crashing then?

[dlqqq]

@jbdyn I wasn't able to reproduce that specific error either, but I am definitely able to reproduce other issues with pycrdt's existing handling of multi-byte characters using the test cases added in this PR.

# fetch PR
git remote add xrl git@github.com:xrl/pycrdt.git
git fetch xrl
git switch 308-fix-utf16-offset-encoding

# check out new test suite on main branch
git checkout main
git checkout 308-fix-utf16-offset-encoding -- tests/test_text.py

# activate env and run new tests against main
uv venv # (if needed)
source .venv/bin/activate
uv pip install -e .
pytest tests/test_text.py

The test cases fail on the main branch with some runtime panics (click to show):

Details

----------------------------------------------------------------------------------------------------------------------------- Captured stderr call -----------------------------------------------------------------------------------------------------------------------------

thread '<unnamed>' (75872785) panicked at /Users/dlq/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/yrs-0.25.0/src/types/text.rs:845:9:
Couldn't remove 2 elements from an array. Only 0 of them were successfully removed.
=========================================================================================================================== short test summary info ============================================================================================================================
FAILED tests/test_text.py::test_unicode_emoji_insert - assert 6 == 3
FAILED tests/test_text.py::test_unicode_emoji_len - assert 6 == 3
FAILED tests/test_text.py::test_unicode_emoji_delete - pyo3_runtime.PanicException: Couldn't remove 1 elements from an array. Only 0 of them were successfully removed.
FAILED tests/test_text.py::test_unicode_emoji_delete_emoji - AssertionError: Got 'A'
FAILED tests/test_text.py::test_unicode_emoji_slice_delete - AssertionError: Got 'A'
FAILED tests/test_text.py::test_unicode_emoji_setitem - pyo3_runtime.PanicException: Couldn't remove 1 elements from an array. Only 0 of them were successfully removed.
FAILED tests/test_text.py::test_unicode_emoji_slice_setitem - AssertionError: Got 'AXYZ'
FAILED tests/test_text.py::test_unicode_cjk - assert 7 == 3
FAILED tests/test_text.py::test_unicode_mixed_scripts - AssertionError: Got 'Hello 世界 м!ир 📊'
FAILED tests/test_text.py::test_unicode_supplementary_plane - assert 11 == 5
FAILED tests/test_text.py::test_unicode_granular_diff[emoji_swap] - pyo3_runtime.PanicException: Couldn't remove 1 elements from an array. Only 0 of them were successfully removed.
FAILED tests/test_text.py::test_unicode_granular_diff[text_change_emoji_stay] - AssertionError: Got 'Here is a star: ⭐ and a rocketar: 🚀', expected 'Here is a star: ⭐ and a rocket: 🚀'
FAILED tests/test_text.py::test_unicode_granular_diff[combining_marks] - pyo3_runtime.PanicException: Couldn't remove 5 elements from an array. Only 0 of them were successfully removed.
FAILED tests/test_text.py::test_unicode_granular_diff[flags] - AssertionError: Got 'Flags: 🇨🇦🇬🇧🇸🇬🇧🇨🇦', expected 'Flags: 🇨🇦🇬🇧🇺🇸'
FAILED tests/test_text.py::test_unicode_granular_diff[zwj_family] - pyo3_runtime.PanicException: Couldn't remove 2 elements from an array. Only 0 of them were successfully removed.
FAILED tests/test_text.py::test_unicode_granular_diff[rtl_ltr] - pyo3_runtime.PanicException: Couldn't remove 1 elements from an array. Only 0 of them were successfully removed.
FAILED tests/test_text.py::test_unicode_granular_diff[keycap] - pyo3_runtime.PanicException: Couldn't remove 1 elements from an array. Only 0 of them were successfully removed.
FAILED tests/test_text.py::test_unicode_granular_diff[emoji_boundaries] - pyo3_runtime.PanicException: Couldn't remove 1 elements from an array. Only 0 of them were successfully removed.
FAILED tests/test_text.py::test_unicode_granular_diff[japanese] - pyo3_runtime.PanicException: Couldn't remove 2 elements from an array. Only 0 of them were successfully removed.
FAILED tests/test_text.py::test_unicode_granular_diff[math_operators] - pyo3_runtime.PanicException: Couldn't remove 2 elements from an array. Only 0 of them were successfully removed.
======================================================================================================================== 20 failed, 16 passed in 0.63s =========================================================================================================================

This exactly matches the behavior we are seeing in AI extensions when they attempt to add multi-byte characters like emojis into notebooks using RTC MCP tools.

[dlqqq]

@xrl Looks good, thank you so much for working on this! The approach is correct and the test coverage is thorough. Approving.

@davidbrochart I think that this PR is good to merge - what do you think?

Also, I think there are two edge cases that aren't covered by this PR. I don't think that these are worth blocking the PR over since they are not regressions, but would be good to address as a follow-up.

1. sticky_index feature: StickyIndex.new() passes the Python code point index directly to yrs without converting to UTF-16 code units, and get_index() returns the raw yrs UTF-16 offset without converting back. This will be wrong for text containing characters above U+FFFF. Not a regression since this wouldn't have worked before anyways.

2. TextEvent deltas from observe(): The retain/delete lengths in change events come from yrs in UTF-16 code units. We should convert the UTF-16 code unit offsets back to the code point offsets that Python consumers expect. This isn't a regression either, since we weren't converting UTF-8 byte offsets back to code point offsets before anyways.

[dlqqq]

I did some digging into this. This is counting the number of UTF-16 code units per code point. Python counts length in code points.

Each code point is always either 1 or 2 UTF-16 code units in length; this is defined by the UTF-16 encoding itself. A surrogate pair (a Python code point comprised of 2 code units) is always composed of a high surrogate (>0xFFFF) and a low surrogate (<=0xFFFF). So this logic is correct.

Since Python indexes strings using code points, the _char_to_utf16() function can be implemented similarly by iterating over every code point and add 1 for each high surrogate to compute the UTF-16 byte offset. That is what sum(1 for ch in prefix if ord(ch) > 0xFFFF) does. So that function is correct as well.

[dlqqq]

Generated a small table for reference:

Visual character	Grapheme clusters (humans)	Code points (Python)	UTF-16 code units (Yjs)
A	1	1	1
é (precomposed, U+00E9)	1	1	1
é (decomposed, U+0065 + U+0301)	1	2	2
★ (U+2605)	1	1	1
中 (U+4E2D)	1	1	1
📊 (U+1F4CA)	1	1	2
🇺🇸 (flag)	1	2	4
👨‍👩‍👧‍👦 (family)	1	7	11
Hello	5	5	5

[dlqqq]

This looks like the correct approach. Yjs is inherently locked into UTF-16 code unit offsets for indices and lengths because JavaScript strings are encoded in UTF-16. This line configures pycrdt documents to also use UTF-16 code unit offsets for indices and lengths returned by Yrs's methods. This is pretty much required for compatibility with Yjs for containing anything except plain ASCII, since UTF-8 byte offsets != UTF-16 code unit offsets for everything outside plain ASCII (even in the BMP).

Reference: https://github.com/y-crdt/y-crdt/blob/195f0f4627be6b3fbc223bd3bfbadddc874458ca/yrs/src/doc.rs#L948

[davidbrochart]

I don't think that pycrdt should be hard-coded to use an offset kind of UTF-16. Just like Yrs, it should be parameterizable.

[davidbrochart]

Suggested change

# Return Python character count, not yrs UTF-16 code unit count

[xrl]

Done in 855af5c — dropped the inline comment and tightened the __len__ docstring to match.

[davidbrochart]

Suggested change

	def _char_to_utf16(text: str, char_index: int) -> int:
	def get_utf16_index(text: str, char_index: int) -> int:

[xrl]

Renamed in 855af5c. Added a get_utf8_index counterpart since offset_kind is now parameterizable per #379 (commit 41c8cc5) — both are re-exported from the top-level package. The dispatcher itself stays private (one-line conditional).

[davidbrochart]

Please use Google style docstrings (Args, Returns).

[xrl]

Done in 855af5c — both get_utf16_index and get_utf8_index now use Google-style Args / Returns docstrings.

[jbdyn]

I don't think that pycrdt should be hard-coded to use an offset kind of UTF-16.

Agreed, as it decides about yjs compatibility (thanks @dlqqq for the reference) and thus also about compatibility with other clients using the UTF-8 offset. The user needs to choose one.
Also, yrs might become the core implementation for all language wrappers/bindings at some point:

Eventually Yrs might replace the Yjs module by generating a wasm module from this package.

An offset parameter would come in handy during migration.

[jbdyn]

@davidbrochart With a paramterizable offset, Text and other datatypes also need a get_utf8_index conversion, right?

[davidbrochart]

@davidbrochart With a paramterizable offset, Text and other datatypes also need a get_utf8_index conversion, right?

I think so, yes.

[xrl]

@krassowski take a look now, please

[Copilot]

Pull request overview

This PR introduces configurable text offset semantics so pycrdt can safely operate on Unicode text (including non-BMP emoji) and interoperate with JS yjs when using UTF-16 code unit offsets.

Changes:

• Add offset_kind to Doc construction (Rust + Python) and expose Doc.offset_kind to Python.
• Update Python Text operations to convert Python character indices into the doc’s internal offset units (UTF-8 bytes or UTF-16 code units).
• Add extensive Unicode-focused tests and export get_utf8_index / get_utf16_index helpers.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/doc.rs	Adds offset_kind parsing to yrs Options and exposes an offset_kind getter.
python/pycrdt/_base.py	Wires offset_kind through to the Rust _Doc and validates consistency when reusing an existing doc.
python/pycrdt/_doc.py	Exposes Doc.offset_kind and documents the new constructor option.
python/pycrdt/_text.py	Adds UTF-8/UTF-16 index conversion helpers and updates Text mutation methods to use converted offsets.
python/pycrdt/__init__.py	Re-exports get_utf8_index / get_utf16_index as public helpers.
python/pycrdt/_pycrdt.pyi	Updates type stubs for the new _pycrdt.Doc constructor arg and property.
tests/test_text.py	Adds parameterized tests for Unicode behavior across both offset kinds and validates offset_kind API behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The offset_kind mismatch check compares the raw user string to doc.offset_kind, but the constructor accepts hyphenated aliases (e.g. utf-16). Passing an existing _Doc plus offset_kind='utf-16' will currently raise even though it's semantically the same as 'utf16'. Normalize offset_kind to the canonical form (e.g. strip hyphens / lowercase) before comparing, or accept both spellings in the comparison.

[xrl]

Resolved in 5e1df54 by dropping the hyphenated aliases entirely — with only 'utf8'/'utf16' accepted, the comparison is correct by construction. Added the accept-path test.

[Copilot]

The block comment says the granular edit uses "SequenceMatcher on byte offsets", but _apply_diff() uses SequenceMatcher(a=old_value, b=new_value) on Python strings and then applies edits using character indices. Update the comment to avoid implying byte-based offsets here (since that distinction is important in this PR).

Suggested change

	# initial content, then applies a granular edit (using SequenceMatcher on
	# byte offsets, matching how jupyter_ydoc.YUnicode.set() works), and verifies
	# the result is correct.
	# initial content, then applies a granular edit using SequenceMatcher on
	# Python strings / character indices in this helper, and verifies the result
	# is correct.

[xrl]

Fixed in 9a73db6.

[Copilot]

This constructor only sets options.offset_kind when offset_kind is explicitly provided, which means Doc() will still use the yrs default (OffsetKind::Bytes). That conflicts with the PR description/title claiming the Doc is configured to use UTF-16 offsets by default. Either set options.offset_kind = OffsetKind::Utf16 unconditionally (and adjust tests/docs accordingly) or update the PR description to reflect that UTF-16 is opt-in via offset_kind.

[krassowski]

update the PR description to reflect that UTF-16 is opt-in via offset_kind

I suppose we agreed on this one.

[xrl]

Description and title updated to reflect the opt-in design.

[Copilot]

__iadd__ now calls str(self) just to compute the append offset, which makes text += ... O(n) in the current text size and allocates the whole string. For appends you can avoid this by using the underlying integrated length in the doc’s offset units (e.g. self.integrated.len(txn._txn)) as the insert index, since that already matches the offset kind.

Suggested change

	current = str(self)
	offset = _char_to_offset(current, len(current), self.doc.offset_kind)
	offset = self.integrated.len(txn._txn)

[xrl]

Done in 63af1e2 — integrated.len(txn) is already in offset units. 2.4x faster appends on a ~1MB doc.

[Copilot]

get_utf16_index() returns char_index + extra, which becomes incorrect when char_index is out of range or negative because Python slicing clamps but char_index is not clamped (e.g. get_utf16_index('A📊B', 10) returns 11 instead of the end offset 4). This can produce offsets beyond the actual text length and break insert/delete/format calls in UTF-16 mode. Compute the result from the sliced prefix length (or clamp char_index to [0, len(text)]) so the returned UTF-16 offset always stays within bounds.

Suggested change

	return char_index + extra
	return len(prefix) + extra

[xrl]

Fixed in 87bb6d1 — both helpers now use Python slice-bound semantics (clamp out-of-range, negative from the end), documented and tested.

[krassowski]

@xrl did you have a chance to look at the Copilot feedback here? It looks like it picked up a conflict since.

[xrl]

Pushed a round of updates — this should be ready for another look:

• Merged main (yrs 0.27 / v0.14.0) and resolved the src/doc.rs conflict.
• Index edge cases (87bb6d1): the conversion helpers now have Python slice-bound semantics — out-of-range clamps, negative counts from the end — identically in both offset kinds. Before this, a utf16 doc would silently edit the wrong position for del t[-1] and panic (with partial mutation) on del t[1:100]. Both offset kinds now behave identically across every operation, and the new tests assert it.
• Dropped the "utf-8"/"utf-16" aliases (5e1df54): only the canonical spellings are accepted, which also fixes the doc-reuse mismatch check Copilot flagged. offset_kind is typed Literal["utf8", "utf16"] following the existing convention.
• Perf (63af1e2): += no longer materializes the document (integrated.len is already in offset units) — 2.4× faster appends on a 1MB doc; removed double string builds in del text[5:] / clear() / format().
• Docs (5566062): usage section on character indices and offset_kind, helpers added to the API reference, and the Doc docstring no longer overclaims what offset_kind does.
• Description rewritten per the earlier agreement — including a correction: @jbdyn was right that the wire format isn't the issue. Mixed offset-kind docs sync fine (now tested); offset_kind is about index semantics, not update compatibility.

One thing I want maintainer eyes on: released jupyter_ydoc works around #308 by passing UTF-8 byte offsets into Text (jupyter-server/jupyter_ydoc#370). Character-based indices break that workaround, so this likely wants to be 0.15.0 with a lockstep jupyter_ydoc change — happy to open that PR.

[xrl]

@krassowski I reworked things a bit, it should be better now

[dergachoff]

Small edge case from 0.14.2 / #408: embeds take one position but disappear from str(text), while this PR derives offsets from that string.

text = Text("AB")
doc["t"] = text
text.insert_embed(1, Map())
text.insert(len(str(text)), "X")
assert str(text) == "ABX"  # gets "AXB"

Might be worth adding an embed test when rebasing onto main.

[xrl]

@davidbrochart @krassowski I’ve merged current main and added embed-aware Text indexing with regression coverage in both offset modes. All 351 tests pass locally. Could you take another look?

@dergachoff thanks for catching the embed issue—would appreciate your eyes on the new indexing contract. Embeds now count as one position; nonempty string-valued embeds explicitly reject indexed operations because Yrs’ delta API cannot distinguish them from text runs.

[dergachoff]

@xrl Thanks for following up! I checked mixed Unicode and embeds in both offset modes, including rejected edits leaving the text unchanged. I didn't find any further issues with the new contract 👍