fix: recognise inline verbatim opening after an attached modifier - #32
Open
docker-run wants to merge 1 commit into
Open
fix: recognise inline verbatim opening after an attached modifier#32docker-run wants to merge 1 commit into
docker-run wants to merge 1 commit into
Conversation
`_`Tree`_` is parsed by tree-sitter-norg as underline(verbatim), but
this parser returned the backticks as plain Special tokens, so the
verbatim was lost.
The attached-modifier opener candidates end with
.then(just(ParagraphSegmentToken::Whitespace).not())
which asserts "not followed by whitespace" but *consumes* the token it
tests. When that token is a backtick the opener swallows it, so
`inline_verbatim` never sees an opening delimiter. With more than one
verbatim on the line the damage is worse than a dropped modifier: in
`_`a` and `b`_` the swallowed first backtick shifts the pairing, so
backticks 2 and 3 pair up and ` and ` becomes the verbatim.
Peek instead of consuming, for '`' only, so every other parse path is
byte-identical; `unravel_candidates` then skips re-emitting the token
it no longer consumed. Rewinding unconditionally also works for
verbatim but regresses `*/hello/*, world!`, where the left-empty opener
stops consuming its right token and `/` is re-claimed as a new opener's
left context.
Adds a snapshot test covering verbatim directly after a modifier,
multiple verbatims in one modifier, raw content inside a nested
verbatim, and the previously-working cases as regression guards.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
_`Tree`_is parsed bytree-sitter-norgas(underline (verbatim)), but this parser returns the backticks as plainSpecialtokens, so the verbatim is lost:With more than one verbatim inside a modifier the result is worse than a dropped node — the pairing shifts:
andbecomes the verbatim, anda/bdo not.Cause
Both attached-modifier opener candidates end with
which expresses "must not be followed by whitespace" but consumes the token it tests. When that token is a backtick, the opener swallows it and
inline_verbatimnever sees an opening delimiter. The swallowed token is then re-emitted as a plainTokenbyunravel_candidates, which is why it surfaces asSpecial('')`.Fix
Peek instead of consuming — but only for
`, so every other parse path is byte-identical:and
unravel_candidatesskips re-emitting the token it no longer consumed.Rewinding unconditionally also fixes verbatim, but regresses
*/hello/*, world!:left_empty_opening_modifierstops consuming its right token, and/is then re-claimed as a new opener's left context, so the whole paragraph degrades to bare tokens. Hence the narrow condition.Result
All matching
tree-sitter-norg.Tests
Existing suite passes unchanged (34/34, no snapshot updates). Adds
verbatim_after_attached_modifiercovering verbatim directly after a modifier, multiple verbatims in one modifier, raw content inside a nested verbatim, and the previously-working cases as regression guards.Found while building a Norg editor on this crate, where the same case silently dropped formatting on read.