Skip to content

fix: recognise inline verbatim opening after an attached modifier - #32

Open
docker-run wants to merge 1 commit into
nvim-neorg:mainfrom
docker-run:fix/verbatim-after-attached-modifier
Open

fix: recognise inline verbatim opening after an attached modifier#32
docker-run wants to merge 1 commit into
nvim-neorg:mainfrom
docker-run:fix/verbatim-after-attached-modifier

Conversation

@docker-run

Copy link
Copy Markdown

Problem

_`Tree`_ is parsed by tree-sitter-norg as (underline (verbatim)), but this parser returns the backticks as plain Special tokens, so the verbatim is lost:

_`Tree`_   =>  AttachedModifier { '_', content: [Special('`'), Text("Tree"), Special('`')] }

With more than one verbatim inside a modifier the result is worse than a dropped node — the pairing shifts:

_`a` and `b`_  =>  [Special('`'), Text("a"), InlineVerbatim([Whitespace, Text("and"), Whitespace]), Text("b"), Special('`')]

and becomes the verbatim, and a/b do not.

Cause

Both attached-modifier opener candidates end with

.then(just(ParagraphSegmentToken::Whitespace).not())

which expresses "must not be followed by whitespace" but consumes the token it tests. When that token is a backtick, the opener swallows it and inline_verbatim never sees an opening delimiter. The swallowed token is then re-emitted as a plain Token by unravel_candidates, which is why it surfaces as Special('')`.

Fix

Peek instead of consuming — but only for `, so every other parse path is byte-identical:

let opener_rhs = || {
    choice((
        just(ParagraphSegmentToken::Special(VERBATIM)).rewind(),
        just(ParagraphSegmentToken::Whitespace).not(),
    ))
};

and unravel_candidates skips re-emitting the token it no longer consumed.

Rewinding unconditionally also fixes verbatim, but regresses */hello/*, world!: left_empty_opening_modifier stops consuming its right token, and / is then re-claimed as a new opener's left context, so the whole paragraph degrades to bare tokens. Hence the narrow condition.

Result

_`Tree`_       =>  AttachedModifier { '_', content: [InlineVerbatim([Text("Tree")])] }
_`a *b* c`_    =>  content stays raw: [Text("a"), Whitespace, Special('*'), Text("b"), ...]
_`a` and `b`_  =>  [InlineVerbatim([Text("a")]), Whitespace, Text("and"), Whitespace, InlineVerbatim([Text("b")])]

All matching tree-sitter-norg.

Tests

Existing suite passes unchanged (34/34, no snapshot updates). Adds verbatim_after_attached_modifier covering verbatim directly after a modifier, multiple verbatims in one modifier, raw content inside a nested verbatim, and the previously-working cases as regression guards.

Found while building a Norg editor on this crate, where the same case silently dropped formatting on read.

`_`Tree`_` is parsed by tree-sitter-norg as underline(verbatim), but
this parser returned the backticks as plain Special tokens, so the
verbatim was lost.

The attached-modifier opener candidates end with

    .then(just(ParagraphSegmentToken::Whitespace).not())

which asserts "not followed by whitespace" but *consumes* the token it
tests. When that token is a backtick the opener swallows it, so
`inline_verbatim` never sees an opening delimiter. With more than one
verbatim on the line the damage is worse than a dropped modifier: in
`_`a` and `b`_` the swallowed first backtick shifts the pairing, so
backticks 2 and 3 pair up and ` and ` becomes the verbatim.

Peek instead of consuming, for '`' only, so every other parse path is
byte-identical; `unravel_candidates` then skips re-emitting the token
it no longer consumed. Rewinding unconditionally also works for
verbatim but regresses `*/hello/*, world!`, where the left-empty opener
stops consuming its right token and `/` is re-claimed as a new opener's
left context.

Adds a snapshot test covering verbatim directly after a modifier,
multiple verbatims in one modifier, raw content inside a nested
verbatim, and the previously-working cases as regression guards.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant