feat: annotated code blocks by clason · Pull Request #65 · neovim/tree-sitter-vimdoc

clason · 2022-11-05T12:29:45Z

This is a Lua block: >lua
    local foo = 'bar'

This is a Vimscript block: >vim
    au FileType lua setl sw=2

This is a C block: >c
    int *p = get_local_errno(); *p = EINVAL
<
This is a standard block: >
    $ nvim --clean +'q'
<

Note that the alternative syntax lua> is much trickier, since the rule for the language marker lua conflicts with normal text rule for the word lua. In either case, the added markers are trivial (and necessary) to add to the legacy syntax file.

Closes #2

grammar.js

clason · 2022-11-05T16:34:50Z

@justinmk I think this is good to go. The next step is to adapt scripts/gen_help_html.lua to

eat the language node
only show the code node wrapped in a <code class='language-<language>'></code> block

and inject highlight.js to the neovim.io/doc/user pages. But for that the ball is in your corner :)

corpus/codeblock.txt

justinmk · 2022-11-06T22:22:40Z

grammar.js

+      optional(alias(token.immediate(/[a-z0-9]+/), $.language)),
      token.immediate('\n'),
-      repeat1(alias($.line_code, $.line)),
+      alias(repeat1(alias($.line_code, $.line)), $.code),


is this for semantics or does it have a functional purpose? Maybe $.codeline fits better with the existing naming patterns.

It's necessary (as far as I can tell) so the whole block gets punted to the injected parser; you can't parse, e.g.,

for k,v, in pairs(foo) do print(k, v, ) end

purely line by line

To be clear, this aliases the whole set of repeated lines as a single node, not each single line.

Why can't we do that with (codeblock) itself?

Because that includes the markers, which are not part of the injected code.

(But I'm not married to the name, of course; I just need a single node that I can capture as @contents; see the injections.scm query.)

Because that includes the markers, which are not part of the injected code.

I want to believe that's solvable because they're anonymous :) Any hope?

Not unless you give me some ;) Honestly, it would help me if you could explain the nature of the objection to doing it the way I was able to.

just elegance/aesthetics (unnecessary nesting)

justinmk · 2022-11-12T14:19:36Z

I found some cases in the help docs that we should test for:

runtime/bugreport.vim|20 col 20| :  !echo "uname -a" >bugreport.txt
runtime/doc/eval.txt|1051 col 2| >0 / 0  =  0x7fffffff	(like positive infinity)
runtime/doc/eval.txt|1057 col 2| >0 / 0  =  0x7fffffffffffffff	(like positive infinity)
runtime/doc/luaref.txt|460 col 24| while (  step  >0 and  var  <=  limit  )
runtime/doc/usr_02.txt|548 col 7| :help >cont
runtime/doc/usr_10.txt|707 col 17| sort <input.txt >output.txt

clason · 2022-11-12T16:56:33Z

runtime/bugreport.vim|20 col 20| : !echo "uname -a" >bugreport.txt

Not a help file.

runtime/doc/eval.txt|1051 col 2| >0 / 0 = 0x7fffffff (like positive infinity)
runtime/doc/eval.txt|1057 col 2| >0 / 0 = 0x7fffffffffffffff (like positive infinity)

Not at the end of the line, so doesn't match.

runtime/doc/luaref.txt|460 col 24| while ( step >0 and var <= limit )
runtime/doc/usr_02.txt|548 col 7| :help >cont
runtime/doc/usr_10.txt|707 col 17| sort <input.txt >output.txt

Inside a codeblock (and not on the end of the line or not [a-z0-9]+), so doesn't match.

justinmk · 2022-11-21T13:40:00Z

Not at the end of the line, so doesn't match.

worth adding an explicit case to the corpus?

clason · 2022-11-21T13:42:55Z

Oh, I see what you mean. Yes, I can see about adding a case with whitespace and not at the end of the line; maybe variants of the usr_*.txt examples.

justinmk · 2022-11-21T13:52:37Z

corpus/codeblock.txt

      (codeblock
-        (line))))
+        (code
+          (line)))))


I'm admittedly going in circles, but: I guess it makes sense for (code) to be named (block). Because the block+lines here (nested in codeblock) is analogous to non-code (block (line...)).

I'm not sure what the best practice/convention is for grammars: is it better to re-use names for similar concepts, or should we use globally unique names like (code (codeline)) ?

I guess it makes sense for (code) to be named (block).

But you can't, since (block) is already defined at this point (as something different) -- you'll get a conflict in the grammar you need to resolve somehow. And a different name is by far the easiest way to do that.

Not sure I follow. It can be aliased, as done elsewhere (e.g. we already alias to (line) here). And conflicts are based on the "fully qualified node path", irrespective of the node names.

Anyways let's go with (code (codeline)) I guess.

Go ahead, try it.

Just tried this, it works fine (no conflicts):

alias(repeat1(alias($.line_code, $.line)), $.block),

But this is just academic. If you're ok with (code (codeline)) , I think I favor that too.

You do realize that the alias is to lines (plural) of code?

Yes. But clearly I'm missing something or we're miscommunicating. Note that (codeline) should be understood as any number of children, e.g.:

(code (codeline) (codeline) …)

If I still misunderstood, sorry. Like I said, my comments aren't blockers.

Somewhat, yes, because I don't get what the end goal is here, so it feels like stabbing in the dark, which is frustrating.

(I do admit that my last comment was sheer pedantry, born out of said feeling.)

well I started this thread by literally saying "I'm admittedly going in circles". The goal was just to explore options.

Just tried this, it works fine (no conflicts):

alias(repeat1(alias($.line_code, $.line)), $.block),

Interesting, that's what I tried. Looks like switching from optional to choice avoids the conflict then; that's good. Feel free to rename as you wish before tagging a release so I can downstream the queries. (And when you do, may want to update the description in the README, which I missed -- sorry!)

Note that (codeline) should be understood as any number of children, e.g.:
...
But this is just academic. If you're ok with (code (codeline)) , I think I favor that too.

There seems to be indeed some miscommunication. My point is that the contents must be a single block that you pass to the injected parser. Otherwise every single line gets parsed in isolation which a) is wasteful and b) leads to errors with constructs spanning multiple lines (as in my -- admittedly academic -- example above).

grammar.js

justinmk

No blockers, just some nits on naming

``` This is a Lua block: >lua local foo = 'bar' This is a Vimscript block: >vim au FileType lua setl sw=2 This is a C block: >c int *p = get_local_errno(); *p = EINVAL < This is a standard block: > $ nvim --clean +'q' < ```

justinmk · 2022-11-21T21:19:06Z

not a blocker, but this bugs me about >lua :

more likely to have false positives than lua>
lua> is more visually noticeable (for non-concealed text) because > is at EOL as usual

This is just thinking out loud. Will merge this later this week.

clason · 2022-11-21T21:36:17Z

Sorry, I tried to make that work; I failed.

clason mentioned this pull request Nov 5, 2022

docs: syntax highlighting for (some) code examples neovim/neovim#20912

Closed

7 tasks

lewis6991 reviewed Nov 5, 2022

View reviewed changes

grammar.js Outdated Show resolved Hide resolved

lewis6991 reviewed Nov 5, 2022

View reviewed changes

grammar.js Outdated Show resolved Hide resolved

justinmk reviewed Nov 6, 2022

View reviewed changes

corpus/codeblock.txt Show resolved Hide resolved

justinmk reviewed Nov 6, 2022

View reviewed changes

justinmk reviewed Nov 21, 2022

View reviewed changes

grammar.js Outdated Show resolved Hide resolved

justinmk approved these changes Nov 21, 2022

View reviewed changes

feat: annotated code blocks

d6ee1fc

``` This is a Lua block: >lua local foo = 'bar' This is a Vimscript block: >vim au FileType lua setl sw=2 This is a C block: >c int *p = get_local_errno(); *p = EINVAL < This is a standard block: > $ nvim --clean +'q' < ```

justinmk merged commit ce20f13 into neovim:master Nov 21, 2022

clason deleted the injections branch November 22, 2022 09:01

Conversation

clason commented Nov 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

clason commented Nov 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clason Nov 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justinmk Nov 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justinmk commented Nov 12, 2022

Uh oh!

clason commented Nov 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

justinmk commented Nov 21, 2022

Uh oh!

clason commented Nov 21, 2022

Uh oh!

justinmk Nov 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justinmk Nov 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justinmk Nov 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justinmk Nov 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clason Nov 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

justinmk left a comment

Choose a reason for hiding this comment

Uh oh!

justinmk commented Nov 21, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clason commented Nov 21, 2022

Uh oh!

clason commented Nov 5, 2022 •

edited

Loading

clason commented Nov 5, 2022 •

edited

Loading

clason Nov 6, 2022 •

edited

Loading

justinmk Nov 7, 2022 •

edited

Loading

clason commented Nov 12, 2022 •

edited

Loading

justinmk Nov 21, 2022 •

edited

Loading

justinmk Nov 21, 2022 •

edited

Loading

justinmk Nov 21, 2022 •

edited

Loading

justinmk Nov 21, 2022 •

edited

Loading

clason Nov 22, 2022 •

edited

Loading

justinmk commented Nov 21, 2022 •

edited

Loading