Great! This would be the ideal answer to #1472. I was able to solve with a tricky scanner, but this sounds much better.
Question: how will this treat white space? I ask because the notes above mention it will be triggered (in part?) by the word token being defined, and this is connected, I found, with the regex notion of word boundaries, such as white space. For my case I need to exclude reserved words from identifiers even if they are broken up by spaces. Spaces need to be eaten up as if they didn't exist. Put another way, will it be possible to fully control what a word boundary is?

Ok, I believe this PR is now working correctly. I've opened PRs making use of the API in some grammars. There are some improvements that I'd like to make before merging and releasing this.

EDIT - moved these TODOs to tasks in the PR description.

Trying another way: can the property be massaged to produce an error when an identifier contains an embedded reserved word?

tl;dr

Suppose and is the token for logical and, and we are emulating a parser that expects all spaces to be stripped at a prior stage. Then sand and grand is an error because it is seen as s and and gr and. TS will handle that fine, but will typically allow a declaration of sand, since and is not expected in that context. But in languages like this such a declaration is forbidden to protect the programmer. My solution was to lex identifiers using a scanner that rejects anything with a reserved word embedded in it. Wondering if the PR would provide a scanner-free solution.

First of all, thank you very much for this - I think this looks like a great approach and has the potential to be a very straightforward API for working with error recovery and incomplete documents.

I have a couple of notes, having tried this out today, and I hope to be able to provide some more detailed notes soon:

1. This seems to impact the speed of compilation quite drastically, i.e. on the order of a 10x slowdown. Is that expected? Perhaps I'm building it incorrectly, I'm not a Rust user, but I've built a release build using cargo and temporarily just copied it into my node_modules folder.
2. I've had trouble so far if I try to use the $ in the reserved: ($) => ... top-level configuration, i.e. if my reserved tokens are anything but strings. Specifically, I want to be able to use regexes to allow for case-insensitive reserved keywords, but the results don't seem to be as good as when using string literals.
3. Am I right to assume that if a keyword is in the reserved context at some point, then there should never be a parse tree result which includes a name with that keyword as its text, in that context? If so, I'm definitely seeing cases where that is not the case (if this is a bug, I can try to distill this down to some minimal test cases). And if not, is there an equally simple way for me to think about what reserved should restrict in the resulting parse trees?

Thanks!

This seems to impact the speed of compilation quite drastically

I didn't expect that. I'd like to fix it before merging. Can you share a grammar that reproduces the issue?

Specifically, I want to be able to use regexes to allow for case-insensitive reserved keywords

They shouldn't need to be strings. I do think that right now, they need to be existing tokens in the grammar which "overlap" (match some common strings) with your grammar's word token. I'd be curious about cases where it's not working as expected with regexes.

Am I right to assume that if a keyword is in the reserved context at some point, then there should never be a parse tree result which includes a name with that keyword as its text, in that context?

Yeah, although there is some subtlety right now in determining which keywords are in the context at a given point. This is related to the second problem I mentioned above, about the reserved function being a bit hard to use correctly.

can the property be massaged to produce an error when an identifier contains an embedded reserved word?

@dfgordon For right now, I don't think this feature will support that. I think using custom code in an external scanner is probably the best way to do it.

@maxbrunsfeld Thanks for your reply. We're putting together a small grammar which reproduces the issue we're seeing, and we will follow up shortly.

@maxbrunsfeld we put up demo grammar as @paf31 promised: https://github.com/DeepChannel/reserved-word

Here are some of the issues we are experiencing:

1. We can not use plain text keywords inside reserved: [...] array. When compiling TS complains with following message "Reserved words must be tokens".

This works:

reserved: ($) => [$.SELECT, $.AS, $.UNION, $.INTERSECT, $.EXCEPT],

But this does not:

reserved: ($) => ["SELECT", "AS", "UNION", "INTERSECT", "EXCEPT"],

2. Behaviour is same for both - case insensitive and plain text rules. kywords.js includes following two lines which allow to switch between such rules:

result[value.toUpperCase()] = (_) => toCaseInsensitiveRule(value);
// result[value.toUpperCase()] = (_) => value.toUpperCase();

3. Overriding reserved keywords doesn't have any effect also. Below is the example where for the first choice global reserved: [] set should work and for the second rule there is an override:

alias: ($) =>
  choice(
    $.identifier,
    seq(
      $.AS,
      // Allow any reserved keywords except SELECT
      reserved([$.SELECT], $.identifier)
    )
  ),

4. In grammar.json I see all of the defined reserved rules:

"reserved": [
    {
      "type": "SYMBOL",
      "name": "SELECT"
    },
    {
      "type": "SYMBOL",
      "name": "AS"
    },
    {
      "type": "SYMBOL",
      "name": "UNION"
    },
    {
      "type": "SYMBOL",
      "name": "INTERSECT"
    },
    {
      "type": "SYMBOL",
      "name": "EXCEPT"
    }
]

5. In parser.c there is a definition for reserved keywords count but final result doesn't actually work:

#define MAX_RESERVED_WORD_SET_SIZE 5

Maybe this is the issue related to .wasm?

Following example should not parse UNION keyword as identifier:

SELECT a + 
UNION 
SELECT 1 SELECT, a - b

Parse tree:

program [0, 0] - [3, 0]
  statement [0, 0] - [2, 22]
    select_stmt [0, 0] - [2, 22]
      simple_select [0, 0] - [2, 22]
        SELECT [0, 0] - [0, 6]
        ERROR [0, 7] - [2, 6]
          target_el [0, 7] - [2, 6]
            expr [0, 7] - [1, 5]
              binary_expr [0, 7] - [1, 5]
                left: expr [0, 7] - [0, 8]
                  identifier [0, 7] - [0, 8]
                    name [0, 7] - [0, 8]
                right: expr [1, 0] - [1, 5]
                  identifier [1, 0] - [1, 5]
                    name [1, 0] - [1, 5]
            alias [2, 0] - [2, 6]
              identifier [2, 0] - [2, 6]
                name [2, 0] - [2, 6]
        target_list [2, 7] - [2, 22]
          target_el [2, 7] - [2, 15]
            expr [2, 7] - [2, 8]
              int [2, 7] - [2, 8]
            alias [2, 9] - [2, 15]
              identifier [2, 9] - [2, 15]
                name [2, 9] - [2, 15]
          target_el [2, 17] - [2, 22]
            expr [2, 17] - [2, 22]
              binary_expr [2, 17] - [2, 22]
                left: expr [2, 17] - [2, 18]
                  identifier [2, 17] - [2, 18]
                    name [2, 17] - [2, 18]
                right: expr [2, 21] - [2, 22]
                  identifier [2, 21] - [2, 22]
                    name [2, 21] - [2, 22]

I would expect it to look something like this:

program [0, 0] - [3, 0]
  statement [0, 0] - [2, 22]
    select_stmt [0, 0] - [2, 22]
      simple_select [0, 0] - [0, 12]
        SELECT [0, 0] - [0, 6]
        target_list [0, 7] - [0, 12]
          target_el [0, 7] - [0, 12]
            expr [0, 7] - [0, 12]
              [ERROR]
                binary_expr [0, 7] - [0, 12]
                  left: expr [0, 7] - [0, 8]
                    identifier [0, 7] - [0, 8]
                      name [0, 7] - [0, 8]
      select_combinator [1, 0] - [1, 5]
        UNION [1, 0] - [1, 5]
      select_stmt [2, 0] - [2, 22]
        simple_select [2, 0] - [2, 22]
          SELECT [2, 0] - [2, 6]
          target_list [2, 7] - [2, 22]
            target_el [2, 7] - [2, 15]
              expr [2, 7] - [2, 8]
                int [2, 7] - [2, 8]
              alias [2, 9] - [2, 15]
                identifier [2, 9] - [2, 15]
                  name [2, 9] - [2, 15]
            target_el [2, 17] - [2, 22]
              expr [2, 17] - [2, 22]
                binary_expr [2, 17] - [2, 22]
                  left: expr [2, 17] - [2, 18]
                    identifier [2, 17] - [2, 18]
                      name [2, 17] - [2, 18]
                  right: expr [2, 21] - [2, 22]
                    identifier [2, 21] - [2, 22]
                      name [2, 21] - [2, 22]

@maxbrunsfeld below are benchmark results for our relatively large grammar.

Benchmark results for master branch:

Benchmark 1: npm run build
  Time (mean ± σ):     13.106 s ±  0.553 s    [User: 13.244 s, System: 0.773 s]
  Range (min … max):   12.451 s … 14.053 s    10 runs

Benchmark results for reserved-rule branch:

Benchmark 1: npm run build
  Time (mean ± σ):     196.793 s ±  9.894 s    [User: 196.355 s, System: 0.953 s]
  Range (min … max):   185.614 s … 211.372 s    10 runs

npm run build resolves to tree-sitter generate && tree-sitter build-wasm

Machine:

MacBook Pro (16-inch, 2019)
Processor 2,3 GHz 8-Core Intel Core i9
Memory 16 GB 2667 MHz DDR4

Am I right to assume that if a keyword is in the reserved context at some point, then there should never be a parse tree result which includes a name with that keyword as its text, in that context?

Yeah, although there is some subtlety right now in determining which keywords are in the context at a given point. This is related to the second problem I mentioned above, about the reserved function being a bit hard to use correctly.

I meant to follow up with a bit more context here, based on the example repo that @ioseb provided. If you try the playground in that repo, you can see the problem that I was originally referring to:

• If you paste the input text SELECT x UNION there, you will see that UNION is correctly marked as an error node, because UNION is reserved, so it cannot take the place of the alias.
• SELECT x as UNION parses with UNION as an identifier - again this is correct, because the nested reserved call has removed UNION from the reserved set in that scope, as I understand it.
• SELECT x SELECT y is interesting. I'll paste the output below.
• SELECT x AS SELECT y seems to result in a similar tree.

Output for SELECT x SELECT y:

program [0, 0] - [1, 0]
  statement [0, 0] - [0, 17]
    select_stmt [0, 0] - [0, 17]
      simple_select [0, 0] - [0, 17]
        SELECT [0, 0] - [0, 6]
        ERROR [0, 7] - [0, 15]
          target_el [0, 7] - [0, 15]
            expr [0, 7] - [0, 8]
              identifier [0, 7] - [0, 8]
                name [0, 7] - [0, 8]
            alias [0, 9] - [0, 15]
              identifier [0, 9] - [0, 15]
                name [0, 9] - [0, 15]
        target_list [0, 16] - [0, 17]
          target_el [0, 16] - [0, 17]
            expr [0, 16] - [0, 17]
              identifier [0, 16] - [0, 17]
                name [0, 16] - [0, 17]

As you can see, the ERROR range was introduced surrounding x SELECT, but the second SELECT was still parsed as a name, even though it was in the reserved set.

If you could please help me to understand what the intended behavior should be here, I'd appreciate it, because my understanding is that the parser should have backtracked at this point, and returned some tree in which no SELECT keyword is marked as a name. Hopefully that's the intent, and I've not misunderstood, but please correct me if I'm wrong.

Thanks again!

I'm working on a D grammar for tree-sitter. D is a C family language, and like most such languages has a some (D has a plethora of them) words that are reserved in all contexts. There are a ton of conflicts in the grammar that I'm very happy that this PR will address. (At one point I considered moving the entire logic of isolating keywords from identifiers in the lexer -- I had an implementation that did it even -- but I very much like the approach here better.) Btw, my lexer does identify strings, comments, and a couple of other oddball constructs so that they show up as tokens and cannot conflict with reserved words.

There is also another way that help when we need soft keywords see here https://sqlite.org/src/doc/trunk/doc/lemon.html#pfallback then %fallback directive.

I'm working on a pretty simple search DSL that allows boolean operators: and, or, not, and search tokens. From other discussions, it seems the only way is to do this in an external scanner.

I want to contribute one way of doing it using regex. Basically I'm negating and/or/not, but it's very ugly as negative lookahead is not supported.

        // original regex: /([^<>\s\\'"()]|\\.)+/
        // exclude the keywords: and, or, not
        // regex logic is: 
        // 1: any string not start with a, o, or n
        // 2: if starts with a,   must not follow n
        // 3: if starts with an,  must not follow d
        // 4: if starts with and, must follow with more characters
        // ... (repeat the same logic for or, not)
        token: _ => /((?:(?:[^aon<>\s\\'"()]|\\.)(?:[^<>\s\\'"()]|\\.)*)|a|(?:a[^n<>\s\\'"()](?:[^<>\s\\'"()]|\\.)*)|an|(?:an[^d<>\s\\'"()](?:[^<>\s\\'"()]|\\.)*)|(?:and(?:[^<>\s\\'"()]|\\.)+)|o|(?:o[^r<>\s\\'"()](?:[^<>\s\\'"()]|\\.)*)|(?:or(?:[^<>\s\\'"()]|\\.)+)|n|(?:n[^o<>\s\\'"()](?:[^<>\s\\'"()]|\\.)*)|no|(?:no[^t<>\s\\'"()](?:[^<>\s\\'"()]|\\.)*)|(?:not(?:[^<>\s\\'"()]|\\.)+))/i,

I very much look forward to having this feature merged.

Hey, just commenting for everyone in this thread that's interested, #3896 supersedes this PR, and is now working with everything on the todo list fixed minus the Allow specifying reserved words that aren't used elsewhere. item, since Max and I agreed that it doesn't really make sense to reserve keywords that aren't used anywhere - it feels like a weird feature, and doesn't seem like it'd be used.

Questions/Discussion should happen in the follow-up PR, so I'll be closing this