At the moment we use our Rust macros 1.0 (aka DSL v1) to define the language for our parser. We already have:

1. YAML spec
2. DSL v1
3. DSL v2

So to cut down the different definitions, we should migrate to the newest definition (see #617).

Migration PR:

• Use DSL v2 for parser codegen #650

Impacts the AST:

• remove allow_empty from DSL v2 #693
• Un-inline the enums in the CST #698
• Clean up precedence parser codegen since DSL v2 #699
  • Clean up PrecedenceExpression#rule_name and adapt the codegen model to fit the v2 definition better
  • Precedence items should also produce their own NonTerminalKind kind, wrapping the single child (PrecedenceExpression or PrimaryExpression).
  • PrecedenceExpression should get their own parse_XXX() function.

Does not impact the AST:

• [ ] Re-introduce EOF/EOI (End of Input) virtual token/trivia parser
• [ ] Change LeadingTrivia definition in v2 to allow being terminated by EOI as well (changed in Use DSL v2 for parser codegen #650) (tracked in Define end-of-file trivia as part of the language definition #1020)
• Don't monkey-patch SourceUnit rule to always be followed by EndOfFileTrivia (copies v1; trivia model in v2 is a bit different)
• Always try to parse ZeroOrMore(AnyTriviaItem) at the end when parsing any production with pub fn parse
• Stop leaking identifiers by adapting either v1 or v2 models (Stop leaking identifiers in parser generator and use more DSLv2 types #977)
• Keyword trie inclusion should be reworked to not require synthetic rules over all keywords (v1 model) (Disentangle grammar resolution and related PG code #1018)
• Revert use v1 grammar for string literals #646 (done in support unicode characters in string literals up to 0.7.0 #800)
• Ordering of parsing statements variants (context).~ (separated into ordering of parsing statements variants #1019)
• See if we can remove Scanner::not_followed_by completely and automatically calculate it instead (context).
• Remove redundant versioning checks in precedence expressions (context).
• Precedence individual operators need to be deduped across versions (context).
• Delete codegen_grammar crate. (Co-locate parser generator logic #972)