• We can have a top-level Production::Trivium in the grammar, to indicate these new nodes.

  • Each version defines a ScannerRef (like tokens)
  • It will have productions like Whitespace, EndOfLine, LineComment, etc...

• Each Trivium production will have a placement enum field to indicate if it is:

  • Leading: MultiLineComment
  • Trailing: EndOfLineComment and EndOfLine
  • LeadingAndTrailing: Whitespace
  • This way, we can remove TrailingTrivia and TrailingTrivia nodes from grammar and CST

• In ParseOutput, we can add a new field trailing_trivia which contains anything not captured by root_node's last token (including newlines).

  • This way, we can remove EndOfFileTrivia and SourceUnitMembersList nodes from grammar.
  • This also means that parsing any RuleKind with trivia will succeed, as we will always detect and consume leading/trailing trivia around it.
  • Adding validation to ensure nothing (parsers, scanners, or operators) can reference trivia nodes in grammar.

• introduce a TokenKind::is_trivia() -> bool helper to distinguish between this and other scanner tokens

Suggestion extracted from #438