Part of #2499

• Document the limitations of ANTLR (see the open kotlin tasks)

  • Output of ANTLR is a parse tree, and not a syntax tree. Therefore the tree is overly detailed. Ideally one would convert the parse tree into a custom AST, with custom nodes, which can be extends with custom attributes (e.g. for XPath attributes), etc.
  • Antlr nodes cannot be extended (e.g. adding additional getters/attributes to expose some information for XPath). The issue is, that they're all generated into the same file...

    Of note, that the generated tree is not an abstract syntax tree in any way, it's a parse tree. There's a node for every token, or every error, which makes gigantic trees. Antlr recommends building a separate AST if we want one, but then we would have to define all the nodes and visitor interface ourselves... This is a fundamental problem with using antlr, possibly it should be mentioned somewhere in the doc.

  • FYI - Kotlin task is: [kotlin] Improve Kotlin AST #4337
  • FYI - Issue for this: [xpath] Custom XPath attributes for ANTLR based languages #4338

• Update/verify existing doc "adding_a_new_antlr_based_language.md"

  • See [doc] Add ANTLR documentation #1881 for the existing doc
  • There exists some doc, but that needs to be overhauled for changes in API. See also [core,swift] Refactor antlr implementation #2463
  • Key users of the doc: language implementers, that want to add a new language to PMD based
    on a antlr grammar.
  • Other users of the doc: rule writers, that want to write a new rule for a language
    implemented using antlr (here comes the difference between AnltrBaseRule and AbstractRule...)

• This task depends on the API Clarification [core] Clarify API for ANTLR based languages #2500