TODO on master

• [core] Deprecate CharStream & generated classes #2244 Internalize CharStream and its implementations
• [core] Deprecate CharStream & generated classes #2244 Internalize all classes generated by JavaCC (in particular, *ParserConstants, *ParserTreeConstants, *ParserTokenManager).

TODO on 7.0

• PMD languages to port
  • Java, Modelica [core] Merge JavaCC build scripts #2211
  • VM [vm] Adapt VM module to the new javacc wrapper #2289
  • VF, JSP [vf, jsp] Adapt JSP & VF modules to new javacc wrapper #2294
  • PLSQL [plsql] Adapt PLSQL to new javacc wrapper #2369
• CPD languages to port (must set the property no-jjtree on the target in the pom.xml)
  • Python [python] Make python use new javacc wrapper #2293
  • ObjectiveC [objectivec] Use new javacc wrapper script in Objective C module #2361
  • Javascript [javascript] Use new javacc wrapper in JS module #2362
  • Matlab [matlab] Use new javacc wrapper script in Matlab module #2359
  • C++ [cpp] Use new javacc wrapper script in C++ module #2360
• TODO once all languages are ported:
  • Move CharStream to impl.javacc package
  • Cleanup TokenMgrError
    • Remove "error codes"
  • Cleanup AbstractNode ([core] Cleanup Node and AbstractNode #2447)
    • Remove token methods & fields and put them on AbstractJjtreeNode
    • Remove text position fields (beginLine/etc)
    • Remove ID field
  • Cleanup Node interface ([core] Cleanup Node and AbstractNode #2447)
    • Remove JJTree methods (jjtClose, jjtGetId, etc)
  • Decide on *TokenKinds: published API or internal?
  • Decide on *TokenManager: published API or internal?

Should we really consider *TokenKinds a published API? Since token kinds are compile-time constants, adding or removing tokens in the grammar may break binary compatibility implicitly.

Remaining tasks

• [core] Rename TokenMgrError to LexException, Tokenizer to CpdLexer #4065
  • maybe rename to something not javacc specific (eg LexException/ScanException)
  • LexException, then Tokenizer -> CpdLexer