Description

PMD utilizes the Apex Jorje library to parse Apex source and generate an AST. The fact that this library is closed-source introduces two major headaches for us:

• Without source transparency, as a policy, there are severe restrictions on how and where code can be run in our environment. The closed-source Jorje dependency is tainting the whole PMD Apex analyzer.
• This limits enhancement or modification, either to provide early patches for new language features or for other extensions.

I can't imagine it's particularly pleasant to develop on top of an undocumented API. Or to slice-and-dice and integrate new versions.

Describe the solution you'd like

I'd like to explore (and contribute to) the transition to a fully open-source Apex front-end.

In my organization, we're using @nawforce's Apex grammar to parse almost 1MLoC, now with zero issues. Built on top of this, we've got an implementation to convert the resulting parse tree to an abstract syntax tree data structure-- excluding the SOQL query syntax, which isn't relevant here. This is not yet open-sourced, but that is obviously a prerequisite to move forward.

The integration of this code ought to be fairly mechanical, to translate this AST data structure to the existing PMD AST. If there are any gaps or incompatibilities, I guess I'm volunteering to accommodate the requested modifications. Validating the semantic equivalence of the representation is a little trickier, but my sense is that this wouldn't be a big hurdle.

The biggest risk is likely in swapping out one parser for another, and that the tool no longer parses an Apex file that it previously accepted. OTOH, any of these issues could actually be fixed upstream, which is not currently a possibility.

So... that's the idea and the offer... wanted to gauge your interest and gather your concerns.

The chosen open source front-end: Summit AST

... is Summit AST
Maven Repo for Snapshots: https://google.oss.sonatype.org/content/repositories/snapshots/com/google/summit/summit-ast/

Releases: https://central.sonatype.dev/artifact/com.google.summit/summit-ast/1.0.0/overview
https://repo1.maven.org/maven2/com/google/summit/summit-ast/

PR List

Note: All PRs are merged into branch https://github.com/pmd/pmd/tree/experimental-apex-parser

• [apex] Summit-AST Apex module - Part 1 #4093
• [apex] Summit-AST Apex module - Part 2 - expression nodes #4151
• [apex] Summit-AST Apex module - Part 3 - initializers #4171
• [apex] Summit-AST Apex module - Part 4 - statements #4206
• [apex] Summit-AST Apex module - Part 5 - annotations, triggers, misc. #4219
• [apex] Merge 6.52 into experimental-apex-parser #4242
• [apex] Summit-AST Apex module - Part 6 Passing testsuite #4251
• [apex] Bump summit-ast to new release 2.1.0 (and remove workaround) #4448
• [apex] Merge main (7.x) branch into experimental-apex-parser and fix tests #4479
• [apex] Merge master into experimental apex parser #4702
• [apex] New Parser - parsing errors and rule differences #4722
• [apex] Fixes for experimental apex parser #4739
• [apex] Use apex-parser and Summit AST #4806

Related work

• [apex] Remove Jorje leaks outside ast package #4081
• [visualforce] Replace uses of Jorje types in pmd-visualforce #4226
• [apex] Deprecate ApexRootNode.getApexVersion #4280