[core] Refactor CPD token filtering#1039
Conversation
- Define a generic `TokenFilter` interface in pmd-core - Provide a base, extension-friendly `JavaCCTokenFilter` to process and filter JavaCC token streams, honoring `CPD-OFF` and `CPD-ON` comments - Refactor the `JavaTokenizer` to use `JavaCCTokenFilter` by extending it and adding custom Java-specific token filters
|
The core is done, I'm starting to add it to different languages. |
|
@adangel @oowekyala this is ready now. I left out 3 JavaCC based languages, the reasons for each are detailed on the PR notes. |
adangel
left a comment
There was a problem hiding this comment.
Awesome! Looks good to me. I'll merge it tomorrow.
| Back in PMD 5.6.0 we introduced the ability to suppress CPD warnings in Java using comments, by | ||
| including `CPD-OFF` (to start ignoring code), or `CPD-ON` (to resume analysis) during CPD execution. | ||
| This has proved to be much more flexible and versatile than the old annotation-based approach, | ||
| and has since been the prefered way to suppress CPD warnings. |
| This has proved to be much more flexible and versatile than the old annotation-based approach, | ||
| and has since been the prefered way to suppress CPD warnings. | ||
|
|
||
| On this ocassion, we are xtending support for comment-based suppressions to many other languages: |
| .getDefaultVersion().getLanguageVersionHandler(); | ||
| reader = new StringReader(buffer.toString()); | ||
| TokenManager tokenManager = languageVersionHandler | ||
| TokenFilter tokenFilter = new JavaCCTokenFilter(languageVersionHandler |
There was a problem hiding this comment.
I just noticed, that we are using Rhino for parsing javascript, but have an extra es5 javacc grammar for tokenizing. I think, we should streamline this and use both Rhino for parsing and tokenizing. This could be done together with #699. What do you think?
There was a problem hiding this comment.
Sounds reasonable, at the very least for consistency.
TokenFilterinterface in pmd-coreJavaCCTokenFilterto process and filter JavaCC token streams, honoringCPD-OFFandCPD-ONcommentsJavaTokenizerto useJavaCCTokenFilterby extending it and adding custom Java-specific token filtersSupported languages: