Skip to content

[java] Update comment rules for java#2802

Merged
oowekyala merged 26 commits into
pmd:pmd/7.0.xfrom
oowekyala:java-comments
Oct 30, 2020
Merged

[java] Update comment rules for java#2802
oowekyala merged 26 commits into
pmd:pmd/7.0.xfrom
oowekyala:java-comments

Conversation

@oowekyala

@oowekyala oowekyala commented Sep 24, 2020

Copy link
Copy Markdown
Member

Describe the PR

  • Remove the deprecated CommentUtil, AbstractCommentRule
  • There's a new "comment assignment pass", which assigns javadoc comments to the relevant nodes using a data key. The implementation is much more straightforward than before, and is done exactly once, instead of once per AbstractCommentRule (and so, zero times if a ruleset has no AbstractCommentRule). But it fails to detect the comment when there are annotations or modifiers before the comment, which is a rare corner case IMO. We can improve that later
  • The interface JavadocCommentOwner can be used to access them.
  • I wonder, whether FormalComment should be a regular java node, manually inserted in the tree by this pass. We can question that later.
  • This adds a compareLocation method to the Node interface, which this PR was using before I simplified the implementation. So it's unused, but I think this is relevant as many usages of the methods getBeginLine/getEndLine are actually just used to sort some nodes. This may be implemented more efficiently by nodes (eg, jjtree nodes can compare their offsets in the file)
  • Reactivate tests of most rules of the documentation category:
    • UncommentedEmptyMethodBody
    • CommentRequired
    • CommentSize
    • CommentContent

Related issues

Ready?

  • Added unit tests for fixed bug/feature
  • Passing all unit tests
  • Complete build ./mvnw clean verify passes (checked automatically by travis)
  • Added (in-code) documentation (if needed)

@oowekyala oowekyala added this to the 7.0.0 milestone Sep 24, 2020
@ghost

ghost commented Sep 24, 2020

Copy link
Copy Markdown
2 Messages
📖 Compared to pmd/7.0.x:
This changeset introduces 83134 new violations, 4 new errors and 0 new configuration errors,
removes 41 violations, 1 errors and 0 configuration errors.
Full report
📖 Compared to master:
This changeset introduces 927 new violations, 10 new errors and 0 new configuration errors,
removes 1072 violations, 16 errors and 2 configuration errors.
Full report
Compared to pmd/7.0.x:
This changeset introduces 63083 new violations, 3 new errors and 0 new configuration errors,
removes 17 violations, 0 errors and 0 configuration errors.
Full report
Compared to master:
This changeset introduces 4862 new violations, 9 new errors and 0 new configuration errors,
removes 24436 violations, 16 errors and 2 configuration errors.
Full report
Compared to pmd/7.0.x:
This changeset introduces 59624 new violations, 3 new errors and 0 new configuration errors,
removes 35 violations, 0 errors and 0 configuration errors.
Full report
Compared to master:
This changeset introduces 61474 new violations, 9 new errors and 0 new configuration errors,
removes 84585 violations, 16 errors and 2 configuration errors.
Full report
Compared to pmd/7.0.x:
This changeset introduces 63066 new violations, 3 new errors and 0 new configuration errors,
removes 0 violations, 0 errors and 0 configuration errors.
Full report
Compared to master:
This changeset introduces 6661 new violations, 9 new errors and 0 new configuration errors,
removes 26763 violations, 16 errors and 2 configuration errors.
Full report
No java rules are changed!

Generated by 🚫 Danger

@oowekyala oowekyala marked this pull request as ready for review September 28, 2020 16:19
@oowekyala oowekyala marked this pull request as draft September 30, 2020 10:30
@oowekyala oowekyala marked this pull request as ready for review September 30, 2020 15:27
@oowekyala

Copy link
Copy Markdown
Member Author

@adangel Apparently danger fails on the latest commits because of a missing master baseline zip:

https://travis-ci.org/github/pmd/pmd/jobs/736633219#L7492-L7502

@oowekyala

Copy link
Copy Markdown
Member Author

Actually the diff is weird because the file names are truncated in the PR diff?

Eg this section has just the file name and not its full path

https://chunk.io/pmd/038abe34ece14f87822db43bb27fd398/diff2/checkstyle/index.html#A14249

Compare to the baseline for the same file

https://chunk.io/pmd/038abe34ece14f87822db43bb27fd398/diff2/checkstyle/index.html#A1

@adangel

adangel commented Oct 22, 2020

Copy link
Copy Markdown
Member

@adangel Apparently danger fails on the latest commits because of a missing master baseline zip:

https://travis-ci.org/github/pmd/pmd/jobs/736633219#L7492-L7502

I guess, this was a temporary hickup downloading the baseline from sourceforge. The wget command didn't fail (if it did, we would have seen a error message there already), it downloaded something. But maybe the file was not complete or corrupt (due to mirroring on sourceforge hosts?).

I see, that the regression tester runs again...

@adangel

adangel commented Oct 22, 2020

Copy link
Copy Markdown
Member

Actually the diff is weird because the file names are truncated in the PR diff?

Only the first part of the path should be truncated (e.g. the working directory), so that it doesn't matter, where exactly you run the regression tester.

The baseline looks Ok: checkstyle/src/it/java/com/google/checkstyle/test/base/AbstractIndentationTestSupport.java
But pmd7 doesn't: AbstractIndentationTestSupport.java

I feel like I remember having fixed this already.... hm.... I think, this was the path of processing errors, which was wrong (269f5bd). Maybe that's something new? I wonder, from where it came....

Looking at the comparison between this PR and pmd/7 - there are only trucated files... there is something broken on pmd/7.0.x....

@adangel

adangel commented Oct 23, 2020

Copy link
Copy Markdown
Member

@oowekyala The new regression report is available. We have many removed violations compared to master:

Compared to master:
This changeset introduces 4862 new violations, 9 new errors and 0 new configuration errors,
removes 24436 violations, 16 errors and 2 configuration errors.
https://chunk.io/pmd/e39b418f45cf42b683c24592dd69dc28/diff2/index.html

One example is this:
In OuterTypeFilenameTest.java we should report the missing comment for the 3 test methods, but we apparently don't. The first method is annotated with @Override, so we don't require comment for "getPackageLocation()", which is Ok. Maybe that leads to skipping the check for the other methods?

I'll look into this....

@adangel

adangel commented Oct 23, 2020

Copy link
Copy Markdown
Member

Ok, so this is what happens:

We see the annotation @Test, figure out, it is @org.junit.Test, but junit is not on the auxclasspath. So, this classref is unresolved. When comparing to @java.lang.Override, we consider it by default a subtype:

} else if (hasUnresolvedSymbol(t)) {
// This also considers types with an unresolved symbol
// subtypes of (nearly) anything. This allows them to
// pass bound checks on type variables.
return Convertibility.subtypeIf(s instanceof JClassType); // excludes array or so

This results then in a false negative. It seems, that we have on master a different default behavior for incomplete auxclasspaths. Returning here simply Convertibility.NEVER would solve this special case, but maybe it's not too bad, and we should leave it. It would give us an argument: "you need to call PMD correctly to get accurate results".

I'll continue now on pmd/pmd-regression-tester#48, so that we call PMD correctly during regression testing.

@adangel

adangel commented Oct 23, 2020

Copy link
Copy Markdown
Member

Updated numbers for comparing this PR against master for checkstyle is available: https://chunk.pmd-code.org/pmd/f465ddd3-2188-4976-9f61-f16591faea2c/diff/index.html
The numbers look much better now....

  • 1076 new violations, 420 removed violations
  • 7 new errors, 21 removed errors

@adangel

adangel commented Oct 23, 2020

Copy link
Copy Markdown
Member

Here's the preview for spring-framework comparing this PR against master: https://chunk.pmd-code.org/pmd/37f43255-f654-4295-a72b-76e83ef2a06e/diff/index.html

The numbers look much better here as well:

  • 4296 new violations, 3458 removed violations
  • 3 new errors, 22 removed errors

@oowekyala

Copy link
Copy Markdown
Member Author

This results then in a false negative. It seems, that we have on master a different default behavior for incomplete auxclasspaths. Returning here simply Convertibility.NEVER would solve this special case, but maybe it's not too bad, and we should leave it. It would give us an argument: "you need to call PMD correctly to get accurate results".

Thanks for diagnosing this. I think isConvertible needs to stay that way for type inference to work properly (ie, be liberal when inferring unresolved types). But I changed TypeTestUtil to use a better strategy (ie, always return false if we're not sure, which is the behavior of the rest of TypeTestUtil)

@oowekyala

oowekyala commented Oct 29, 2020

Copy link
Copy Markdown
Member Author

Quick review of the regression report:

This fixes many FNs of CommentRequired

Line numbers changed a bit, eg this violation is reported on the method identifier and not the return type, this is expected.

A few files are shown as if all violations are removed: those are the files excluded by the exclude pattern, refs 331f8f9

But notably, CommentRequired still does not report on annotation type declarations, and record declarations.

I think globally this is very positive, this new comment assignment method works better than the old.

@oowekyala oowekyala self-assigned this Oct 30, 2020
@oowekyala oowekyala merged commit 9096ef8 into pmd:pmd/7.0.x Oct 30, 2020
@oowekyala oowekyala deleted the java-comments branch October 30, 2020 19:15
@oowekyala oowekyala mentioned this pull request Nov 10, 2020
64 tasks
@adangel adangel mentioned this pull request Nov 25, 2020
4 tasks
@adangel adangel mentioned this pull request Jan 23, 2023
55 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants