Skip to content

[java] Added 4 performance rules originating from PMD-jPinpoint-rules#1932

Closed
jborgers wants to merge 3 commits into
pmd:masterfrom
jborgers:master
Closed

[java] Added 4 performance rules originating from PMD-jPinpoint-rules#1932
jborgers wants to merge 3 commits into
pmd:masterfrom
jborgers:master

Conversation

@jborgers

Copy link
Copy Markdown
Contributor

Happy to add the following Java performance rules:

  • AvoidApacheCommonsFileItemNonStreaming
  • AvoidCalendarDateCreation
  • AvoidConcatInAppend and
  • AvoidConcatInLoop.

Including unit tests and examples. Originating from github.com/jborgers/PMD-jPinpoint-rules.

  • Tests and build run without problems.

PR Description:
This is the first set of rules from PMD-jPinpoint-rules, sponsored by Rabobank. I discussed the rules and PR with Juan Martín Sotuyo Dodero.

  • There is overlap with other rules for the concatenation rules. I am not sure how much overlap.
  • I am not sure about the severity of the rules, I left them at the level we use them.
  • We want to provide more extensive documentation in the future, and link to it from the rules doc.

Regards, Jeroen

…oidCalendarDateCreation, AvoidConcatInAppend and AvoidConcatInLoop.

Including unit tests and examples. Originating from github.com/jborgers/PMD-jPinpoint-rules.
Links to more doc later.
@ghost

ghost commented Jul 22, 2019

Copy link
Copy Markdown
1 Message
📖 This changeset introduces 168 new violations, 0 new errors and 0 new configuration errors,
removes 0 violations, 2 errors and 2 configuration errors.
Full report
This changeset introduces 168 new violations, 0 new errors and 0 new configuration errors,
removes 0 violations, 2 errors and 2 configuration errors.
Full report
This changeset introduces 154 new violations, 0 new errors and 0 new configuration errors,
removes 0 violations, 2 errors and 2 configuration errors.
Full report

Generated by 🚫 Danger

@oowekyala oowekyala left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! I think some things can be simplified/ are debatable

Also, please note for the future:

  • the documentation is generated by the CI build - you don't need to generate it manually. Though now it's there, it can stay
  • it would be nice to make a separate PR for each rule next time

Comment thread pmd-java/src/main/resources/category/java/performance.xml Outdated
</example>
</rule>

<rule name="AvoidConcatInAppend"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't that exactly what InefficientStringBuffering is meant to be?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possibly. We don't have that rule in the subset we use, probably because we didn't update that selection for some time. There is certainly overlap, not sure if they are exactly the same.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On a separate PR for each rule: Juan asked me to create a PR per batch, a couple of rules together. Not sure of best choice here.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They appear to have the same intention, but there are some differences in their implementation:

For example

  • your rule flags wrappedLine.append(str + "c", a, c); correctly whereas the other rule gives up
  • your rule flag sb.append(1 + 2), which is not a concatenation, so that's a false positive
  • your rule considers concatenation with a non-static final field as a constant expression, which it isn't
  • the other rule doesn't flag any cases of new StringBuilder().append(...), ie use of append directly on the constructor invocation, whereas yours does it correctly

I think we should fix the false negatives of the existing rule but not merge this new rule.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually this rule has many more problems... It uses too many // tests, so that you end up testing unrelated nodes. Consider this violation where the rule sees an append, a string concatenation somewhere in another statement (the throw statement) and reports on a common ancestor (the for statement), even though the concatenation and the append are unrelated. All the violations I've seen on the spring framework in the report are false positives either of this kind, or where there's no concatenation but just numeric addition.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, we should enhance the existing rule InefficientStringBuffering instead of adding this new rule.

FWIW: I've noticed, that the found violation don't highlight the append - at least for the 3 violations found in checkstyle: I wouldn't know, what to fix, if I get these violations. We should aim to report exactly the node which contains the concatenation. Maybe the reason for this is the //, that Clément mentioned?

Comment thread pmd-java/src/main/resources/category/java/performance.xml

@oowekyala oowekyala left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks again for your work on this. I've taken a look at the CI report though, and there are many false positives. Please take a look at the following comments

About using one PR per rule, I think this would have made it easier to review rules separately, and to merge incrementally. While the rule about FileItems and AvoidConcatInLoops are not far from ready, AvoidConcatInAppend should IMO not be merged, and changes to CalendarDateCreation may delay the other rules. So I think separate PRs would have been a better choice

class="net.sourceforge.pmd.lang.rule.XPathRule"
typeResolution="true"
externalInfoUrl="${pmd.website.baseurl}/pmd_rules_java_performance.html#avoidcalendardatecreation">
<description>Problem: A Calendar is a heavyweight object and expensive to create. &#13;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The assumption of the rule doesn't hold if the calendar is not created just to call this method, or if it is eg passed as a parameter and not created locally. Consider for example this violation in the Spring Framework, which looks like a false positive.

</example>
</rule>

<rule name="AvoidConcatInAppend"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually this rule has many more problems... It uses too many // tests, so that you end up testing unrelated nodes. Consider this violation where the rule sees an append, a string concatenation somewhere in another statement (the throw statement) and reports on a common ancestor (the for statement), even though the concatenation and the append are unrelated. All the violations I've seen on the spring framework in the report are false positives either of this kind, or where there's no concatenation but just numeric addition.

<property name="version" value="2.0"/>
<property name="xpath">
<value><![CDATA[
(//ForStatement | //WhileStatement | //DoStatement)//AssignmentOperator[

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rule should be checking that the variable being concatenated is defined outside the loop.
Consider this violation:

      for (String fileExtension : fileExtensions) {
				if (fileExtension.charAt(0) != '.') {
					fileExtension = "." + fileExtension; // here
				}
				this.fileExtensions.add(fileExtension);
			}

The concatenation is local to each iteration and replacing it with a StringBuilder would only be noise.

</example>
</rule>

<rule name="AvoidApacheCommonsFileItemNonStreaming"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe a better name would be UseIOStreamsWithApacheCommonsFileItem? IMHO "streaming" immediately hints at the Stream API and not IO streams, so I think it would be clearer

externalInfoUrl="${pmd.website.baseurl}/pmd_rules_java_performance.html#avoidapachecommonsfileitemnonstreaming">
<description>
Problem: Use of FileItem.get and FileItem.getString could exhaust memory since they load the entire file into memory&#13;
Solution: Use streaming methods and buffering.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you mention getInputStream() explicitly in the description? Like for the name of the rule, the "streaming methods" here sounds unclear to me

class="net.sourceforge.pmd.lang.rule.XPathRule"
typeResolution="true"
externalInfoUrl="${pmd.website.baseurl}/pmd_rules_java_performance.html#avoidconcatinappend">
<description>Concatenation of Strings is used inside an StringBuilder.append argument. Problem: Each statement with one or more +-operators creates a hidden temporary StringBuilder, a char[] and a new String object, which all have to be garbage collected.&#13;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: Just use markdown syntax for the description. No need to use &#13; for a new line, just add a empty line. We'll parse the text here (including the line breaks) and use it in our markdown-based rule documentation.
In order to easily read this on github, also consider breaking the lines after 120 characters.

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://pmd.sourceforge.net/rule-tests http://pmd.sourceforge.net/rule-tests_1_0_0.xsd">
<test-code>
<description>Violation: Avoid contact in append method invocations</description>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo: contact -> concat

@adangel adangel added a:new-rule Proposal to add a new built-in rule is:WIP For PRs that are not fully ready, or issues that are actively being tackled labels Jul 27, 2019

private String good(String arg) {
StringBuilder sb = new StringBuilder();
sb.append("arg='").append(arg).append("'");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The better one:

StringBuilder sb = new StringBuilder("arg='").append(arg).append("'");

Perhaps, it's better to use best practices or at least mention that this example isn't the best.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, we should change the sample code. Note: performance.md is generated, the actual change has to happen in performance.xml.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: performance.md is generated, the actual change has to happen in performance.xml.

Is it something that can be automated with Danger? :)

String log = "";
List<String> values = Arrays.asList("tic ", "tac ", "toe ");
for (String val : values) {
log += val;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is string concatenation is still an issue nowadays (2019) ? I've heard that there were optimizations for that in JDK9, for example.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is indeed some optimization done with Java 9:

Before Java9, inside the loop, the code generated was basically:

StringBuilder temp = new StringBuilder();
log = temp.append(log).append(val).toString();

Since Java9, it looks more like (roughly):

log = StringConcatFactory.makeConcatWithConstants(..., log, val);

So, the rule is definitely less relevant for java9+.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's worth noting, that this specific example would be unaffected by Java 9+ optimizations, as the concat is done within a loop

adangel added a commit to adangel/pmd that referenced this pull request Jun 13, 2020
adangel added a commit to adangel/pmd that referenced this pull request Jun 13, 2020
adangel added a commit to adangel/pmd that referenced this pull request Jun 13, 2020
adangel added a commit to adangel/pmd that referenced this pull request Jun 14, 2020
Originally from pmd#1932

Author: jborgers <jborgers@jpinpoint.com>
@adangel adangel closed this Jun 14, 2020
adangel added a commit to adangel/pmd that referenced this pull request Jun 18, 2020
And also fix false positive when concatenation is local to each
for-loop iteration.

Test cases originally from pmd#1932

Author: jborgers <jborgers@jpinpoint.com>
adangel added a commit to adangel/pmd that referenced this pull request Jun 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

a:new-rule Proposal to add a new built-in rule is:WIP For PRs that are not fully ready, or issues that are actively being tackled

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants