Issue #13830: Support Java 21 string template syntax#13991
Conversation
| int y = 20; | ||
|
|
||
| // most basic example, empty string template expression | ||
| String result1 = STR.""; |
There was a problem hiding this comment.
|--VARIABLE_DEF -> VARIABLE_DEF [9:4]
| |--MODIFIERS -> MODIFIERS [9:4]
| |--TYPE -> TYPE [9:4]
| | `--IDENT -> String [9:4]
| |--IDENT -> result1 [9:11]
| |--ASSIGN -> = [9:19]
| | `--EXPR -> EXPR [9:24]
| | `--DOT -> . [9:24]
| | |--IDENT -> STR [9:21]
| | `--STRING_LITERAL -> "" [9:25]
| `--SEMI -> ; [9:27]
There was a problem hiding this comment.
I suppose it is still a template, we should do https://github.com/checkstyle/checkstyle/pull/13991/files#r1421784134, I will update the code.
There was a problem hiding this comment.
| |--ASSIGN -> = [9:19]
| | `--EXPR -> EXPR [9:24]
| | `--DOT -> . [9:24]
| | |--IDENT -> STR [9:21]
| | `--STRING_LITERAL -> "" [9:25]
is consistent with what we do now.
| | `--DOT -> . [15:24]
| | |--IDENT -> STR [15:21]
| | `--STRING_TEMPLATE_BEGIN -> "
| | |--STRING_TEMPLATE_CONTENT ->
| | `--STRING_TEMPLATE_END -> "
is to clear pointer it is String Template syntax.
if we reuse old tree, we might immediately need to update all our Checks to exclude this from consideration. and it will be hard to find by AST that it is String Template.
I vote for STRING_TEMPLATE_BEGIN approach.
| String result1 = STR.""; | ||
|
|
||
| // simple single value interpolation | ||
| String result2 = STR. "x\{ x }x"; |
There was a problem hiding this comment.
|--VARIABLE_DEF -> VARIABLE_DEF [12:4]
| |--MODIFIERS -> MODIFIERS [12:4]
| |--TYPE -> TYPE [12:4]
| | `--IDENT -> String [12:4]
| |--IDENT -> result2 [12:11]
| |--ASSIGN -> = [12:19]
| | `--EXPR -> EXPR [12:24]
| | `--DOT -> . [12:24]
| | |--IDENT -> STR [12:21]
| | `--STRING_TEMPLATE_BEGIN -> "x\{ [12:26]
| | |--EMBEDDED_EXPRESSION -> EMBEDDED_EXPRESSION [12:31]
| | | `--IDENT -> x [12:31]
| | `--STRING_TEMPLATE_END -> }x" [12:33]
| `--SEMI -> ; [12:36]
There was a problem hiding this comment.
Need to add a case with no embedded expression (like no x here)
There was a problem hiding this comment.
in my imagination, it bit more trrucrured:
`--STRING_TEMPLATE_BEGIN -> " [12:26]
|--TEMPLATE_CONTENT -> x
|--EMBEDED_BEGIN -> "\{
|--EMBEDDED_EXPRESSION -> EMBEDDED_EXPRESSION [12:31]
| |--IDENT -> x [12:31]
|--EMBEDED_END -> }
|--TEMPLATE_CONTENT -> x [12:33]
`--STRING_TEMPLATE_END -> " [12:33]
proposed tree is not strict, just hint to have TEMPLATE_CONTENT and EMBEDED_BEGIN/END
There was a problem hiding this comment.
I also prefer your proposal, however this will be a bit more challenging to create. Let me see what I can do. However, the current implementation is more reflective of the actual java grammar.
There was a problem hiding this comment.
Ok:
STRING_TEMPLATE_BEGIN -> " [12:26]
|--STRING_TEMPLATE_CONTENT -> x [12:27]
|--EMBEDDED_EXPRESSION_BEGIN -> \{ [12:28]
|--EMBEDDED_EXPRESSION -> EMBEDDED_EXPRESSION [12:31]
| `--IDENT -> x [12:31]
|--EMBEDDED_EXPRESSION_END -> } [12:33]
|--STRING_TEMPLATE_CONTENT -> x [12:34]
`--STRING_TEMPLATE_END -> " [12:35]
There was a problem hiding this comment.
All other examples have been edited, this is the new AST for this code (it has moved down in the source code file):
|--VARIABLE_DEF -> VARIABLE_DEF [15:4]
| |--MODIFIERS -> MODIFIERS [15:4]
| |--TYPE -> TYPE [15:4]
| | `--IDENT -> String [15:4]
| |--IDENT -> result2 [15:11]
| |--ASSIGN -> = [15:19]
| | `--EXPR -> EXPR [15:24]
| | `--DOT -> . [15:24]
| | |--IDENT -> STR [15:21]
| | `--STRING_TEMPLATE_BEGIN -> " [15:26]
| | |--STRING_TEMPLATE_CONTENT -> x [15:27]
| | |--EMBEDDED_EXPRESSION_BEGIN -> \{ [15:28]
| | |--EMBEDDED_EXPRESSION -> EMBEDDED_EXPRESSION [15:31]
| | | `--IDENT -> x [15:31]
| | |--EMBEDDED_EXPRESSION_END -> } [15:33]
| | |--STRING_TEMPLATE_CONTENT -> x [15:34]
| | `--STRING_TEMPLATE_END -> " [15:35]
| `--SEMI -> ; [15:36]
| String result2 = STR. "x\{ x }x"; | ||
|
|
||
| // simple single value interpolation with a method call | ||
| String result3 = STR. "\{ y(y("y")) }"; |
There was a problem hiding this comment.
|--VARIABLE_DEF -> VARIABLE_DEF [18:4]
| |--MODIFIERS -> MODIFIERS [18:4]
| |--TYPE -> TYPE [18:4]
| | `--IDENT -> String [18:4]
| |--IDENT -> result3 [18:11]
| |--ASSIGN -> = [18:19]
| | `--EXPR -> EXPR [18:24]
| | `--DOT -> . [18:24]
| | |--IDENT -> STR [18:21]
| | `--STRING_TEMPLATE_BEGIN -> " [18:26]
| | |--STRING_TEMPLATE_CONTENT -> [18:27]
| | |--EMBEDDED_EXPRESSION_BEGIN -> \{ [18:27]
| | |--EMBEDDED_EXPRESSION -> EMBEDDED_EXPRESSION [18:31]
| | | `--METHOD_CALL -> ( [18:31]
| | | |--IDENT -> y [18:30]
| | | |--ELIST -> ELIST [18:33]
| | | | `--EXPR -> EXPR [18:33]
| | | | `--METHOD_CALL -> ( [18:33]
| | | | |--IDENT -> y [18:32]
| | | | |--ELIST -> ELIST [18:34]
| | | | | `--EXPR -> EXPR [18:34]
| | | | | `--STRING_LITERAL -> "y" [18:34]
| | | | `--RPAREN -> ) [18:37]
| | | `--RPAREN -> ) [18:38]
| | |--EMBEDDED_EXPRESSION_END -> } [18:40]
| | |--STRING_TEMPLATE_CONTENT -> [18:41]
| | `--STRING_TEMPLATE_END -> " [18:41]
| `--SEMI -> ; [18:42]
| String result3 = STR. "\{ y(y("y")) }"; | ||
|
|
||
| // bit more complex example, multiple interpolations | ||
| String result4 = STR. "\{ x }" + STR. "\{ y }"; |
There was a problem hiding this comment.
|--VARIABLE_DEF -> VARIABLE_DEF [21:4]
| |--MODIFIERS -> MODIFIERS [21:4]
| |--TYPE -> TYPE [21:4]
| | `--IDENT -> String [21:4]
| |--IDENT -> result4 [21:11]
| |--ASSIGN -> = [21:19]
| | `--EXPR -> EXPR [21:36]
| | `--PLUS -> + [21:36]
| | |--DOT -> . [21:24]
| | | |--IDENT -> STR [21:21]
| | | `--STRING_TEMPLATE_BEGIN -> " [21:26]
| | | |--STRING_TEMPLATE_CONTENT -> [21:27]
| | | |--EMBEDDED_EXPRESSION_BEGIN -> \{ [21:27]
| | | |--EMBEDDED_EXPRESSION -> EMBEDDED_EXPRESSION [21:30]
| | | | `--IDENT -> x [21:30]
| | | |--EMBEDDED_EXPRESSION_END -> } [21:32]
| | | |--STRING_TEMPLATE_CONTENT -> [21:33]
| | | `--STRING_TEMPLATE_END -> " [21:33]
| | `--DOT -> . [21:41]
| | |--IDENT -> STR [21:38]
| | `--STRING_TEMPLATE_BEGIN -> " [21:43]
| | |--STRING_TEMPLATE_CONTENT -> [21:44]
| | |--EMBEDDED_EXPRESSION_BEGIN -> \{ [21:44]
| | |--EMBEDDED_EXPRESSION -> EMBEDDED_EXPRESSION [21:47]
| | | `--IDENT -> y [21:47]
| | |--EMBEDDED_EXPRESSION_END -> } [21:49]
| | |--STRING_TEMPLATE_CONTENT -> [21:50]
| | `--STRING_TEMPLATE_END -> " [21:50]
| `--SEMI -> ; [21:51]
| String result10 = STR. "hey \{ | ||
| y(y("y")) + y(y("y")) | ||
| } there \{ | ||
| y(y("y")) + y(y("y")) | ||
| + STR."x\{ x }x" | ||
| }"; |
There was a problem hiding this comment.
|--VARIABLE_DEF -> VARIABLE_DEF [41:4]
| |--MODIFIERS -> MODIFIERS [41:4]
| |--TYPE -> TYPE [41:4]
| | `--IDENT -> String [41:4]
| |--IDENT -> result10 [41:11]
| |--ASSIGN -> = [41:20]
| | `--EXPR -> EXPR [41:25]
| | `--DOT -> . [41:25]
| | |--IDENT -> STR [41:22]
| | `--STRING_TEMPLATE_BEGIN -> " [41:27]
| | |--STRING_TEMPLATE_CONTENT -> hey [41:28]
| | |--EMBEDDED_EXPRESSION_BEGIN -> \{ [41:32]
| | |--EMBEDDED_EXPRESSION -> EMBEDDED_EXPRESSION [42:22]
| | | `--PLUS -> + [42:22]
| | | |--METHOD_CALL -> ( [42:13]
| | | | |--IDENT -> y [42:12]
| | | | |--ELIST -> ELIST [42:15]
| | | | | `--EXPR -> EXPR [42:15]
| | | | | `--METHOD_CALL -> ( [42:15]
| | | | | |--IDENT -> y [42:14]
| | | | | |--ELIST -> ELIST [42:16]
| | | | | | `--EXPR -> EXPR [42:16]
| | | | | | `--STRING_LITERAL -> "y" [42:16]
| | | | | `--RPAREN -> ) [42:19]
| | | | `--RPAREN -> ) [42:20]
| | | `--METHOD_CALL -> ( [42:25]
| | | |--IDENT -> y [42:24]
| | | |--ELIST -> ELIST [42:27]
| | | | `--EXPR -> EXPR [42:27]
| | | | `--METHOD_CALL -> ( [42:27]
| | | | |--IDENT -> y [42:26]
| | | | |--ELIST -> ELIST [42:28]
| | | | | `--EXPR -> EXPR [42:28]
| | | | | `--STRING_LITERAL -> "y" [42:28]
| | | | `--RPAREN -> ) [42:31]
| | | `--RPAREN -> ) [42:32]
| | |--EMBEDDED_EXPRESSION_END -> } [43:12]
| | |--STRING_TEMPLATE_CONTENT -> there [43:13]
| | |--EMBEDDED_EXPRESSION_BEGIN -> \{ [43:20]
| | |--EMBEDDED_EXPRESSION -> EMBEDDED_EXPRESSION [45:12]
| | | `--PLUS -> + [45:12]
| | | |--PLUS -> + [44:22]
| | | | |--METHOD_CALL -> ( [44:13]
| | | | | |--IDENT -> y [44:12]
| | | | | |--ELIST -> ELIST [44:15]
| | | | | | `--EXPR -> EXPR [44:15]
| | | | | | `--METHOD_CALL -> ( [44:15]
| | | | | | |--IDENT -> y [44:14]
| | | | | | |--ELIST -> ELIST [44:16]
| | | | | | | `--EXPR -> EXPR [44:16]
| | | | | | | `--STRING_LITERAL -> "y" [44:16]
| | | | | | `--RPAREN -> ) [44:19]
| | | | | `--RPAREN -> ) [44:20]
| | | | `--METHOD_CALL -> ( [44:25]
| | | | |--IDENT -> y [44:24]
| | | | |--ELIST -> ELIST [44:27]
| | | | | `--EXPR -> EXPR [44:27]
| | | | | `--METHOD_CALL -> ( [44:27]
| | | | | |--IDENT -> y [44:26]
| | | | | |--ELIST -> ELIST [44:28]
| | | | | | `--EXPR -> EXPR [44:28]
| | | | | | `--STRING_LITERAL -> "y" [44:28]
| | | | | `--RPAREN -> ) [44:31]
| | | | `--RPAREN -> ) [44:32]
| | | `--DOT -> . [45:17]
| | | |--IDENT -> STR [45:14]
| | | `--STRING_TEMPLATE_BEGIN -> " [45:18]
| | | |--STRING_TEMPLATE_CONTENT -> x [45:19]
| | | |--EMBEDDED_EXPRESSION_BEGIN -> \{ [45:20]
| | | |--EMBEDDED_EXPRESSION -> EMBEDDED_EXPRESSION [45:23]
| | | | `--IDENT -> x [45:23]
| | | |--EMBEDDED_EXPRESSION_END -> } [45:25]
| | | |--STRING_TEMPLATE_CONTENT -> x [45:26]
| | | `--STRING_TEMPLATE_END -> " [45:27]
| | |--EMBEDDED_EXPRESSION_END -> } [46:12]
| | |--STRING_TEMPLATE_CONTENT -> [46:13]
| | `--STRING_TEMPLATE_END -> " [46:13]
| `--SEMI -> ; [46:14]
|
Need to add Java 21 compilation job to CI |
|
Hello there, |
Hi @nandorholozsnyak , grammar changes are rarely trivial, and require quite a bit of maintainer time. We can certainly cut a release as soon as this is merged if you would like, we do not need to wait for normal release cycle. |
Hello, Take your time and make sure it works properly :) |
|
@nandorholozsnyak , if you have time to contribute, please help us with #12542 , and people will not be blocked by new grammar usage any more . Main point of PR to be merged, release can be done at any time, to ship to users sooner. |
Hey @romani |
d642cb9 to
c0ed0ef
Compare
c0ed0ef to
046a790
Compare
046a790 to
823886a
Compare
9079ffe to
a8faca4
Compare
Let's move it to separate PR and merge sooner. I saw somewhere another PR that try to do same. |
4c028d0 to
5de63cc
Compare
93d5480 to
3319a62
Compare
|
why two commits with same message? |
3319a62 to
fcb2410
Compare
This was a mistake, fixed. |
|
Checker is not green. |
|
@nrmancuso I'll look over again soon, but I saw no response to my comment at #13991 (comment) . |
I can create an issue for this; in the past we have only fixed NPEs in grammar updates |
fcb2410 to
7600cf0
Compare
7600cf0 to
dbc54ca
Compare
| "NUM_DOUBLE", "NUM_FLOAT", "NUM_INT", "NUM_LONG", "IDENT", | ||
| "COMMENT_CONTENT", "STRING_LITERAL", "CHAR_LITERAL", "TEXT_BLOCK_CONTENT") | ||
| "COMMENT_CONTENT", "STRING_LITERAL", "CHAR_LITERAL", "TEXT_BLOCK_CONTENT", | ||
| "STRING_TEMPLATE_CONTENT") |
There was a problem hiding this comment.
This update is strange.
https://checkstyle.org/styleguides/google-java-style-20180523/javaguide.html#s2.3.2-special-escape-sequences
https://checkstyle.org/google_style.html#a2.3.2
Why we ignoring such tokens, as they can have content that we need to validate ?
There was a problem hiding this comment.
We can definitely improve this in separate issue, just curious
There was a problem hiding this comment.
I do not remember reasoning, but we have added all new syntax to this collection. Maybe it was due to not being explicitly mentioned in Googles style guide.
| * at the beginning of a string template. | ||
| * <p>For example:</p> | ||
| * <pre> | ||
| * String s = STR."Hello, \{name}!"; |
There was a problem hiding this comment.
String s = STR."Hello, \{firstName + lastName}!";
| * at the end of a string template. | ||
| * <p>For example:</p> | ||
| * <pre> | ||
| * String s = STR."Hello, \{name}!"; |
There was a problem hiding this comment.
String s = STR."Hello, \{firstName + lastName}!";
| * An expression embedded within a string template. | ||
| * <p>For example:</p> | ||
| * <pre> | ||
| * String s = STR."Hello, \{name}!"; |
There was a problem hiding this comment.
String s = STR."Hello, \{getName("Mr. ")}!";
| * template. | ||
| * <p>For example:</p> | ||
| * <pre> | ||
| * String s = STR."Hello, \{name}!"; |
There was a problem hiding this comment.
String s = STR."Hello, \{getName("Mr. ")}!";
There was a problem hiding this comment.
Changed these a bit to reuse the firstName, lastName concept but with a method call as suggested.
| * template may have more than one node of this type. | ||
| * <p>For example:</p> | ||
| * <pre> | ||
| * String s = STR."Hello, \{name}!"; |
There was a problem hiding this comment.
String s = STR."Hello, \{firstName + lastName}!";
| * The opening delimiter of an embedded expression within a string template. | ||
| * <p>For example:</p> | ||
| * <pre> | ||
| * String s = STR."Hello, \{name}!"; |
There was a problem hiding this comment.
String s = STR."Hello, \{getName("Mr. ")}!";
dbc54ca to
8cfb314
Compare
|
@nrmancuso Is there any news for the text block template? |
|
@NoSugarCoffee news is here: #14805 |
|
@nrmancuso It is clear and make sense,thanks |
Closes #13830
New grammar summary (from here):
Reading:
Attention: mutliline strings delimited with only single
"are possible.Link to reports: nrmancuso/nrmancuso.github.io#56 (comment)
Diff Regression projects: https://raw.githubusercontent.com/checkstyle/contribution/d1348582c72049059c64d46601ce6aaca45baae2/checkstyle-tester/projects-to-test-on-for-github-action.properties
Diff Regression config: https://gist.githubusercontent.com/nrmancuso/4e014025168093baa3f61986a32a96b7/raw/294f634d72297b7b318de79788ca2b153ab33bd2/base_config.xml
Diff Regression patch config: https://gist.githubusercontent.com/nrmancuso/fa9d105fd211d31124cb852aaac54f73/raw/fc90a8f543660dca8ea8acb4afe03e3eee75c76e/patch_config.xml
Projects file: projects-to-test-on.properties
Patch branch: issue-13830
https://checkstyle-reports.s3.us-east-2.amazonaws.com/reports/issue-13830/2023-12-28-T-16-11-57/sevntu-check-regression_part_1/index.html
https://checkstyle-reports.s3.us-east-2.amazonaws.com/reports/issue-13830/2023-12-28-T-16-11-57/part4/index.html
https://checkstyle-reports.s3.us-east-2.amazonaws.com/reports/issue-13830/2023-12-28-T-16-11-57/part1/index.html
https://checkstyle-reports.s3.us-east-2.amazonaws.com/reports/issue-13830/2023-12-28-T-16-11-57/part5/index.html
https://checkstyle-reports.s3.us-east-2.amazonaws.com/reports/issue-13830/2023-12-28-T-16-11-57/sevntu-check-regression_part_2/index.html
https://checkstyle-reports.s3.us-east-2.amazonaws.com/reports/issue-13830/2023-12-28-T-16-11-57/part6/index.html
https://checkstyle-reports.s3.us-east-2.amazonaws.com/reports/issue-13830/2023-12-28-T-16-11-57/part2/index.html
https://checkstyle-reports.s3.us-east-2.amazonaws.com/reports/issue-13830/2023-12-28-T-16-11-57/part3/index.html
https://checkstyle-reports.s3.us-east-2.amazonaws.com/reports/issue-13830/2023-12-28-T-16-11-57/antlr-report/index.html
All differences are either new violations in files that we were not able to parse previously, or change in exception lines in noncompilable code.