[`macro_metavar_expr_concat`] Add support for literals by c410-f3r · Pull Request #126841 · rust-lang/rust

c410-f3r · 2024-06-22T21:11:27Z

Adds support for things like ${concat($variable, 123)} or ${concat("hello", "_world")} .

rustbot · 2024-06-22T21:11:35Z

rustbot has assigned @petrochenkov.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

compiler/rustc_expand/src/mbe/metavar_expr.rs

petrochenkov · 2024-06-24T12:47:30Z

concat_idents! doesn't currently support literal, but the motivation here is pretty clear, you want to combine ident and 0 into ident0, but 0 is not an identifier lexically.

However, restricting literals to simple string literals "abc" (no raws, no suffixes) would be enough for that.
All possible identifier parts should fit into that.

petrochenkov · 2024-06-24T12:53:09Z

Also, is there any validation for the concatenation result, like in https://doc.rust-lang.org/stable/proc_macro/struct.Ident.html#method.new ?

When concatenating arbitrary strings you can get something that is

not lexically an identifier
not NFC normalized (https://doc.rust-lang.org/reference/identifiers.html?highlight=nfc#normalization)

Both lexer and proc macro interface prevent creation of such ill-formed identifiers.

c410-f3r · 2024-06-24T13:43:03Z

Also, is there any validation for the concatenation result, like in https://doc.rust-lang.org/stable/proc_macro/struct.Ident.html#method.new ?

When concatenating arbitrary strings you can get something that is
* not lexically an identifier

* not NFC normalized (https://doc.rust-lang.org/reference/identifiers.html?highlight=nfc#normalization)
Both lexer and proc macro interface prevent creation of such ill-formed identifiers.

Yeah, this is bad :|

The current code allows the creation of invalid identifiers so I need investigate how to perform such verification.

compiler/rustc_expand/src/mbe/transcribe.rs

c410-f3r · 2024-06-29T20:17:57Z

Thanks for the review

@rustbot review

compiler/rustc_expand/src/mbe/metavar_expr.rs

compiler/rustc_expand/src/mbe/transcribe.rs

c410-f3r · 2024-07-01T13:34:12Z

Nice!

@rustbot review

petrochenkov · 2024-07-01T19:42:29Z

tests/ui/macros/macro-metavar-expr-concat/syntax-errors.rs

+macro_rules! starting_invalid_unicode {
+    ($ident:ident) => {{
+        let ${concat("\u{999999}", $ident)}: () = ();
+        //~^ ERROR invalid unicode character escape


Where is this error coming from?
I don't see any additional checking in transcribe.rs that would result in this.

Also, will concat!("\x41") produce a or an invalid identifier \x41?

Where is this error coming from?

rust/compiler/rustc_parse/src/lexer/unescape_error_reporting.rs

Lines 36 to 41 in 8a8ad34

EscapeError::LoneSurrogateUnicodeEscape => {

dcx.emit_err(UnescapeError::InvalidUnicodeEscape { span: err_span, surrogate: true })

}

EscapeError::OutOfRangeUnicodeEscape => {

dcx.emit_err(UnescapeError::InvalidUnicodeEscape { span: err_span, surrogate: false })

}

Looks like the parse eagerly halts when an invalid unicode literal is found. Changed to "\u{00BD}" (½).

Also, will concat!("\x41") produce a or an invalid identifier \x41?

At the current time it is not possible to accept literal parameters but I expect to see a ${concat(..)} is not generating a valid identifier error like in https://github.com/rust-lang/rust/pull/126841/files#diff-c527d06b3a89d2cf1181cf0b3122091572d2992346b42217dc290a18d2e2366cR75

cc #126841 (comment)

Okay, the current behavior looks acceptable to me - either we have a literal that doesn't have \ in it so its escaped and unescaped values are the same, or it has \ and then it's not a valid identifier and we produce an error.

c410-f3r · 2024-07-06T21:05:28Z

@rustbot review

petrochenkov · 2024-07-07T10:07:36Z

As usual, literals turned out more complex than it may seem.
@bors r+

bors · 2024-07-07T10:07:39Z

📌 Commit c990e00 has been approved by petrochenkov

It is now in the queue for this repository.

[`macro_metavar_expr_concat`] Add support for literals Adds support for things like `${concat($variable, 123)}` or `${concat("hello", "_world")}` . cc rust-lang#124225

Rollup of 12 pull requests Successful merges: - rust-lang#113128 (Support tail calls in mir via `TerminatorKind::TailCall`) - rust-lang#126841 ([`macro_metavar_expr_concat`] Add support for literals) - rust-lang#126881 (Make `NEVER_TYPE_FALLBACK_FLOWING_INTO_UNSAFE` a deny-by-default lint in edition 2024) - rust-lang#126921 (Give VaList its own home) - rust-lang#127276 (rustdoc: Remove OpaqueTy) - rust-lang#127367 (Run alloc sync tests) - rust-lang#127431 (Use field ident spans directly instead of the full field span in diagnostics on local fields) - rust-lang#127437 (Uplift trait ref is knowable into `rustc_next_trait_solver`) - rust-lang#127439 (Uplift elaboration into `rustc_type_ir`) - rust-lang#127451 (Improve `run-make/output-type-permutations` code and improve `filename_not_in_denylist` API) - rust-lang#127452 (Fix intrinsic const parameter counting with `effects`) - rust-lang#127459 (rustdoc-json: add type/trait alias tests) r? `@ghost` `@rustbot` modify labels: rollup

Rollup of 10 pull requests Successful merges: - rust-lang#126841 ([`macro_metavar_expr_concat`] Add support for literals) - rust-lang#126881 (Make `NEVER_TYPE_FALLBACK_FLOWING_INTO_UNSAFE` a deny-by-default lint in edition 2024) - rust-lang#126921 (Give VaList its own home) - rust-lang#127367 (Run alloc sync tests) - rust-lang#127431 (Use field ident spans directly instead of the full field span in diagnostics on local fields) - rust-lang#127437 (Uplift trait ref is knowable into `rustc_next_trait_solver`) - rust-lang#127439 (Uplift elaboration into `rustc_type_ir`) - rust-lang#127451 (Improve `run-make/output-type-permutations` code and improve `filename_not_in_denylist` API) - rust-lang#127452 (Fix intrinsic const parameter counting with `effects`) - rust-lang#127459 (rustdoc-json: add type/trait alias tests) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of rust-lang#126841 - c410-f3r:concat-again, r=petrochenkov [`macro_metavar_expr_concat`] Add support for literals Adds support for things like `${concat($variable, 123)}` or `${concat("hello", "_world")}` . cc rust-lang#124225

Rollup of 10 pull requests Successful merges: - rust-lang#126841 ([`macro_metavar_expr_concat`] Add support for literals) - rust-lang#126881 (Make `NEVER_TYPE_FALLBACK_FLOWING_INTO_UNSAFE` a deny-by-default lint in edition 2024) - rust-lang#126921 (Give VaList its own home) - rust-lang#127367 (Run alloc sync tests) - rust-lang#127431 (Use field ident spans directly instead of the full field span in diagnostics on local fields) - rust-lang#127437 (Uplift trait ref is knowable into `rustc_next_trait_solver`) - rust-lang#127439 (Uplift elaboration into `rustc_type_ir`) - rust-lang#127451 (Improve `run-make/output-type-permutations` code and improve `filename_not_in_denylist` API) - rust-lang#127452 (Fix intrinsic const parameter counting with `effects`) - rust-lang#127459 (rustdoc-json: add type/trait alias tests) r? `@ghost` `@rustbot` modify labels: rollup

rustbot assigned petrochenkov Jun 22, 2024

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Jun 22, 2024

c410-f3r force-pushed the concat-again branch 2 times, most recently from 1cf8398 to 1a52f42 Compare June 22, 2024 21:18

c410-f3r commented Jun 22, 2024

View reviewed changes

compiler/rustc_expand/src/mbe/metavar_expr.rs Outdated Show resolved Hide resolved

c410-f3r force-pushed the concat-again branch from 1a52f42 to fffdd21 Compare June 22, 2024 23:05

petrochenkov reviewed Jun 24, 2024

View reviewed changes

compiler/rustc_expand/src/mbe/metavar_expr.rs Outdated Show resolved Hide resolved

compiler/rustc_expand/src/mbe/metavar_expr.rs Show resolved Hide resolved

compiler/rustc_expand/src/mbe/metavar_expr.rs Outdated Show resolved Hide resolved

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 24, 2024

c410-f3r force-pushed the concat-again branch from fffdd21 to a9852de Compare June 24, 2024 13:33

c410-f3r force-pushed the concat-again branch from a9852de to 4a9dcc6 Compare June 29, 2024 20:03

c410-f3r commented Jun 29, 2024

View reviewed changes

compiler/rustc_expand/src/mbe/transcribe.rs Outdated Show resolved Hide resolved

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 29, 2024

petrochenkov reviewed Jul 1, 2024

View reviewed changes

compiler/rustc_expand/src/mbe/metavar_expr.rs Outdated Show resolved Hide resolved

compiler/rustc_expand/src/mbe/transcribe.rs Outdated Show resolved Hide resolved

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 1, 2024

c410-f3r force-pushed the concat-again branch from 4a9dcc6 to 49e479a Compare July 1, 2024 13:33

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 1, 2024

petrochenkov reviewed Jul 1, 2024

View reviewed changes

petrochenkov added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 1, 2024

c410-f3r force-pushed the concat-again branch from 49e479a to f4a4781 Compare July 6, 2024 20:54

Add support for literals

c990e00

c410-f3r force-pushed the concat-again branch from f4a4781 to c990e00 Compare July 6, 2024 21:00

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 6, 2024

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 7, 2024

jieyouxu mentioned this pull request Jul 8, 2024

Rollup of 12 pull requests #127475

Closed

jieyouxu mentioned this pull request Jul 8, 2024

Rollup of 10 pull requests #127476

Merged

bors merged commit 2c16d65 into rust-lang:master Jul 8, 2024

rustbot added this to the 1.81.0 milestone Jul 8, 2024

tgross35 added the F-macro_metavar_expr_concat `#![feature(macro_metavar_expr_concat)]` label Aug 24, 2024

	EscapeError::LoneSurrogateUnicodeEscape => {
	dcx.emit_err(UnescapeError::InvalidUnicodeEscape { span: err_span, surrogate: true })
	}
	EscapeError::OutOfRangeUnicodeEscape => {
	dcx.emit_err(UnescapeError::InvalidUnicodeEscape { span: err_span, surrogate: false })
	}

Uh oh!

Conversation

c410-f3r commented Jun 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rustbot commented Jun 22, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

petrochenkov commented Jun 24, 2024

Uh oh!

petrochenkov commented Jun 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

c410-f3r commented Jun 24, 2024

Uh oh!

Uh oh!

c410-f3r commented Jun 29, 2024

Uh oh!

Uh oh!

Uh oh!

c410-f3r commented Jul 1, 2024

Uh oh!

petrochenkov Jul 1, 2024

Choose a reason for hiding this comment

Uh oh!

c410-f3r Jul 6, 2024

Choose a reason for hiding this comment

Uh oh!

petrochenkov Jul 7, 2024

Choose a reason for hiding this comment

Uh oh!

c410-f3r commented Jul 6, 2024

Uh oh!

petrochenkov commented Jul 7, 2024

Uh oh!

bors commented Jul 7, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

c410-f3r commented Jun 22, 2024 •

edited

Loading

petrochenkov commented Jun 24, 2024 •

edited

Loading