Nested macro_rules and edition spans

[ehuss]

This issue is to decide if and how edition hygiene should work for nested macros.

Background

Today (I think) we have a fairly simple principle that the edition behavior follows the crate where the code is written. When calling a macro in an external crate, the code in the macro definition follows that defining crate's edition. This helps ensure that the macro call should work whether or not the caller or the macro_rules crate changes their editions.

There's also a (IMHO) reasonable expectation that the input to the macro follows the caller's edition. This helps ensure a certain level of consistency when looking at the code as written in the caller's crate, that it is all following the same rules. It also helps maintain SemVer compatibility.

This works with basic macro definitions. The problem arises when the macro defines an inner macro, and the caller provides input that gets substituted into that macro.

Which span counts for edition behavior?

Each edition feature needs to look at some token to decide if the code follows a particular edition's behavior. The exact token used isn't really documented. It is also somewhat implicit in the compiler's code, since most of the time it is doing something like item.span.at_least_rust_2024(), and the machinery that answers that question is heavily abstracted.

For example, some edition features:

• Keywords — Usually the identifier token itself drives whether it is interpreted as a keyword or identifier.
• into_iter — In let _ = [1i32].into_iter().collect::<Vec<i32>>();, the into_iter token determines the behavior.
• unsafe extern — In extern "C" { fn foo(_: i32); }, the edition of the opening brace token determines whether or not the unsafe keyword is needed.

This becomes complicated because some macros mix tokens in non-terminals from multiple places (like the macro definition itself, it's input, nested macros, etc.). Not to mention proc-macros which do all sorts of strange things with token respanning.

Example

This problem is illustrated in https://github.com/rust-lang/rust/blob/master/tests/ui/editions/nested-macro-rules-edition.rs, but to summarize:

// crate bar, edition 2021
#[macro_export]
macro_rules! make_macro_with_input {
    ($i:ident) => {
        macro_rules! macro_inner_input {
            () => {
                pub fn $i() {}
            };
        }
    };
}

// crate foo, edition 2024
bar::make_macro_with_input!{gen}
macro_inner_input!{}

Today (Rust 1.86) this passes, when I think it should not, because the gen identifier is a keyword in 2024.

Conversely, if bar is Rust 2024, and foo is Rust 2021, then it fails when I think it should pass, because gen is not a keyword in Rust 2021.

This behavior makes it a potentially SemVer breaking change to update bar to the new edition because that can cause callers to start failing.

Implementation

I have not yet had time to really research why this is happening. I believe this is happening around here. When the inner macro is generated, its spans are all set to the def-site's edition.

However, there's also this strange thing which I do not yet understand. There is some more context of that in #84429.

I do not yet know how feasible it would be to maintain track of the edition of the inputs (substitutions) to a nested macro.

cc @rust-lang/lang @rust-lang/wg-macros

[Aditya-PS-05]

@rustbot claim

[workingjubilee]

@Aditya-PS-05 Hi! I know you're eager to help, but this issue is not ready for any implementation work. If you would like to share in our labors, please message me on Zulip and I have a ton of things you can help out with!

[theemathas]

cc #86539

[joshtriplett]

@ehuss Thanks for the great summary!