You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Rust edition upgrade, DX fix for generated token macros (#2)
# Breaking changes
## Generated token macros are no longer `#[macro_export]`
The `Token` derive macro generates a `macro_rules!` macro for each of the enum
variants. E.g., for Lox's `Token` enum in `examples/lox`:
```rs
// examples/lox/src/tokens.rs
#[derive(DebugLispToken, PartialEq, Token, Lexer)]
pub enum Token {
#[subset_of(Ident)]
#[pattern = "and|class|else|false|for|fun|if|nil|or|print|return|super|this|true|var|while"]
Keyword(Substr, Span),
#[pattern = "[a-zA-Z_][a-zA-Z0-9_]*"]
Ident(Substr, Span),
#[pattern = r"[(){}]"]
Brace(Substr, Span),
#[pattern = "[,.;]"]
Punct(Substr, Span),
#[pattern = "[=!<>]=?"]
#[pattern = "[-+*/]"]
Operator(Substr, Span),
#[pattern = "[0-9]+"]
NumLit(Substr, Span),
#[pattern = r#""[^"]*""#]
StrLit(Substr, Span),
}
```
We get `macro_rules!` macros named `keyword`, `ident`, `brace`, `punct`,
`operator`, `num_lit` and `str_lit`. These are mostly useful for the `Parse`
implementations, e.g.:
```rs
// examples/lox/src/decl.rs
impl Parse for ClassDecl {
type Stream = ParseStream;
fn parse(input: &mut Self::Stream) -> Result<Self> {
// ...
let superclass = if input.check(operator![<]) {
input.consume(operator![<])?;
Some(input.consume_kind(TokenKind::Ident)?)
} else {
None
};
// ...
}
}
```
Previously, those generated macros were declared like this:
```rs
#(
#[macro_export]
macro_rules! #snake_case_variant_ident {
// ...
}
)*
```
That has always [caused some headaches](rust-lang/rust#52234 (comment))
which I was never really sure how to deal with. In the examples here, and in my
consuming projects, I had resorted to using `*` imports everywhere to work
around the problem (in hindsight though, I think that likely just hides the lint
by obscuring what we're doing instead of actually addressing the issue).
This PR attempts an alternative solution. `#[macro_export]` is intended for
libraries to expose macros to external consumers, but I don't foresee the macros
generated by the `Token` derive being actually useful in that context. So
instead, the generated macros are now declared like this:
```rs
#(
macro_rules! #snake_case_variant_ident {
// ...
}
pub(crate) use #snake_case_variant_ident;
)*
```
This is a breaking change for two reasons:
1. If you were depending on those `#[macro_export]` attributes to make the
generated macros available to external consumers of your library, that is no
longer going to work. Again, I don't imagine this was actually a real-world
use case for anyone, but I've been wrong before! Let me know if this is a
problem for you and I'll see what we can do about it.
2. If you had been importing those macros from the root of your crate, but your
`Token` enum is _not_ declared in the crate root, you'll need to update your
import paths to instead import them from the module where the enum is
declared. E.g.:
```rs
mod token {
use gramatika::{DebugLisp, Span, Substr, Token as _};
#[derive(DebugLispToken, PartialEq, Token, Lexer)]
pub enum Token {
#[pattern = ".+"]
Word(Substr, Span),
}
}
mod foo {
// use crate::word; // 👎
use crate::token::word; // 👍
}
```
On the bright side, tools like `rust-analyzer` should now find and
automatically suggest the correct import paths for those macros, so the
fastest way to migrate will probably be to just delete your existing `use`
statement and invoke your editor's suggestion feature to re-import any
unresolved symbols from their correct paths.
# Other changes
* Updated the Rust edition to 2021 and fixed any resulting errors
* Fixed any new `clippy` lints as a result of upgrading my environment to
v1.77.2
* Performed some low-hanging-fruit dependency upgrades.
`regex-automata` and `syn` are still out of date for now — I attempted
to update the former, but couldn't figure out how to migrate after ~10 minutes
of poking around, and unfortunately I have other priorities that need to take
precedence. Didn't even attempt `syn` because it's a major version bump, and
that crate is basically the backbone of this whole project, so it'll have to
wait for now.
0 commit comments