Thank you for implementing this! 🎉

Notes

I chose the review style of refactoring your PR myself to see what I think should be kept/changed. You can view the end result of it in this branch. (It's all squashed since I'm no jj voodoo master 😅.) Here are the things I think should be changed:

• In non-math realizations, the SymbolElem should be eagerly transformed into TextElem in the else branch of visit_math_rules. This way, we

  • can skip much of the regex matching complexity
  • don't need any special handling of symbols in inline layout
  • can remove SymbolElem from the grouping rules

• I'm not really comfortable with adding a symbol selector. But I think we don't actually need it! If we transform SymbolElem to TextElem eagerly in non-math as mentioned above, the only way to regex match a SymbolElem is from find_regex_match_in_text directly on a SymbolElem. For that, your lone symbol check should suffice. And we can also remove the TextualKind stuff. On top, I suggest that we call the field of SymbolElem text so that we have minimal breakage in existing text show rules that will now get a symbol. Since SymbolElem is temporary, having the cleanest name doesn't matter much.

• I'm not comfortable with a generic Content -> char cast, since it is a fundamentally lossy operation. The existing thing in AccentElem is a hack and I would prefer for it to stay isolated to it. If we just let escapes and shorthands stay Symbol instead of SymbolElem (keeping the return type of the Evals at Value rather than Content), from my testing, we don't need this change to fix Single letter strings get parsed as italicized in math mode #274.

• You had to update issue-math-realize-scripting because of missing spaces. The reason for that is that joining content and symbol still turned the symbol into a TextElem. Can be fixed by updating fn join in foundations/ops.rs.

• The SymbolElem can go into foundations/symbol.rs and we just spell out crate::foundations::Repr in the impl.

• In two places &c.to_string() can be replaced by c.encode_utf8(&mut [0; 4]) to skip an allocation.

• The refactor in layout/math/text.rs looks fine!

• I think the intent of the content-fields-complex isn't really kept since it tries to evaluate a normal-looking equation, but now there are various [..] thingies. If we switch the field of SymbolElem to text, we can replace func == text with elem.has("text") to fix it completely within let compute (alongside two of your changes)

As said, the branch linked above contains the result I ended with, containing all these changes. If you disagree with some of my proposed changes, then I'm curious to hear your thoughts. If not, I don't mind whether one of us pushes the squashed changes to this PR or whether you incorporate them into your cleaner commit history. I assume the individual commits were mostly for review convenience, so maybe it's unnecessary work.

Responses

I'm not sure what we want for the documentation of SymbolElem, I feel like it should be more of a hidden element that isn't exposed to the user for now, and the value type Symbol should be the main visible API.

No documentation is necessary. Even though SymbolElem is observable in a few cases, let's consider it internal for now. (SequenceElem is way more important and also not yet documented.)

TODO: Can we get away with not checking the RTL/LTR direction?

Not relevant if we transform to TextElem eagerly.

TODO: Update traversal with std::ops::ControlFlow to exit early.

Seems like a nice change in general, but unrelated to the PR. :)

TODO: Should we try to improve these error messages? [In cast! for char]

I'm not sure what change you are envisioning, but also probably not tied to this PR.

TODO: Not sure if this should change. [In Repr impl]

Having SymbolElem's repr be just like TextElem's is good for now.

Thanks! I've incorporated all of your changes 👍

Can be fixed by updating fn join in foundations/ops.rs.

Aha!

&c.to_string() can be replaced by c.encode_utf8(&mut [0; 4]) to skip an allocation.

I thought there should be a way to do this! Couldn't come up with the right incantation 😁

I rebased onto main and used jj squash <path> --to <commit> to put your changes into the right commits (after making a new commit matching your files; jj new, jj restore --from <commit>).

I did make some changes, but only to comments! You can see them on this commit in my repository (which matches the rebased update exactly). I mostly added more context or fixed typos.

If those look good, I'm ready to merge.

Fast-forward merged locally to keep the good commits. :)

Thank you!!

As a user affected by #274: very nice, thanks!

Hi @laurmaedje @wrzian! The description of this PR hints at an upcoming overhaul of how Typst handles math text. I couldn't find another obvious place where ideas are being collected for that overhaul, so I'll include a small note here. If there's a more appropriate place to put this, could you direct me to it? Thanks :)

The current math mode behavior of removing spaces between consecutive variables makes typesetting expressions difficult in contexts where juxtaposition indicates some common operation, e.g. function application in lambda calculus or logic, or multiplication in arithmetic. The current workarounds are quite cumbersome, so it would be really nice if the math text overhaul allowed this space-removal between consecutive variables to be turned on or off with a #set rule. Alternatively, if the new math mode is able to provide a type for variables in math mode (e.g. the type of $a$ and $b$ in the expression $a b$) which could be targeted by a #show rule, then the style could be adjusted, solving the problem. I think it would be wise to think about this already at the design stage, as it might be more difficult to fix if the new design calcifies the current behavior.

If there's a more appropriate place to put this, could you direct me to it?

Much of the development discussions happen on Discord. There's a dedicated thread for math there.

(I think it's fairly unfortunate that many of the design discussions around Typst are enclosed on Discord. Various people cannot access it easily, and it is not easily searchable by web engines, does not have public archives, etc. This trove of content is going to be lost in a few years when Discord enshittifies in response to the next popular chat app, and the early history of Typst is going to be basically lost.)