Math spec by ollpu · Pull Request #733 · pulldown-cmark/pulldown-cmark

ollpu · 2023-10-16T21:48:29Z

Opening this to discuss a spec for $math$ .

I intend to write about the rationale for the decisions made in the initial spec written in this PR soon. EDIT: This is now written below.

For reference, my implementation of this spec lives in #622 / rhysd#1

TODO

commonmark-hs does not accept empty display math $$$$. I suppose we don't need to either.
Escaped pipes inside tables should match the behavior of code spans.

Motivation

Support for embedding LaTeX math markup in documents has been discussed for a while. There are two types of math items in LaTeX, "inline mode" and "display mode". Display mode renders larger and is put (centered) on its own row by default.

Most syntaxes have to be recognized by the parser in the same phase as e.g. code spans are parsed. Pre- or post-processing is not sufficient, which is why support has to be implemented in pulldown-cmark itself. Consider these cases (written in the plain $...$ syntax):

1: $a*b*c$
2: $a<b>c$
3: `$example$`

Post-processing would not work properly for cases 1 and 2.

Case 3 highlights why pre-processing is problematic. You could try to evade this with heuristics, but not all cases can reasonably be covered that way.

General syntax

A great overview of the existing syntaxes can be found here.

The basic syntax proposed in this PR is $...$ for inline math, $$...$$ for display math.

The choice of syntax is (and has been) a topic of endless debate. My main arguments in favor of this syntax are that:

it comes directly from TeX/LaTeX, so it is familiar to users that know them. (Though \[...\] is now preferred for display math in LaTeX.)
it is very commonly supported across other markdown parsers (built-in or as a provided plugin), and sites and services that use them.
it feels markdown-y in the sense that it is similar to e.g. emphasis and code spans. It also doesn't conflict with any existing markdown syntax.

The main argument against this syntax is that dollar characters may be used in existing documents where math isn't intended. This is mitigated somewhat by whitespace/flanking rules, discussed later.

It is also worth noting that math parsing will be optional in pulldown-cmark, not enabled by default. Personally, making the syntax more complicated only to avoid some existing false positives seems fraught, because it solves a (mostly) temporary problem at the expense of making the feature permanently less convenient. Not to mention the existing markdown content written for other parsers that already uses this syntax.

Alternatives

Some remarks on potential alternative syntaxes which I consider relevant bringing up here.

$`...`$ for inline mode

To my understanding, this was initially used by GitLab. It is most prominently supported by GitHub and GitLab, though both now also support the $...$ syntax.

This is the only syntax that can almost be implemented in post-processing only: detect parsed code spans surrounded by $ text.

Almost, because CommonMark code spans don't preserve line breaks. TeX comments for example, are sensitive to line breaks.
```
$`
f(x) = % comment
2x + 1
`$
```
(Escaping the dollars poses a minor problem, too. Currently the pulldown-cmark parser output does not distinguish between a freestanding $ and an escaped \$.)

I am not opposed to supporting this in addition to $...$ , but it could be done in an external crate.

GitHub seems to refer to this as a cop-out for when the plain $...$ syntax doesn't work for whatever reason (they parse math by post-processing). If we do things properly, it should essentially never be necessary to use to $`...`$.
```math blocks for display mode.

This is quite simple to do in another crate or by the crate user directly.
$...$ for inline mode or \[...\] for display mode.

These are both also supported in LaTeX, though $...$ isn't very commonly used. They clash quite badly with how CommonMark allows escaping punctuation, so I'm not fond of them within markdown.
$$\n...\n$$ for display mode, mimicking fenced code blocks.

It is very common in LaTeX to write a display mode math item all on one line:
```
$$ f(x) = 2x + 1 $$
```
Also refer to the Display Is Inline section below.

Specifics

Given the choice to go with the $...$ and $$...$$ syntax, commonmark-hs's math extension seems like a good place to loan specifics from. Some of it is described well in its spec as well. commonmark-hs with math support is available in Pandoc under the commonmark+math and commonmark_x modes.

Surrounding rules for inline math

To avoid some false positives in inline math items, we require this:

In inline math, the opening $ must not be followed by a whitespace, and the closing $ must not be preceeded by whitespace.
This is not math: 2000$.
And neither is this $ 4 $.
Or this $4
$.

Pandoc recently (?) transitioned to this slightly more refined version, which is also a viable option:

The opening $ must have a non-space character immediately to its right, while the closing $ must have a non-space character immediately to its left, and must not be followed immediately by a digit. Thus, $20,000 and $30,000 won’t parse as math.

GitHub has some very inconsistent rules for this, which I haven't been able to fully decipher. Even if we aim for GitHub compatibility, I don't think we should replicate its rules exactly.

Display Is Inline

Because of how display math is rendered centered as a block, one might assume that it should also be parsed as a block level item.

However, it is common in LaTeX to write even display math within paragraphs, as part of a sentence.

The equation $$ 2x + 1 = 0 $$ has one root.

TeX/LaTeX also does not support empty lines within display math, as they signify a paragraph change:

$$
causes

errors
$$

Therefore it actually makes sense to parse display mode math as inline items as well. This is also easier to implement: inline and display mode math can reuse much of the same parsing logic.

Display math has a tendency to be split across multiple lines more liberally than inline math, however. CommonMark has a principle that all syntax that indicates block structure has higher priority than inline items. This does mean that display math can be broken by certain syntax:

$$ f(x) = 2x
- 1 oops it's a list $$

$$
f(x)
=
oops it's a setext-style heading
$$

This can be annoying, but I think it's an acceptable tradeoff. commonmark-hs does this, and it's not too surprising because it follows the same principle that applies to e.g. code spans as well.

Pandoc's own markdown flavor (different from commonmark-hs) has the best of both worlds: display math is an inline element, but also isn't preempted by block level syntax. Actually the same applies for code spans and inline math in the Pandoc flavor as well.

If we were to only make display math special and parse it in the first phase, cases like these would break:

`x$$y` ... `x$$y`

$x$$y$ ... $x$$y$

So this is a wholesale deal: either all of the constructs are higher priority than block-level indicators, or none of them are.

What about special casing? If a paragraph were to only consist of one display math item, it is parsed as such in the first phase. Other kinds of display math items are parsed in the second phase. This is reminiscent of how HTML blocks work in CommonMark. The difference in behavior depending on subtle newline changes isn't so nice with this approach though, and HTML blocks already cause enough confusion.

Explainer in separate paragraph:

$$
parses
=
ok
$$

Explainer in same paragraph:
$$
setext
=
heading...
$$

Nested braces

commonmark-hs recently also supports having $s within math items, as long as they are inside nested, balanced braces. (An escaped \$ is not the same, because it would also be escaped in the LaTeX code, so rendered as a dollar symbol.)

$\text{math $x$ again}$

This is not proposed in this PR as it currently stands. I feel it adds complexity to the implementation for negligible gain. Dollars can generally be replaced by alternative syntax like  \[ \]. After seeing #734, I am on the fence about this.

Output

The markdown parser only needs to recognize where there are math items. Any further processing or compilation is not in scope for this proposal. It should be relatively easy for a user of the crate to map the resulting Item::Math to whatever HTML they please.

The default HTML rendering of math items is something we have to specify, however.

A common way to render LaTeX math in HTML is to do it on the browser side (MathJax/KaTeX). This is facilitated either by using heuristics to find $...$/$$...$$ etc., or by having <span>s with special classes. Since we've already parsed the math items, it doesn't make sense to emit them back without semantics only to be parsed by another set of rules again. Spans with classes are likely going to be what most users want.

The choice of classes (math inline, math display) is a little arbitrary though. For the spec, I copied what Pandoc does in the "KaTeX" math emitting mode. I have no strong opinions on this.

Based on pulldown-cmark#622 and copied from https://github.com/ollpu/pulldown-cmark/tree/alt-math. Co-authored-by: rhysd <lin90162@yahoo.co.jp>

rambip · 2023-10-17T17:29:02Z

Wow this spec is very complete !

And I cannot disagree with any of the decisions that were made.

The only choice that seems pretty arbitrary is the name of the css classes (math display and math inline)

Martin1887 · 2023-10-17T17:37:59Z

Thanks, it looks nice.

I think any decision should be compared against Pandoc and mdBook math extensions. In addition, the mdBook math code is written in Rust and it uses pulldoown-cmark, so it could be reused or be inspiring.

I also think involving mdBook developers in this discussion would be very interesting, maybe opening an issue in the mdBook repository is the best manner of reaching them?

rambip · 2023-10-17T20:22:35Z

I think one of the main points to discuss is wether or not pulldown_cmark will allow to go further and to compile math to html.

Pros

that's what a lot of static site generators using pulldown_cmark want. In particular mdbook_katex, the mdbook math plugin, does exactly that
no need for javascript dependencies at all in the html
no need for the user to chose a js dependency (mathjax, katex) and do the gluing to include in the html.

Cons

it will require to use external code, that will not have the same guarantees and will not be tested with the same standards
the best implementation of the tex math syntax is katex, written in js. There are bindings for rust, but they require to emulate js.
This is not resource efficient at all.

My take on this

I wrote the comparison for the sake of it, but IMO the drawbacks outweigh the pros.
That said, the user needs clear documentation on how to setup his own math pipeline, with build-time katex (and some rust code included) or with run-time mathjax/katex (with some js code included)

rambip · 2023-10-17T20:27:19Z

If I understand correctly, if math parsing is implemented here, then mdbook-katex will need to post-process the markdown and look for <span class="math ..."> instead of preprocessing it

ollpu · 2023-10-17T22:06:30Z

@rambip It should be relatively easy to change how the math is rendered by maping the output of the Parser for Event::Math (or whatever it's called in the final implementation), and converting it to appropriate Event::Html. So this is done after parsing the markdown, but before rendering to HTML, which are separated in the API anyway.

Given that it can be done just as well in a different crate, I see no reason to support further compilation in pulldown-cmark itself.

Caellian · 2023-10-18T04:49:41Z

I think one of the main points to discuss is wether or not pulldown_cmark will allow to go further and to compile math to html.

I'd like to add that displaying math is something that has no ideal solution:

katex doesn't support whole latex
MathML isn't fully supported by browsers (MathML Core is; not feature complete either)
Rendering to SVG is tricky and requires system dependencies (not self-contained)
- CON: selection of text within expressions doesn't work
- PRO: provides best
some expressions might require additional packages to render properly which again requires running system latex

Tectonic is a self contained engine that could be used but its HTML generation is a bit tricky.

So I too agree that generating a span for further processing is the way to go because other options would unnecessarily bloat the crate in cases where dependents want to go another route.

The only choice that seems pretty arbitrary is the name of the css classes (math display and math inline).

What about emitting <code> blocks given that the expression is LaTeX and not regular text? Also classes could be more verbose (esp. inline) to prevent accidental collisions - how about using math-inline and math-display?

wooorm · 2023-10-18T13:13:56Z

Hey! I’m not using this project but do care about markdown.
If you are going to support math with dollars, I recommend going with basically exactly how code works in CommonMark. But with $ instead of `. And allowing two dollars to start block math. And not using the first “word” of the info string for a class (the ```js -> language-js part).

That’s how it works in micromark/remark/mdx in the JS world. In markdown-rs and mdxjs-rs in the Rust world. And how the syntax highlighting on GitHub works (with the exception that singleDollarTextMath is not supported there, so as to not conflict with humans writing $ in regular prose).

I do also recommend for compiling to (<pre> and) <code> elements indeed!

Ref

rambip · 2023-10-18T13:21:34Z

Hey! I’m not using this project but do care about markdown. If you are going to support math with dollars, I recommend going with basically exactly how code works in CommonMark. But with $ instead of `. And allowing two dollars to start block math. And not using the first “word” of the info string for a class (the ```js -> language-js part).

That’s how it works in micromark/remark/mdx in the JS world. In markdown-rs and mdxjs-rs in the Rust world. And how the syntax highlighting on GitHub works (with the exception that singleDollarTextMath is not supported there, so as to not conflict with humans writing $ in regular prose).

I do also recommend for compiling to (\<pre\> and) <code> elements indeed!

Thank's for your advice

On of the points where your view disagree with the spec is the following:

$$
x=

1
$$

This will not be rendered as math, as it is the case with a lot of tools like pandoc and stackedit

wooorm · 2023-10-18T13:29:55Z

Right, as there is no spec yet, all tools are different. Making a spec will always result in it being different from some tools.

Martin1887 · 2023-10-18T14:42:38Z

The main purpose of this project is parsing Markdown, with a simple HTML converter (to be transformed in a sub-crate) doing only basic things, so rendering Math elements is totally out of scope.

It seems to not exist a standard manner of rendering Math from HTML, so this is a critical point to evaluate. Again, looking at mdBook and Pandoc is a good idea. Pandoc allows to choose the Math engine and the URL of the JS included in the HTML for some of them, but it seems too complex to be a valid strategy for pulldown-cmark.

Martin1887 · 2023-10-19T17:04:41Z

I have checked the mdBook code and it does nothing with Math elements, but just inserts the MathJaX JavaScript if it is enabled.

Therefore, a possible solution would be generating the Math events (including support for the MathJaX syntax) and leaving them untouched in the HTML generator. Another crate using pulldown-cmark could use KaTeX or anything else to transform Math formulas into SVG or whatever.

I have opened the issue rust-lang/mdBook#2222 in the mdBook repository to ask feedback from mdBook developers and stakeholders.

ollpu · 2023-10-23T23:29:35Z

I have now updated the PR description with a longer write-up.

rambip · 2023-10-24T07:59:57Z

#733 (comment)

Great summary of what's been going on !

mgeisler · 2023-10-25T20:00:38Z

Hey folks! It's super cool to see math support being added!

I come here from google/mdbook-i18n-helpers#105, which I believe is ultimately about how pulldown-cmark-to-cmark turns backslashes in the Markdown AST into Markdown text.

So this made me look around and find this issue. I read Writing mathematical expressions and it seems to gloss over the handling of escaped characters and in particular over the handling of escaped backslashes? It mentions "escape" once and does not mention "backslash" at all 🙂

Worse, the GitHub page it says

To display a dollar sign as a character in the same line as a mathematical expression, you need to escape the non-delimiter $ to ensure the line renders correctly.

Does "same line" refer to the source file here? I hope not — but I wanted to double check with the people writing the implementation of the spec here.

So, how does backslash work inside $...$ ? Is it inactive in the same way as in inline code? I guess other Markdown characters are inactive too, so that $x *y* z$ doesn't add emphasis to y? I think this isn't stated explicitly above and that is perhaps because it's simple that it goes without saying? 🙂

ollpu · 2023-10-25T20:15:23Z

Yes, the intent with the spec in this PR is that backslash escapes are also disabled inside math. I should maybe add some prose comparing how code span vs math contents are treated. (They are almost the same, except newlines are preserved and the surrounding spaces rule (` x ` -> x) is not needed.)

GitHub's implementation is known to be buggy, because it just uses post-processing for the $...$ syntax.

$x *y* z$ -> $x y z$

$\,$ -> $,$ (should be a thin space)

We don't want to replicate this.

To display a dollar sign as a character in the same line as a mathematical expression, you need to escape the non-delimiter $ to ensure the line renders correctly.

This I think just refers to the fact that dollar characters sometimes need to be escaped to prevent them from being recognized as math. If they're surrounded by spaces though, that's not a problem, even on GitHub.

$math$ $500 $600 -> $math$ $500 $600

ollpu · 2023-10-25T20:15:47Z

Sorry, accidentally pressed the close button.

xelatihy · 2023-10-25T20:40:33Z

This spec looks great. I am really hoping that it can be merged in with an implementation since not having math is blocking for many applications. Thanks for the great work.

mgeisler · 2023-10-25T20:44:39Z

Thanks for confirming! I think it would be important to explicitly state how escapes work: LaTeX math normally has a lot of \ and _, both of which are special Markdown characters 🙂

To display a dollar sign as a character in the same line as a mathematical expression, you need to escape the non-delimiter $ to ensure the line renders correctly.

This I think just refers to the fact that dollar characters sometimes need to be escaped to prevent them from being recognized as math. If they're surrounded by spaces though, that's not a problem, even on GitHub.

Ah, I was worried that they were referring to the lines in the source file. A single newline is ignored in an inline code span is ignored. So

foo \` `x y z` bar

and

foo \`
`x y z` bar

has the same AST: without newline, with newline.

In other words, I have to escape the backtick in both cases, regardless of how my source file is wrapped. I was afraid that GitHub made the escaping of a dollar sign depend on how the source file is wrapped.

- Display math can no longer be empty (aligns with commonmark-hs) - Add test case $x$$$$$$$y$$ from raphlinus#734 - Add description about how content is handled compared to code spans: newlines, surrounding spaces, and pipes inside tables. (Almost) implemented in https://github.com/ollpu/pulldown-cmark/tree/alt-math

Analogous to pulldown-cmark#777

duskmoon314 · 2023-12-05T09:26:33Z

Hello. What problems should be solved to further this spec? And which PR (#622 or #734) should we expect to be merged?

I'm trying to build up a mdBook preprocessor that leverages typst to generate SVG of math expression. (mdbook-typst) Thus, I only want to get the math expression and send it to typst. That is, I support this solution:

Therefore, a possible solution would be generating the Math events (including support for the MathJaX syntax) and leaving them untouched in the HTML generator. Another crate using pulldown-cmark could use KaTeX or anything else to transform Math formulas into SVG or whatever.

Maybe something like converting $<math-inline>$ and $$<math-block>$$ to <span> and <div> with specific class name is OK?

Martin1887 · 2023-12-06T17:52:22Z

Hello. What problems should be solved to further this spec? And which PR (#622 or #734) should we expect to be merged?

I'm trying to build up a mdBook preprocessor that leverages typst to generate SVG of math expression. (mdbook-typst) Thus, I only want to get the math expression and send it to typst. That is, I support this solution:

Therefore, a possible solution would be generating the Math events (including support for the MathJaX syntax) and leaving them untouched in the HTML generator. Another crate using pulldown-cmark could use KaTeX or anything else to transform Math formulas into SVG or whatever.

Maybe something like converting $<math-inline>$ and $$<math-block>$$ to <span> and <div> with specific class name is OK?

Before merging Math spec, the 0.10 version must be released. There are few remaining tasks for this release, but these are extremely busy weeks for maintainers, so it will no happen until the end of the month.

This spec (and alternative considerations) and pull requests must be carefully reviewed before taking a decision, so some problems may be detected or maybe some of these pull requests will be merged without changes, but nothing is clear now.

SichangHe · 2024-02-23T09:20:11Z

Hello there, I maintain mdBook-KaTeX. Here are my opinions:

In principle, just pick a spec that is close to what most people are using so their existing markup probably work (otherwise discussions on Markdown can go on forever).
I see no objection to the spec in this PR's math block definitions. Please merge that part right away.
As of what pulldown-cmark should do with the math blocks,
1. Emit a Event::Math. Let the user deal with the preprocessing, rendering, or whatever.
2. Rendering the HTML: just keep math blocks as they are. This would make MathJax or KaTeX work and save the users some \\s, or
3. Rendering to spans with custom classes provided by the user. This also works well with MathJax and KaTeX.

How does this help mdBook-KaTeX: I would probably just replace our current rudimentary heuristics to find math blocks with pulldown-cmark, and use pulldown-cmark-to-cmark to convert the events back… Otherwise we would need to be a renderer (we are a preprocessor) and break everyone's setup.

Regards,
Steven Hé (Sīchàng)

Initial math spec

63a29a1

Based on pulldown-cmark#622 and copied from https://github.com/ollpu/pulldown-cmark/tree/alt-math. Co-authored-by: rhysd <lin90162@yahoo.co.jp>

ollpu mentioned this pull request Oct 16, 2023

Support math extension #622

Closed

notriddle mentioned this pull request Oct 19, 2023

Yet another Math implementation #734

Merged

Martin1887 mentioned this pull request Oct 19, 2023

Future Math support in pulldown-cmark rust-lang/mdBook#2222

Open

ollpu closed this Oct 25, 2023

ollpu reopened this Oct 25, 2023

ollpu added 2 commits November 3, 2023 15:59

Update math spec running instructions

632e05f

Analogous to pulldown-cmark#777

ehuss mentioned this pull request Dec 2, 2023

Are we looking for more maintainers? rust-lang/mdBook#1589

Closed

HU90m mentioned this pull request Jan 29, 2024

Lychee trys to parse math equation leading to erroneous error message. lycheeverse/lychee#1197

Closed

Martin1887 force-pushed the master branch from cc49653 to 2c0aebf Compare March 1, 2024 17:11

Martin1887 closed this Mar 23, 2024

ollpu mentioned this pull request May 5, 2024

Math: single = creates heading in HTML #884

Closed

ollpu mentioned this pull request Aug 24, 2025

Parsing of math broken by newlines differs from Pandoc #1052

Closed

xelatihy mentioned this pull request Jan 14, 2026

Display math interrupted by '-', '=' in existing academic documents #1062

Open

Conversation

ollpu commented Oct 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Motivation

General syntax

Alternatives

Specifics

Surrounding rules for inline math

Display Is Inline

Nested braces

Output

Uh oh!

rambip commented Oct 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Martin1887 commented Oct 17, 2023

Uh oh!

rambip commented Oct 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pros

Cons

My take on this

Uh oh!

rambip commented Oct 17, 2023

Uh oh!

ollpu commented Oct 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Caellian commented Oct 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wooorm commented Oct 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rambip commented Oct 18, 2023

Uh oh!

wooorm commented Oct 18, 2023

Uh oh!

Martin1887 commented Oct 18, 2023

Uh oh!

Martin1887 commented Oct 19, 2023

Uh oh!

ollpu commented Oct 23, 2023

Uh oh!

rambip commented Oct 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mgeisler commented Oct 25, 2023

Uh oh!

ollpu commented Oct 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ollpu commented Oct 25, 2023

Uh oh!

xelatihy commented Oct 25, 2023

Uh oh!

mgeisler commented Oct 25, 2023

Uh oh!

duskmoon314 commented Dec 5, 2023

Uh oh!

Martin1887 commented Dec 6, 2023

Uh oh!

SichangHe commented Feb 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

ollpu commented Oct 16, 2023 •

edited

Loading

rambip commented Oct 17, 2023 •

edited

Loading

rambip commented Oct 17, 2023 •

edited

Loading

ollpu commented Oct 17, 2023 •

edited

Loading

Caellian commented Oct 18, 2023 •

edited

Loading

wooorm commented Oct 18, 2023 •

edited

Loading

rambip commented Oct 24, 2023 •

edited

Loading

ollpu commented Oct 25, 2023 •

edited

Loading