Skip to content

Commit b8d9e4c

Browse files
docs(xish): add CLAUDE.md + processor flow overview docs (#1370)
## 🧰 Changes - [x] add a barebones `CLAUDE.md` file - [x] add [`MDXish Processor Overview.md`](https://github.com/readmeio/markdown/blob/cc9be60d13f130cd2b775731a03b39067627d754/.claude/MDXish%20Processor%20Overview.md) file for Claude ## 🧬 QA & Testing Run Claude Code in your local clone; does it read from the `CLAUDE.md` file correctly? --------- Co-authored-by: Dimas Putra Anugerah <63914983+eaglethrost@users.noreply.github.com>
1 parent 30e037d commit b8d9e4c

3 files changed

Lines changed: 336 additions & 0 deletions

File tree

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
# MDXish Processor Overview
2+
3+
## `mdxishAstProcessor()`
4+
5+
### Preprocessing Step
6+
7+
> **See**: @lib/mdxish.ts#92-103
8+
9+
`preprocessContent` is a string-level preprocessor that runs before the markdown is handed to remarkParse. It exists because several syntactic patterns in ReadMe's flavor of markdown would confuse or break the standard CommonMark/MDX parser if fed to it directly. By patching the raw string first, these issues are sidestepped.
10+
11+
It applies four transforms in sequence:
12+
13+
1. **`normalizeTableSeparator()`**
14+
15+
Fixes malformed GFM table separator rows — e.g. misplaced alignment colons like `|: ---``| :---`. Without this, remarkGfm would fail to recognize the table.
16+
1. **`terminateHtmlFlowBlocks()`**
17+
18+
Inserts blank lines after standalone HTML elements (like `<div>...</div>`) when the next line is regular markdown. CommonMark's HTML flow rules only terminate on blank lines, so without this, the parser would swallow subsequent markdown content into the HTML block token.
19+
1. **`preprocessJSXExpressions()`** (skipped in safeMode)
20+
21+
Handles JSX attribute expressions (`href={someVar}`) and unbalanced braces before the MDX expression tokenizer sees them. It evaluates attribute expressions against jsxContext, converts style objects to CSS strings, and escapes stray braces that would cause MDX parse errors.
22+
1. **`processSnakeCaseComponent()`**
23+
24+
Remark's parser rejects tag names containing underscores (e.g. `<my_component>`). This step replaces known snake_case component names with safe placeholder names (`<MDXishSnakeCase0>`) and returns a mapping so they can be restored later by the `restoreSnakeCaseComponentNames` transformer in the run phase.
25+
26+
##### Where it sits in the flow
27+
28+
```
29+
preprocessContent (string → string)
30+
31+
32+
┌─────────────────────────────────────────────────────┐
33+
│ normalizeTableSeparator — fix table syntax │
34+
│ terminateHtmlFlowBlocks — fix HTML flow │
35+
│ preprocessJSXExpressions — eval/escape JSX │ before
36+
│ processSnakeCaseComponent — placeholder swap │ parsing
37+
└─────────────────────────────┬───────────────────────┘
38+
39+
40+
remarkParse (tokenize)
41+
42+
43+
MDAST transformers...
44+
...
45+
restoreSnakeCaseComponentNames ◄── undo (4)
46+
```
47+
48+
### Processor Pipeline
49+
50+
> **See**: @lib/mdxish.ts#105-178
51+
52+
The core Xish engine which parses Markdown and converts it to an MDAST object. This is the base processor used for both the editor and rendering flows.
53+
54+
```
55+
| ................ process (parse only) ...................... |
56+
| .. parse ........... | .............. run .................. |
57+
58+
NO COMPILER
59+
+--------+ +----------+ (MDAST is
60+
Input ->- | Parser | ->- Syntax Tree ->- | N/A | returned
61+
+--------+ | +----------+ directly)
62+
| |
63+
| X
64+
| |
65+
| +--------------+
66+
| | Transformers |
67+
| +--------------+
68+
| |
69+
┌────────────┘ ┌───┴──────────────────────────────┐
70+
│ │ │
71+
│ PARSER │ MDAST TRANSFORMERS │
72+
│ (micromark) │ (remark plugins) │
73+
│ │ │
74+
│ remarkParse │ remarkFrontmatter │
75+
│ + extensions: │ normalizeEmphasisAST │
76+
│ · magicBlock │ magicBlockTransformer │
77+
│ · legacyVariable │ imageTransformer │
78+
│ · looseHtmlEntity │ defaultTransformers │
79+
│ · mdxExprTextOnly │ (callouts, codeTabs, │
80+
│ │ gemoji, embeds) │
81+
│ + fromMarkdown: │ mdxishComponentBlocks │
82+
│ · magicBlock │ restoreSnakeCaseComponentNames │
83+
│ · legacyVariable │ mdxishTables │
84+
│ · emptyTaskList… │ mdxishHtmlBlocks │
85+
│ · looseHtmlEntity │ mdxishJsxToMdast? │
86+
│ · mdxExpression… │ variablesTextTransformer │
87+
│ │ tailwindTransformer? │
88+
│ │ remarkGfm │
89+
│ │ │
90+
└───────────────────────┴──────────────────────────────────┘
91+
```
92+
93+
## `mdxish()`
94+
95+
### Preprocessing Step
96+
97+
> **See**: @lib/mdxish.ts#209-212
98+
99+
These three lines are a protect-strip-restore pattern that removes JSX comments (`{/* ... */}`) from the markdown before anything else processes it. Here's the step-by-step:
100+
101+
1. **`protectCodeBlocks(mdContent)`**
102+
103+
Replaces fenced code blocks and inline code with placeholder tokens (`___CODE_BLOCK_0___`, `___INLINE_CODE_0___`), stashing the originals in arrays. This prevents the next step from stripping things that look like JSX comments but are actually inside code.
104+
2. **`removeJSXComments(protectedContent)`**
105+
106+
Strips all JSX comment expressions from the (now code-protected) string via a single regex. With code blocks safely out of the way, this only hits actual JSX comments in prose/component markup.
107+
3. **`restoreCodeBlocks(withoutComments, protectedCode)`**
108+
109+
Swaps the placeholder tokens back to their original code content, yielding the final `contentWithoutComments` string.
110+
111+
##### Why it's necessary
112+
113+
JSX comments are valid in MDX but have no meaning in the rendered output. If left in, they'd be parsed by the MDX expression tokenizer (the `mdxExprTextOnly` micromark extension) as expression nodes and could appear as literal text or cause parse errors. Stripping them at the string level — before `mdxishAstProcessor` and `preprocessContent` even run — is the simplest way to ensure they're gone.
114+
115+
##### Where it sits in the flow
116+
117+
This runs in `mdxish()` before calling `mdxishAstProcessor`, making it the very first string-level transform — even before `preprocessContent`:
118+
119+
```
120+
mdContent (raw input)
121+
122+
123+
┌──────────────────────────────┐
124+
│ protectCodeBlocks │ ◄── lines 209-212
125+
│ removeJSXComments │ (in mdxish())
126+
│ restoreCodeBlocks │
127+
└──────────────┬───────────────┘
128+
│ contentWithoutComments
129+
130+
┌──────────────────────────────┐
131+
│ preprocessContent │ ◄── inside mdxishAstProcessor()
132+
│ normalizeTableSeparator │
133+
│ terminateHtmlFlowBlocks │
134+
│ preprocessJSXExpressions │
135+
│ processSnakeCaseComponent │
136+
└──────────────┬───────────────┘
137+
│ parserReadyContent
138+
139+
remarkParse → transformers → ...
140+
```
141+
142+
### Processor Pipeline
143+
144+
> **See**: @lib/mdxish.ts#214-239
145+
146+
```
147+
| ................ process (parse + run only) ................. |
148+
| .. parse ........... | .............. run ................... |
149+
150+
NO COMPILER
151+
+--------+ +----------+ (HAST obj
152+
Input ->- | Parser | ->- Syntax Tree ->- | N/A | returned
153+
+--------+ | +----------+ directly)
154+
| X
155+
| |
156+
| +--------------+
157+
| | Transformers |
158+
| +--------------+
159+
| |
160+
| |
161+
┌────────────┘ ┌───┴───────────────────────────────────────────┐
162+
│ │ │
163+
│ PARSER │ MDAST TRANSFORMERS HAST XFORMERS │
164+
│ (micromark) │ (remark plugins) (rehype) │
165+
│ │ │
166+
│ remarkParse │ remarkFrontmatter preserveBool… │
167+
│ + extensions: │ normalizeEmphasisAST rehypeRaw │
168+
│ · magicBlock │ magicBlockTransformer restoreBool… │
169+
│ · legacyVariable │ imageTransformer rehypeFlatten… │
170+
│ · looseHtmlEntity │ defaultTransformers mdxishMermaid… │
171+
│ · mdxExprTextOnly │ (callouts, codeTabs, generateSlug… │
172+
│ │ gemoji, embeds) rehypeMdxish… │
173+
│ + fromMarkdown: │ mdxishComponentBlocks │
174+
│ · magicBlock │ restoreSnakeCase… ▲ │
175+
│ · legacyVariable │ mdxishTables │ │
176+
│ · emptyTaskList… │ mdxishHtmlBlocks │ │
177+
│ · looseHtmlEntity │ mdxishJsxToMdast? │ bridge: │
178+
│ · mdxExpression… │ variablesTextTransformer │ remarkRehype │
179+
│ │ tailwindTransformer? │ (MDAST → HAST) │
180+
│ │ remarkGfm │ │
181+
│ │ evaluateExpressions? │ │
182+
│ │ remarkBreaks │ │
183+
│ │ variablesCodeResolver ───┘ │
184+
│ │ │
185+
└───────────────────────┴───────────────────────────────────────────────┘
186+
```
Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# MDXish Supported Syntax
2+
3+
## Custom Blocks
4+
5+
### Code Tabs
6+
7+
Tabbed interface for multiple code blocks - written as immediately consecutive standard code blocks (i.e. **without** any line breaks between them).
8+
9+
```js Title One
10+
console.log('Tab One');
11+
```
12+
```js Title Two
13+
console.log('Tab Two');
14+
```
15+
16+
### Callouts
17+
18+
Blockquotes that start with an emoji are rendered determine the theme:
19+
20+
> 👍 Success
21+
>
22+
> Your success message here
23+
24+
#### Supported Themes
25+
26+
- **Info**: 📘 or ℹ️ (blue)
27+
- **Success**: 👍 or ✅ (green)
28+
- **Warning**: 🚧 or ⚠️ (orange)
29+
- **Error**: ❗️ or 🛑 (red)
30+
- **Default**: any other emoji (gray)
31+
32+
### Embeds
33+
34+
Simple markdown link with `@embed` title:
35+
36+
[Embed Title](https://youtu.be/example "@embed")
37+
38+
## Data Replacement Syntaxes
39+
40+
### User Variables
41+
42+
Double angle-bracket notation for JWT login variables:
43+
44+
Hi, my name is **<<name>>**!
45+
46+
### Glossary Terms
47+
48+
Double angle-brackets with `glossary:` prefix:
49+
50+
**<<glossary:exogenous>>** and **<<glossary:endogenous>>**
51+
52+
## MDX Syntax
53+
54+
A subset of MDX syntax is supported.
55+
56+
### Custom Components
57+
You can embed React components or reusable Markdown snippets in a document using JSX elements:
58+
59+
<MyComponent prop="value" />
60+
61+
### Logical Expressions
62+
63+
Simple logic is also supported using the JSX-style curly brace syntax:
64+
65+
{(4 * 3) / 2} of 1, a half dozen of another.
66+
67+
This expression syntax can also be used as an alternative for user variables:
68+
69+
Hi, my name is **{user.name}**!
70+
71+
## Standard Markdown Extensions
72+
73+
Full **CommonMark** and **GitHub-flavored Markdown** support, including:
74+
75+
### Emoji Shortcodes
76+
77+
GitHub-style emoji codes:
78+
79+
:sparkles:
80+
81+
### Tables
82+
83+
GFM-style tables with alignment support:
84+
85+
| Left | Center | Right |
86+
|:-----|:--------:|------:|
87+
| L0 | **bold** | $1600 |
88+
89+
### Lists
90+
91+
Standard bulleted (`-` or `*`) and numbered lists (`1.`, `2.`, etc.) are supported, as well as GFM-style checklists:
92+
93+
```md
94+
- [x] finished item
95+
- [ ] unfinished item
96+
```
97+
98+
### Headings
99+
100+
Standard Markdown heading syntaxes (`#` prefixes) are supported, as well as compact and ATX-wrapped variations:
101+
102+
##Compact Heading without a space
103+
104+
## ATX-Style Wrapped Heading ##
105+
106+
Underline notation (using `=` or `-` are also supported for first and second level headings, respectively.
107+
108+
## Legacy Magic Blocks
109+
110+
The engine also supports the legacy JSON-based "magic block" syntax for backwards compatibility.
111+
112+
[block:api-header]
113+
{
114+
"title": "Section Title"
115+
}
116+
[/block]
117+
118+
This is a legacy format that should be transpiled to newer ReadMe-flavored syntax. Supported magic blocks include:
119+
120+
| Feature | Magic Block Name |
121+
|---------------|------------------------|
122+
| Heading | `[block:api-header]` |
123+
| Callout | `[block:callout]` |
124+
| Embed | `[block:embed]` |
125+
| Custom HTML | `[block:html]` |
126+
| Image | `[block:image]` |
127+
| Table | `[block:parameters]` |
128+
129+
## Additional Features
130+
131+
- **Auto-generated heading anchors** with incremental IDs for duplicate headings
132+
- **Table of Contents generation** from markup
133+
- **Custom `doc:` and `ref:` protocols** for internal documentation links
134+
- **Both JSX and HTML comments** for non-rendered notes and annotations

CLAUDE.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# `@readme/markdown`
2+
3+
This repo contains two Markdown processing engines, both built on top of Unified.js + Remark.
4+
5+
## RMDX
6+
7+
A strict MDX processor written on top of Unified.js + Remark. RMDX handles standard Markdown + GFM, as well as ReadMe's flavored custom syntax. Because it is an MDX-first processor, all RMDX-processed docs must adhere to strict JSX syntax rules. (See @lib/mdx.ts)
8+
9+
## MDXish (aka Xish)
10+
11+
The Xish processor supports standard Markdown with GFM extensions, as well as ReadMe's flavored syntax. This engine also supports a subset of MDX functionality (specifically custom components and logical expressions) without requiring strict JSX compliance. (See @lib/mdxish.ts)
12+
13+
### Further Context
14+
15+
- @.claude/context/MDXish/Processor Overview.md
16+
- @.claude/context/MDXish/Supported Syntax.md

0 commit comments

Comments
 (0)