|
| 1 | +# MDXish Processor Overview |
| 2 | + |
| 3 | +## `mdxishAstProcessor()` |
| 4 | + |
| 5 | +### Preprocessing Step |
| 6 | + |
| 7 | +> **See**: @lib/mdxish.ts#92-103 |
| 8 | +
|
| 9 | +`preprocessContent` is a string-level preprocessor that runs before the markdown is handed to remarkParse. It exists because several syntactic patterns in ReadMe's flavor of markdown would confuse or break the standard CommonMark/MDX parser if fed to it directly. By patching the raw string first, these issues are sidestepped. |
| 10 | + |
| 11 | +It applies four transforms in sequence: |
| 12 | + |
| 13 | +1. **`normalizeTableSeparator()`** |
| 14 | + |
| 15 | + Fixes malformed GFM table separator rows — e.g. misplaced alignment colons like `|: ---` → `| :---`. Without this, remarkGfm would fail to recognize the table. |
| 16 | +1. **`terminateHtmlFlowBlocks()`** |
| 17 | + |
| 18 | + Inserts blank lines after standalone HTML elements (like `<div>...</div>`) when the next line is regular markdown. CommonMark's HTML flow rules only terminate on blank lines, so without this, the parser would swallow subsequent markdown content into the HTML block token. |
| 19 | +1. **`preprocessJSXExpressions()`** (skipped in safeMode) |
| 20 | + |
| 21 | + Handles JSX attribute expressions (`href={someVar}`) and unbalanced braces before the MDX expression tokenizer sees them. It evaluates attribute expressions against jsxContext, converts style objects to CSS strings, and escapes stray braces that would cause MDX parse errors. |
| 22 | +1. **`processSnakeCaseComponent()`** |
| 23 | + |
| 24 | + Remark's parser rejects tag names containing underscores (e.g. `<my_component>`). This step replaces known snake_case component names with safe placeholder names (`<MDXishSnakeCase0>`) and returns a mapping so they can be restored later by the `restoreSnakeCaseComponentNames` transformer in the run phase. |
| 25 | + |
| 26 | +##### Where it sits in the flow |
| 27 | + |
| 28 | +``` |
| 29 | + preprocessContent (string → string) |
| 30 | + │ |
| 31 | + ▼ |
| 32 | +┌─────────────────────────────────────────────────────┐ |
| 33 | +│ normalizeTableSeparator — fix table syntax │ |
| 34 | +│ terminateHtmlFlowBlocks — fix HTML flow │ |
| 35 | +│ preprocessJSXExpressions — eval/escape JSX │ before |
| 36 | +│ processSnakeCaseComponent — placeholder swap │ parsing |
| 37 | +└─────────────────────────────┬───────────────────────┘ |
| 38 | + │ |
| 39 | + ▼ |
| 40 | + remarkParse (tokenize) |
| 41 | + │ |
| 42 | + ▼ |
| 43 | + MDAST transformers... |
| 44 | + ... |
| 45 | + restoreSnakeCaseComponentNames ◄── undo (4) |
| 46 | +``` |
| 47 | + |
| 48 | +### Processor Pipeline |
| 49 | + |
| 50 | +> **See**: @lib/mdxish.ts#105-178 |
| 51 | +
|
| 52 | +The core Xish engine which parses Markdown and converts it to an MDAST object. This is the base processor used for both the editor and rendering flows. |
| 53 | + |
| 54 | +``` |
| 55 | +| ................ process (parse only) ...................... | |
| 56 | +| .. parse ........... | .............. run .................. | |
| 57 | +
|
| 58 | + NO COMPILER |
| 59 | + +--------+ +----------+ (MDAST is |
| 60 | +Input ->- | Parser | ->- Syntax Tree ->- | N/A | returned |
| 61 | + +--------+ | +----------+ directly) |
| 62 | + | | |
| 63 | + | X |
| 64 | + | | |
| 65 | + | +--------------+ |
| 66 | + | | Transformers | |
| 67 | + | +--------------+ |
| 68 | + | | |
| 69 | + ┌────────────┘ ┌───┴──────────────────────────────┐ |
| 70 | + │ │ │ |
| 71 | + │ PARSER │ MDAST TRANSFORMERS │ |
| 72 | + │ (micromark) │ (remark plugins) │ |
| 73 | + │ │ │ |
| 74 | + │ remarkParse │ remarkFrontmatter │ |
| 75 | + │ + extensions: │ normalizeEmphasisAST │ |
| 76 | + │ · magicBlock │ magicBlockTransformer │ |
| 77 | + │ · legacyVariable │ imageTransformer │ |
| 78 | + │ · looseHtmlEntity │ defaultTransformers │ |
| 79 | + │ · mdxExprTextOnly │ (callouts, codeTabs, │ |
| 80 | + │ │ gemoji, embeds) │ |
| 81 | + │ + fromMarkdown: │ mdxishComponentBlocks │ |
| 82 | + │ · magicBlock │ restoreSnakeCaseComponentNames │ |
| 83 | + │ · legacyVariable │ mdxishTables │ |
| 84 | + │ · emptyTaskList… │ mdxishHtmlBlocks │ |
| 85 | + │ · looseHtmlEntity │ mdxishJsxToMdast? │ |
| 86 | + │ · mdxExpression… │ variablesTextTransformer │ |
| 87 | + │ │ tailwindTransformer? │ |
| 88 | + │ │ remarkGfm │ |
| 89 | + │ │ │ |
| 90 | + └───────────────────────┴──────────────────────────────────┘ |
| 91 | +``` |
| 92 | + |
| 93 | +## `mdxish()` |
| 94 | + |
| 95 | +### Preprocessing Step |
| 96 | + |
| 97 | +> **See**: @lib/mdxish.ts#209-212 |
| 98 | +
|
| 99 | +These three lines are a protect-strip-restore pattern that removes JSX comments (`{/* ... */}`) from the markdown before anything else processes it. Here's the step-by-step: |
| 100 | + |
| 101 | +1. **`protectCodeBlocks(mdContent)`** |
| 102 | + |
| 103 | + Replaces fenced code blocks and inline code with placeholder tokens (`___CODE_BLOCK_0___`, `___INLINE_CODE_0___`), stashing the originals in arrays. This prevents the next step from stripping things that look like JSX comments but are actually inside code. |
| 104 | +2. **`removeJSXComments(protectedContent)`** |
| 105 | + |
| 106 | + Strips all JSX comment expressions from the (now code-protected) string via a single regex. With code blocks safely out of the way, this only hits actual JSX comments in prose/component markup. |
| 107 | +3. **`restoreCodeBlocks(withoutComments, protectedCode)`** |
| 108 | + |
| 109 | + Swaps the placeholder tokens back to their original code content, yielding the final `contentWithoutComments` string. |
| 110 | + |
| 111 | +##### Why it's necessary |
| 112 | + |
| 113 | +JSX comments are valid in MDX but have no meaning in the rendered output. If left in, they'd be parsed by the MDX expression tokenizer (the `mdxExprTextOnly` micromark extension) as expression nodes and could appear as literal text or cause parse errors. Stripping them at the string level — before `mdxishAstProcessor` and `preprocessContent` even run — is the simplest way to ensure they're gone. |
| 114 | + |
| 115 | +##### Where it sits in the flow |
| 116 | + |
| 117 | +This runs in `mdxish()` before calling `mdxishAstProcessor`, making it the very first string-level transform — even before `preprocessContent`: |
| 118 | + |
| 119 | +``` |
| 120 | +mdContent (raw input) |
| 121 | + │ |
| 122 | + ▼ |
| 123 | +┌──────────────────────────────┐ |
| 124 | +│ protectCodeBlocks │ ◄── lines 209-212 |
| 125 | +│ removeJSXComments │ (in mdxish()) |
| 126 | +│ restoreCodeBlocks │ |
| 127 | +└──────────────┬───────────────┘ |
| 128 | + │ contentWithoutComments |
| 129 | + ▼ |
| 130 | +┌──────────────────────────────┐ |
| 131 | +│ preprocessContent │ ◄── inside mdxishAstProcessor() |
| 132 | +│ normalizeTableSeparator │ |
| 133 | +│ terminateHtmlFlowBlocks │ |
| 134 | +│ preprocessJSXExpressions │ |
| 135 | +│ processSnakeCaseComponent │ |
| 136 | +└──────────────┬───────────────┘ |
| 137 | + │ parserReadyContent |
| 138 | + ▼ |
| 139 | + remarkParse → transformers → ... |
| 140 | +``` |
| 141 | + |
| 142 | +### Processor Pipeline |
| 143 | + |
| 144 | +> **See**: @lib/mdxish.ts#214-239 |
| 145 | +
|
| 146 | +``` |
| 147 | +| ................ process (parse + run only) ................. | |
| 148 | +| .. parse ........... | .............. run ................... | |
| 149 | +
|
| 150 | + NO COMPILER |
| 151 | + +--------+ +----------+ (HAST obj |
| 152 | +Input ->- | Parser | ->- Syntax Tree ->- | N/A | returned |
| 153 | + +--------+ | +----------+ directly) |
| 154 | + | X |
| 155 | + | | |
| 156 | + | +--------------+ |
| 157 | + | | Transformers | |
| 158 | + | +--------------+ |
| 159 | + | | |
| 160 | + | | |
| 161 | + ┌────────────┘ ┌───┴───────────────────────────────────────────┐ |
| 162 | + │ │ │ |
| 163 | + │ PARSER │ MDAST TRANSFORMERS HAST XFORMERS │ |
| 164 | + │ (micromark) │ (remark plugins) (rehype) │ |
| 165 | + │ │ │ |
| 166 | + │ remarkParse │ remarkFrontmatter preserveBool… │ |
| 167 | + │ + extensions: │ normalizeEmphasisAST rehypeRaw │ |
| 168 | + │ · magicBlock │ magicBlockTransformer restoreBool… │ |
| 169 | + │ · legacyVariable │ imageTransformer rehypeFlatten… │ |
| 170 | + │ · looseHtmlEntity │ defaultTransformers mdxishMermaid… │ |
| 171 | + │ · mdxExprTextOnly │ (callouts, codeTabs, generateSlug… │ |
| 172 | + │ │ gemoji, embeds) rehypeMdxish… │ |
| 173 | + │ + fromMarkdown: │ mdxishComponentBlocks │ |
| 174 | + │ · magicBlock │ restoreSnakeCase… ▲ │ |
| 175 | + │ · legacyVariable │ mdxishTables │ │ |
| 176 | + │ · emptyTaskList… │ mdxishHtmlBlocks │ │ |
| 177 | + │ · looseHtmlEntity │ mdxishJsxToMdast? │ bridge: │ |
| 178 | + │ · mdxExpression… │ variablesTextTransformer │ remarkRehype │ |
| 179 | + │ │ tailwindTransformer? │ (MDAST → HAST) │ |
| 180 | + │ │ remarkGfm │ │ |
| 181 | + │ │ evaluateExpressions? │ │ |
| 182 | + │ │ remarkBreaks │ │ |
| 183 | + │ │ variablesCodeResolver ───┘ │ |
| 184 | + │ │ │ |
| 185 | + └───────────────────────┴───────────────────────────────────────────────┘ |
| 186 | +``` |
0 commit comments