add ast output

quantizor · quantizor · commit 450d2bb113e2 · 2025-10-26T01:50:29.000-04:00
diff --git a/.changeset/ast-option.md b/.changeset/ast-option.md
@@ -0,0 +1,31 @@
+---
+'markdown-to-jsx': minor
+---
+
+Added `ast` option to compiler to expose the parsed AST directly. When `ast: true`, the compiler returns the AST structure (`ParserResult[]`) instead of rendered JSX.
+
+**First time the AST is accessible to users!** This enables:
+
+- AST manipulation and transformation before rendering
+- Custom rendering logic without parsing
+- Caching parsed AST for performance
+- Linting or validation of markdown structure
+
+**Usage:**
+
+```typescript
+import { compiler } from 'markdown-to-jsx'
+import type { MarkdownToJSX } from 'markdown-to-jsx'
+
+// Get the AST structure
+const ast: MarkdownToJSX.AST[] = compiler('# Hello world', {
+  ast: true,
+})
+
+// Inspect/modify AST
+console.log(ast) // Array of parsed nodes
+
+// Render AST to JSX using createRenderer (not implemented yet)
+```
+
+The AST format is `MarkdownToJSX.AST[]`. When footnotes are present, the returned value will be an object with `ast` and `footnotes` properties instead of just the AST array.
diff --git a/Performance Ideas.md b/Performance Ideas.md
@@ -327,10 +327,15 @@ state.prevCapture = captureParts.join('')
 - ✅ One allocation instead of many
 - ❌ Array overhead
 - ❌ Extra joins
+- ❌ **Doesn't work with recursive parsing**
 
 **Complexity**: Medium - refactor concatenation points
 **Expected Gain**: 10-15% reduction
 
+**Attempted**: Failed because nested parsing relies on incremental updates to `state.prevCapture`. The buffer approach doesn't update the state during nested parsing, breaking lookback functionality used by list items and other nested rules.
+
+**Status**: ❌ Not viable due to recursive parsing requirements
+
 ---
 
 #### Idea G: Avoid String Operations Entirely Where Possible
@@ -1047,10 +1052,71 @@ The performance gap is from **fundamental architectural differences**:
 
 ### Potential Improvements (Without Full Rewrite)
 
-1. **Token Abstraction**: Parse to intermediate token format, render separately
-2. **Lazy JSX Creation**: Generate JSX only at render time
-3. **Memoization**: Cache token trees, regenerate JSX from tokens
-4. **String Reduction**: Use token positions instead of substring calls
+1. ✅ **AST Exposure**: Parse to AST and expose it (IMPLEMENTED - Feature)
+2. **Profile React Rendering**: Measure time spent in React.createElement
+3. **Optimize Rendering**: Reduce JSX creation overhead
+4. **Memoization**: Cache rendered JSX for repeated ASTs
+5. ~~**String Reduction**: Use token positions instead of substring calls~~ (❌ Attempted - incompatible with recursive parsing)
+
+### Next Steps for Performance:
+
+**Phase 1: Measure rendering performance**
+
+- Profile `render()` function separately
+- Measure React.createElement overhead
+- Determine parsing vs rendering split
+
+**Phase 2: Optimize based on findings**
+
+- If rendering is the bottleneck: optimize JSX creation
+- If parsing is the bottleneck: focus on parsing optimizations (already done)
+- If memory is the bottleneck: reduce allocations during rendering
+
+## Next: Profile React Rendering Performance
+
+We've focused heavily on parsing performance, but **we haven't profiled the React rendering phase yet**.
+
+### Questions to Answer:
+
+1. **How much time is spent in React.createElement()?**
+
+   - Each AST node creates JSX elements
+   - For a 27KB document, we might have hundreds of React.createElement calls
+   - Is this a bottleneck?
+
+2. **What's the overhead of JSX creation?**
+
+   - Every AST node → JSX element conversion
+   - Component overhead vs plain objects
+   - React reconciliation preparation
+
+3. **Can we optimize rendering?**
+   - Memoization opportunities
+   - Reducing object allocations during rendering
+   - Optimizing the render function
+
+### How to Profile Rendering:
+
+```javascript
+// Profile rendering separately from parsing
+const ast = compiler(markdown, { ast: true })
+
+// Measure rendering time
+const t0 = performance.now()
+const jsx = renderAST(ast) // Use createRenderer
+const t1 = performance.now()
+
+console.log('Render time:', t1 - t0, 'ms')
+```
+
+We need to:
+
+1. Add profiling hooks to the `render()` function
+2. Measure time spent in `React.createElement`
+3. Measure memory allocations during rendering
+4. Compare parsing time vs rendering time
+
+**Hypothesis**: React rendering might be a significant portion of the 14ms total time.
 
 ### Should We Rewrite?
 
diff --git a/README.md b/README.md
@@ -25,6 +25,7 @@ The most lightweight, customizable React markdown component.
     - [options.namedCodesToUnicode](#optionsnamedcodestounicode)
     - [options.disableAutoLink](#optionsdisableautolink)
     - [options.disableParsingRawHTML](#optionsdisableparsingrawhtml)
+    - [options.ast](#optionsast)
   - [Syntax highlighting](#syntax-highlighting)
   - [Handling shortcodes](#handling-shortcodes)
   - [Getting the smallest possible bundle size](#getting-the-smallest-possible-bundle-size)
@@ -573,6 +574,32 @@ compiler('This text has <span>html</span> in it but it won't be rendered', { dis
 <span>This text has &lt;span&gt;html&lt;/span&gt; in it but it won't be rendered</span>
 ```
 
+#### options.ast
+
+When `ast: true`, the compiler returns the parsed AST structure instead of rendered JSX. **This is the first time the AST is accessible to users!**
+
+```tsx
+import { compiler } from 'markdown-to-jsx'
+import type { MarkdownToJSX } from 'markdown-to-jsx'
+
+// Get the AST directly
+const ast = compiler('# Hello world', { ast: true })
+
+// TypeScript: AST is MarkdownToJSX.AST[]
+console.log(ast) // Array of parsed nodes with types
+
+// You can manipulate, transform, or analyze the AST before rendering
+```
+
+The AST format is `MarkdownToJSX.AST[]` and enables:
+
+- AST manipulation and transformation
+- Custom rendering logic without re-parsing
+- Caching parsed AST for performance
+- Linting or validation of markdown structure
+
+When footnotes are present, the returned value will be an object with `ast` and `footnotes` properties instead of just the AST array.
+
 ### Syntax highlighting
 
 When using [fenced code blocks](https://www.markdownguide.org/extended-syntax/#syntax-highlighting) with language annotation, that language will be added to the `<code>` element as `class="lang-${language}"`. For best results, you can use `options.overrides` to provide an appropriate syntax highlighting integration like this one using `highlight.js`:
