Skip to content

Parsing markdown with mix of ordered and unordered list runs into RecursionError #356

Closed
Closed
Parsing markdown with mix of ordered and unordered list runs into RecursionError#356
Labels
bug
@ketanbhatt

Description

@ketanbhatt
ketanbhatt
opened on Sep 21, 2023

Hello 👋🏼

I am using mistune==3.0.1.

When I try to parse a markdown that is one big list of ordered and unordered list, mistune runs into Recursion Error. I suspect it might be taking it as one big nested list.

Here is a sample markdown:

- Fruits
1. Apple
2. Orange

- Vegetables
1. Eggplant
2. Onion

If this markdown continued like this to have more lines than the sys.getrecursionlimit(), parsing this markdown would result in a recursion error.

Here is some code that reproduces the error:

import mistune
import sys

list_to_repeat = """
- Fruits
1. Apple
2. Orange
"""

breaking_md = ""

# Get number of list items such that we hit the recursion limit
repeat_times = sys.getrecursionlimit() // len(list_to_repeat.splitlines()) + 10
print(f"Recursion Limit: {sys.getrecursionlimit()}")
print(f"Repeat times: {repeat_times}")

for idx in range(repeat_times):
    breaking_md += list_to_repeat

print("\nTesting breaking markdown...")
print(f"Number of lines in breaking markdown: {len(breaking_md.splitlines())}")

try:
    mistune.create_markdown(renderer="ast")(breaking_md)
except RecursionError as exc:
    print(f"RecursionError: {exc}")


non_breaking_md = ""

for idx in range(repeat_times):
    non_breaking_md += list_to_repeat
    non_breaking_md += "\n---\n"

print("\nTesting non-breaking markdown...")
print(f"Number of lines in non-breaking markdown: {len(non_breaking_md.splitlines())}")

mistune.create_markdown(renderer="ast")(non_breaking_md)
print("No error")

Output:

Recursion Limit: 1000
Repeat times: 260

Testing breaking markdown...
Number of lines in breaking markdown: 1040
RecursionError: maximum recursion depth exceeded while calling a Python object

Testing non-breaking markdown...
Number of lines in non-breaking markdown: 1560
No error
Here is the stacktrace when the recursion error is hit

I set the recursion limit to 100 so I can get a smaller full stacktrace. The stacktrace is similar with limit 1000.

Traceback (most recent call last):
  File "/Users/ketanbhatt/Library/Application Support/JetBrains/PyCharm2023.1/scratches/scratch_3.py", line 24, in <module>
    mistune.create_markdown(renderer="ast")(breaking_md)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/markdown.py", line 110, in __call__
    return self.parse(s)[0]
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/markdown.py", line 85, in parse
    self.block.parse(state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 446, in parse
    end_pos = self.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 150, in _parse_list_item
    end_pos = block.parse_method(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/core.py", line 168, in parse_method
    return func(m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/block_parser.py", line 379, in parse_list
    return parse_list(self, m, state)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 63, in parse_list
    groups = _parse_list_item(block, bullet, groups, token, state, rules)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 91, in _parse_list_item
    text, continue_width = _compile_continue_width(text, leading_width)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/list_parser.py", line 205, in _compile_continue_width
    text = expand_tab(text)
  File "/Users/ketanbhatt/.venv/lib/python3.8/site-packages/mistune/util.py", line 18, in expand_tab
    return _expand_tab_re.sub(repl, text)
RecursionError: maximum recursion depth exceeded while calling a Python object

Thank you for your work : )

Metadata

Metadata

Assignees

No one assigned

    Labels

    bug

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions