Describe the issue

It seems that syntax highlighting breaks if you reconfigure the editor's language too many times.

To reproduce: run the application in my sample repo with the developer console open. (edit: or just run the sandbox script provided at the end of this description)

Looking at https://github.com/robbertbrak/cm-stream-language-parsing/blob/main/src/App.js, you can see that it defines a trivial stream language that is able to parse the string hello world into three tokens (and marks everything else as invalid). I render a codemirror instance with this language and the string hello world, and subsequently reconfigure the language in a loop, until the resulting DOM is no longer correct.

The expected content DOM is

<div class="cm-activeLine cm-line"><span class="cm-hello">hello</span><span class="cm-space"> </span><span class="cm-world">world</span></div>

On my machine, it breaks on the 16381th iteration and outputs the incorrect:

<div class="cm-activeLine cm-line">hello world</div>

The fact that it is always the 16381th iteration (even across browsers) suggests that there is some kind of overflow somewhere deep in codemirror.

Some observations I made:

• If you change the size of the token table, it fails at a different iteration
• Destroying and re-creating the codemirror instance between iterations makes no difference: the above issue still occurs after a certain predictable amount of iterations.

As to why you would want to reconfigure the language so often:

• For the use case described in https://discuss.codemirror.net/t/re-parse-existing-content/7381
• Also, in Javascript specs.

Browser and platform

No response

Reproduction link

codemirror sandbox script