For #2070 I had to look at the tests of TokenStreamTransformer.simplify. And from these tests, a token stream is defined as:

type TokenStream = Array<string | Token | TokenStream>;

interface Token {
    type: string;
    content: string | Token | TokenStream;
    // some other properties
}

Yes, nested token streams.

But I always assumed that a token stream looks like this:

type TokenStream = Array<string | Token>;
// Token same as above

And I even added the condition that a token stream doesn't contain adjacent strings. I assumed this because the token stream produced by tokenize has these properties and because these properties are very useful.

Point is: I wrote code with these assumptions, so do I have to rewrite it?

What is the correct definition of a token stream?

/cc @Golmote @mAAdhaTTah