I ran some performance test with

• three different parsers: handwritten, tatsu, and regex-based
• reading the input string from memory or harddisk
• line-wise iterating over the input line-wise or reading it as one big string
• using the built-in re module (only for handwritten and tatsu) or using the extended regex module
• using CPython or PyPy
• using a precompiled tatsu grammar or loading it from the ebnf file during runtime.

The results can be summarized as follows:

• Tatsu is by far slower than the others. The fastest run was line-wise reading using a precompiled grammar on PyPy, which took about 7 seconds. Switching to CPython or reading the file as a whole each add an order of magnitude to that (yep, the fastest tatsu run on CPython was 49s), while not precompiling only adds a minor overhead. The other parsers all took less than 5s no matter what interpreter was used.
• Using the built-in re module is a little bit faster than the extended regex module, while also not failing with weird unicode symbols (see the regex bug mentioned above). Similarly, the handwritten parser is a little bit faster than the one relying on the extended regex functionality.
• Reading line-wise instead of the whole file at once is marginally faster and needs a little less memory (unsurprisingly), but the numbers vary a little bit depending on whether we read from memory or from disk.

See b85e0a4 for the code used for testing. Note that I didn't test much with invalid data, but I guess the results for the good case are clear enough (and tatsu is even more prone to spending too much time on broken inputs).

Supporting three different parser implementations and giving the user the possibility to choose from them adds a lot of complexity on both sides. As the handwritten parser is faster than the other two, I decided to only ship that one to the users, drastically simplifying the code in cd97f4b. The regex parser is no longer needed and the tatsu parser has been moved to the testing code, so that we can use it as reference when parsing files from the corpus.