lexer SQL to use whitespace token - regarding #1905 #1908
Conversation
|
Thanks for preparing this change! On the SQLite prompt: I'd like to see it produce whitespace and not put it into the prompt itself, similar to the PgSQL prompt. I assume if you add manual whitespace after the prompt it will actually highlight it as whitespace? In which case all whitespace should have been marked as such. |
|
The two additional commits deal with the sqlite prompt, Thanks to the well prepared test file, i caught a bug I'd introduce with the second commit only. Leaving the trailing whitespace "in the open" for further lexing, it could result in grouping the space with other long running tokens, such as string literals in the test file. |
|
OK! Now the sqlite diff looks much more as expected. This should complete it. |
|
Merged, thanks a lot! Output looks indeed quite nice! |
This PR specifes whitespace-only matches with the respective Text.Whitespace token.
It is submitted in the context of #1905. The changes are minimal, and related test files have been adapted.
Anyhow, I noticed a possible inconsitency and would like a second opinion on how to deal with it.
While the following regex does not include the trailing white space, the sqlite prompt does.
pygments/pygments/lexers/sql.py
Line 266 in 339ad2b
pygments/pygments/lexers/sql.py
Lines 786 to 801 in 339ad2b
In this state there are Prompt Token lexed such as tests/examplefiles/psql/psql_session.txt.output line 27 (matched by the re_prompt regex) which are trailed by a Whitespace token. However, the
sqlite>prompt does include the trailing space in the token group (matched via the startswith expressions). The latter has an example in tests/examplefiles/sqlite3/sqlite3.sqlite3-console.output line 98 (at the end of the diff)