I confirmed that \w matches all meaningful identifiers:

import re
import sys
import unicodedata

cre = re.compile(r'\w')
for cp in range(sys.maxunicode + 1):
    s = chr(cp)
    if s.isidentifier() and not re.match(cre, s):
        print(hex(cp), unicodedata.name(s))

Valid identifiers not matched by \w:

0x1885 MONGOLIAN LETTER ALI GALI BALUDA
0x1886 MONGOLIAN LETTER ALI GALI THREE BALUDA
0x2118 SCRIPT CAPITAL P
0x212e ESTIMATED SYMBOL

Beyond that, it appears that the clauses for the try-except at the start are backwards, causing python 2 to raise and python 3 to not have unicode identifier support.

Need to add a test to this one but I fixed the reverse thing and added characters missing. Oddly enough I can only reproduce \w not matching on the latter two but not the mongolian characters.

Need to also check not keyword.iskeyword(value) to catch things like True and class which str.isidentifier returns true for.

Or maybe not, since they'll raise SyntaxError later if used improperly.

env.from_string('{{ ℮ }}') (or any of the four special case characters) raises TemplateSyntaxError: unexpected char '℮' at 3. Other Unicode characters work as expected.

Pushed a simple test, currently fails for the four special case characters.

The regex is specifically failing on the \b boundaries, if they are removed the special cases work. This is because \b means "transition from \w to \W", and the special cases are not in \w.

I wonder if that was not already broken before.

Just checked, e.from_string('{{ ℮ }}') works on master.

I don't think the \b is necessary, it's not present in the previous stringdef regex. All tests still pass with the boundaries removed.

There were boundaries on the Python 2 regex, but they don't seem necessary either.

I reported the \w issue to Python: http://bugs.python.org/issue30838

@lf- thanks for helping figure this out.

Writing more tests based on the previous stringdef, https://raw.githubusercontent.com/pallets/jinja/2.9.5/jinja2/_stringdefs.py, turns up some invalid characters in the previous regex. The following start characters were matched but were not valid as start or continue characters:

\u309B KATAKANA-HIRAGANA VOICED SOUND MARK
\u309C KATAKANA-HIRAGANA SEMI-VOICED SOUND MARK

The following were matched and are valid continue characters, but are no longer matched by \w:

\u309B MIDDLE DOT
\u309C GREEK ANO TELEIA

Then I realized we were missing a test in that script: ('a' + c).isidentifier() for characters that are valid for continue but not start. That reveals a lot more that aren't matched by \w.

2097 characters not matched. 😞

Going to have to bring back _stringdefs, but still simplified.

Original _stringdefs:

         len   sizeof
start    48194 96462
continue 49826 99726
total    98020 196188

New _identifier:

len: 635
sizeof: 2616

Timing is hard to test, but on my machine re.purge(); re.compile(...) is down from ~100 ms to ~840 µs.

Strategy was to collapse contiguous ranges and rely on str.isidentifier to validate so that the regex is simpler.

That sounds good. Ideally we can reuse the same thing as we had before to regenerate the regex though we need to make sure we hit the set that is missing in all versions of Python 3.

At the cost of about twice as much space, the regex could be made more accurate by omitting \w and generating the full range of valid characters. At the cost of twice again as much space, we could remove the need for calling str.isidentifer during lexing by distinguishing start and continue characters in the regex. For now, I'm happy with where we are and will leave \w and isidentifier in.