Skip to content

Commit 83b5491

Browse files
JakobDevJakobDev
authored
Import Luau lexer (#1209)
This imports `pygments.lexers.scripting.LuauLexer` from Pygments. Some notes: - The generated XML file is not 100% valid as as some elements have 2 `state` attributes, but it does work - I needed to tweak the import script to correctly convert the RegEx - I needed to add `OperatorReserved` as a `TokenType` - `.luaurc` is added as a filename to the JSON Lexer, as [it's used by Luau](https://rfcs.luau.org/config-luaurc.html) Implements #1062
1 parent da72a97 commit 83b5491

9 files changed

Lines changed: 363 additions & 104 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ translators for Pygments lexers and styles.
4747
| I | Idris, Igor, INI, Io, ISCdhcpd
4848
| J | J, Janet, Java, JavaScript, JSON, JSONata, Jsonnet, Julia, Jungle
4949
| K | Kakoune, Kotlin
50-
| L | Lean4, Lighttpd configuration file, LLVM, lox, Lua
50+
| L | Lean4, Lighttpd configuration file, LLVM, lox, Lua, Luau
5151
| M | Makefile, Mako, markdown, Markless, Mason, Materialize SQL dialect, Mathematica, Matlab, MCFunction, Meson, Metal, MiniZinc, MLIR, Modelica, Modula-2, Mojo, MonkeyC, MoonScript, MorrowindScript, Myghty, MySQL
5252
| N | NASM, Natural, NDISASM, Newspeak, Nginx configuration file, Nim, Nix, NSIS, Nu
5353
| O | Objective-C, ObjectPascal, OCaml, Octave, Odin, OnesEnterprise, OpenEdge ABL, OpenSCAD, Org Mode

‎_tools/pygments2chroma_xml.py‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,10 @@
5454
'''
5555

5656

57-
def xml_regex(s):
57+
def xml_regex(s: str) -> str:
58+
# Convert Python-style capture groups to .NET-style capture groups
59+
s = s.replace('(?P', '(?')
60+
5861
return xml_string(s)
5962

6063
def xml_string(s):

‎lexers/embedded/json.xml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
<filename>*.jsonc</filename>
77
<filename>*.json5</filename>
88
<filename>*.avsc</filename>
9+
<filename>.luaurc</filename>
910
<mime_type>application/json</mime_type>
1011
<dot_all>true</dot_all>
1112
<not_multiline>true</not_multiline>

‎lexers/embedded/lua.xml‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,8 @@
22
<config>
33
<name>Lua</name>
44
<alias>lua</alias>
5-
<alias>luau</alias>
65
<filename>*.lua</filename>
76
<filename>*.wlua</filename>
8-
<filename>*.luau</filename>
97
<mime_type>text/x-lua</mime_type>
108
<mime_type>application/x-lua</mime_type>
119
</config>

‎lexers/embedded/luau.xml‎

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
<lexer>
2+
<config>
3+
<name>Luau</name>
4+
<alias>luau</alias>
5+
<filename>*.luau</filename>
6+
</config>
7+
<rules>
8+
<state name="root">
9+
<rule pattern="#!.*"><token type="CommentHashbang"/><push state="base"/></rule>
10+
<rule><push state="base"/></rule>
11+
</state>
12+
<state name="ws">
13+
<rule pattern="(?:--\[(?&lt;level&gt;=*)\[[\w\W]*?\](?=level)\])"><token type="CommentMultiline"/></rule>
14+
<rule pattern="(?:--.*$)"><token type="CommentSingle"/></rule>
15+
<rule pattern="\s+"><token type="TextWhitespace"/></rule>
16+
</state>
17+
<state name="base">
18+
<rule><include state="ws"/></rule>
19+
<rule pattern="\{"><token type="Punctuation"/><push state="closing_brace_base" state="expression"/></rule>
20+
<rule pattern="\("><token type="Punctuation"/><push state="closing_parenthesis_base" state="expression"/></rule>
21+
<rule pattern="::?"><token type="Punctuation"/><push state="type_end" state="type_start"/></rule>
22+
<rule pattern="&#x27;"><token type="LiteralStringSingle"/><push state="string_single"/></rule>
23+
<rule pattern="&quot;"><token type="LiteralStringDouble"/><push state="string_double"/></rule>
24+
<rule pattern="`"><token type="LiteralStringBacktick"/><push state="string_interpolated"/></rule>
25+
<rule pattern="\.\.\."><token type="Punctuation"/></rule>
26+
<rule pattern="type\b(?=(?:(?:--\[(?&lt;level&gt;=*)\[[\w\W]*?\](?=level)\])|(?:--.*$)|\s+)+[a-zA-Z_])"><token type="KeywordReserved"/><push state="type_declaration"/></rule>
27+
<rule pattern="export\b(?=(?:(?:--\[(?&lt;level&gt;=*)\[[\w\W]*?\](?=level)\])|(?:--.*$)|\s+)+[a-zA-Z_])"><token type="KeywordReserved"/></rule>
28+
<rule pattern="(?:\.\.|//|[+\-*\/%^&lt;&gt;=])=?"><token type="Operator"/><push state="expression"/></rule>
29+
<rule pattern="~="><token type="Operator"/><push state="expression"/></rule>
30+
<rule pattern="(and|or|not)\b"><token type="OperatorWord"/><push state="expression"/></rule>
31+
<rule pattern="(elseif|for|if|in|repeat|return|until|while)\b"><token type="KeywordReserved"/><push state="expression"/></rule>
32+
<rule pattern="local\b"><token type="KeywordDeclaration"/><push state="expression"/></rule>
33+
<rule pattern="function\b"><token type="KeywordReserved"/><push state="expression" state="func_name"/></rule>
34+
<rule pattern="[\])};]+"><token type="Punctuation"/></rule>
35+
<rule><include state="expression_static"/></rule>
36+
<rule pattern="0[xX][\da-fA-F_]*"><token type="LiteralNumberHex"/></rule>
37+
<rule pattern="0[bB][\d_]*"><token type="LiteralNumberBin"/></rule>
38+
<rule pattern="\.?\d[\d_]*(?:\.[\d_]*)?(?:[eE][+-]?[\d_]+)?"><token type="LiteralNumberFloat"/></rule>
39+
<rule pattern="(true|false|nil)\b"><token type="KeywordConstant"/></rule>
40+
<rule pattern="\[(=*)\[[.\n]*?\]\1\]"><token type="LiteralString"/></rule>
41+
<rule pattern="(\.)([a-zA-Z_]\w*)(?=%s*[({&quot;\&#x27;])"><bygroups><token type="Punctuation"/><token type="NameFunction"/></bygroups></rule>
42+
<rule pattern="(\.)([a-zA-Z_]\w*)"><bygroups><token type="Punctuation"/><token type="NameVariable"/></bygroups></rule>
43+
<rule pattern="[a-zA-Z_]\w*(?:\.[a-zA-Z_]\w*)*(?=(?:(?:--\[(?&lt;level&gt;=*)\[[\w\W]*?\](?=level)\])|(?:--.*$)|\s+)*[({&quot;\&#x27;])"><token type="NameOther"/></rule>
44+
<rule pattern="[a-zA-Z_]\w*(?:\.[a-zA-Z_]\w*)*"><token type="Name"/></rule>
45+
<rule pattern="[\[.,]"><token type="Punctuation"/><push state="expression"/></rule>
46+
</state>
47+
<state name="expression_static">
48+
<rule pattern="(break|continue|do|else|elseif|end|for|if|in|repeat|return|then|until|while)\b"><token type="KeywordReserved"/></rule>
49+
</state>
50+
<state name="expression">
51+
<rule><include state="ws"/></rule>
52+
<rule pattern="if\b"><token type="KeywordReserved"/><push state="ternary" state="expression"/></rule>
53+
<rule pattern="local\b"><token type="KeywordDeclaration"/></rule>
54+
<rule pattern="\{"><token type="Punctuation"/><push state="#pop" state="closing_brace_base" state="expression"/></rule>
55+
<rule pattern="\("><token type="Punctuation"/><push state="#pop" state="closing_parenthesis_base" state="expression"/></rule>
56+
<rule pattern="::?"><token type="Punctuation"/><push state="#pop" state="type_end" state="type_start"/></rule>
57+
<rule pattern="&#x27;"><token type="LiteralStringSingle"/><push state="#pop" state="string_single"/></rule>
58+
<rule pattern="&quot;"><token type="LiteralStringDouble"/><push state="#pop" state="string_double"/></rule>
59+
<rule pattern="`"><token type="LiteralStringBacktick"/><push state="#pop" state="string_interpolated"/></rule>
60+
<rule pattern="\.\.\."><token type="Punctuation"/><pop depth="1"/></rule>
61+
<rule pattern="function\b"><token type="KeywordReserved"/><push state="func_name"/></rule>
62+
<rule><include state="expression_static"/></rule>
63+
<rule pattern="0[xX][\da-fA-F_]*"><token type="LiteralNumberHex"/><pop depth="1"/></rule>
64+
<rule pattern="0[bB][\d_]*"><token type="LiteralNumberBin"/><pop depth="1"/></rule>
65+
<rule pattern="\.?\d[\d_]*(?:\.[\d_]*)?(?:[eE][+-]?[\d_]+)?"><token type="LiteralNumberFloat"/><pop depth="1"/></rule>
66+
<rule pattern="(true|false|nil)\b"><token type="KeywordConstant"/><pop depth="1"/></rule>
67+
<rule pattern="\[(=*)\[[.\n]*?\]\1\]"><token type="LiteralString"/><pop depth="1"/></rule>
68+
<rule pattern="(\.)([a-zA-Z_]\w*)(?=%s*[({&quot;\&#x27;])"><bygroups><token type="Punctuation"/><token type="NameFunction"/></bygroups><pop depth="1"/></rule>
69+
<rule pattern="(\.)([a-zA-Z_]\w*)"><bygroups><token type="Punctuation"/><token type="NameVariable"/></bygroups><pop depth="1"/></rule>
70+
<rule pattern="[a-zA-Z_]\w*(?:\.[a-zA-Z_]\w*)*(?=(?:(?:--\[(?&lt;level&gt;=*)\[[\w\W]*?\](?=level)\])|(?:--.*$)|\s+)*[({&quot;\&#x27;])"><token type="NameOther"/><pop depth="1"/></rule>
71+
<rule pattern="[a-zA-Z_]\w*(?:\.[a-zA-Z_]\w*)*"><token type="Name"/><pop depth="1"/></rule>
72+
<rule><pop depth="1"/></rule>
73+
</state>
74+
<state name="ternary">
75+
<rule><include state="ws"/></rule>
76+
<rule pattern="else\b"><token type="KeywordReserved"/><pop depth="1"/></rule>
77+
<rule pattern="(then|elseif)\b"><token type="OperatorReserved"/><push state="expression"/></rule>
78+
<rule><pop depth="1"/></rule>
79+
</state>
80+
<state name="closing_brace_pop">
81+
<rule pattern="\}"><token type="Punctuation"/><pop depth="1"/></rule>
82+
</state>
83+
<state name="closing_parenthesis_pop">
84+
<rule pattern="\)"><token type="Punctuation"/><pop depth="1"/></rule>
85+
</state>
86+
<state name="closing_gt_pop">
87+
<rule pattern="&gt;"><token type="Punctuation"/><pop depth="1"/></rule>
88+
</state>
89+
<state name="closing_parenthesis_base">
90+
<rule><include state="closing_parenthesis_pop"/></rule>
91+
<rule><include state="base"/></rule>
92+
</state>
93+
<state name="closing_parenthesis_type">
94+
<rule><include state="closing_parenthesis_pop"/></rule>
95+
<rule><include state="type"/></rule>
96+
</state>
97+
<state name="closing_brace_base">
98+
<rule><include state="closing_brace_pop"/></rule>
99+
<rule><include state="base"/></rule>
100+
</state>
101+
<state name="closing_brace_type">
102+
<rule><include state="closing_brace_pop"/></rule>
103+
<rule><include state="type"/></rule>
104+
</state>
105+
<state name="closing_gt_type">
106+
<rule><include state="closing_gt_pop"/></rule>
107+
<rule><include state="type"/></rule>
108+
</state>
109+
<state name="string_escape">
110+
<rule pattern="\\z\s*"><token type="LiteralStringEscape"/></rule>
111+
<rule pattern="\\(?:[abfnrtvz\\&quot;\&#x27;`\{\n])|[\r\n]{1,2}|x[\da-fA-F]{2}|\d{1,3}|u\{\}[\da-fA-F]*\}"><token type="LiteralStringEscape"/></rule>
112+
</state>
113+
<state name="string_single">
114+
<rule><include state="string_escape"/></rule>
115+
<rule pattern="&#x27;"><token type="LiteralStringSingle"/><pop depth="1"/></rule>
116+
<rule pattern="[^\\&#x27;]+"><token type="LiteralStringSingle"/></rule>
117+
</state>
118+
<state name="string_double">
119+
<rule><include state="string_escape"/></rule>
120+
<rule pattern="&quot;"><token type="LiteralStringDouble"/><pop depth="1"/></rule>
121+
<rule pattern="[^\\&quot;]+"><token type="LiteralStringDouble"/></rule>
122+
</state>
123+
<state name="string_interpolated">
124+
<rule><include state="string_escape"/></rule>
125+
<rule pattern="\{"><token type="Punctuation"/><push state="closing_brace_base" state="expression"/></rule>
126+
<rule pattern="`"><token type="LiteralStringBacktick"/><pop depth="1"/></rule>
127+
<rule pattern="[^\\`\{]+"><token type="LiteralStringBacktick"/></rule>
128+
</state>
129+
<state name="func_name">
130+
<rule><include state="ws"/></rule>
131+
<rule pattern="[.:]"><token type="Punctuation"/></rule>
132+
<rule pattern="[a-zA-Z_]\w*(?=(?:(?:--\[(?&lt;level&gt;=*)\[[\w\W]*?\](?=level)\])|(?:--.*$)|\s+)*[.:])"><token type="NameClass"/></rule>
133+
<rule pattern="[a-zA-Z_]\w*"><token type="NameFunction"/></rule>
134+
<rule pattern="&lt;"><token type="Punctuation"/><push state="closing_gt_type"/></rule>
135+
<rule pattern="\("><token type="Punctuation"/><pop depth="1"/></rule>
136+
</state>
137+
<state name="type">
138+
<rule><include state="ws"/></rule>
139+
<rule pattern="\("><token type="Punctuation"/><push state="closing_parenthesis_type"/></rule>
140+
<rule pattern="\{"><token type="Punctuation"/><push state="closing_brace_type"/></rule>
141+
<rule pattern="&lt;"><token type="Punctuation"/><push state="closing_gt_type"/></rule>
142+
<rule pattern="&#x27;"><token type="LiteralStringSingle"/><push state="string_single"/></rule>
143+
<rule pattern="&quot;"><token type="LiteralStringDouble"/><push state="string_double"/></rule>
144+
<rule pattern="[|&amp;\.,\[\]:=]+"><token type="Punctuation"/></rule>
145+
<rule pattern="-&gt;"><token type="Punctuation"/></rule>
146+
<rule pattern="typeof\("><token type="NameBuiltin"/><push state="closing_parenthesis_base" state="expression"/></rule>
147+
<rule pattern="[a-zA-Z_]\w*"><token type="NameClass"/></rule>
148+
</state>
149+
<state name="type_start">
150+
<rule><include state="ws"/></rule>
151+
<rule pattern="\("><token type="Punctuation"/><push state="#pop" state="closing_parenthesis_type"/></rule>
152+
<rule pattern="\{"><token type="Punctuation"/><push state="#pop" state="closing_brace_type"/></rule>
153+
<rule pattern="&lt;"><token type="Punctuation"/><push state="#pop" state="closing_gt_type"/></rule>
154+
<rule pattern="&#x27;"><token type="LiteralStringSingle"/><push state="#pop" state="string_single"/></rule>
155+
<rule pattern="&quot;"><token type="LiteralStringDouble"/><push state="#pop" state="string_double"/></rule>
156+
<rule pattern="typeof\("><token type="NameBuiltin"/><push state="#pop" state="closing_parenthesis_base" state="expression"/></rule>
157+
<rule pattern="[a-zA-Z_]\w*"><token type="NameClass"/><pop depth="1"/></rule>
158+
</state>
159+
<state name="type_end">
160+
<rule><include state="ws"/></rule>
161+
<rule pattern="[|&amp;\.]"><token type="Punctuation"/><push state="type_start"/></rule>
162+
<rule pattern="-&gt;"><token type="Punctuation"/><push state="type_start"/></rule>
163+
<rule pattern="&lt;"><token type="Punctuation"/><push state="closing_gt_type"/></rule>
164+
<rule><pop depth="1"/></rule>
165+
</state>
166+
<state name="type_declaration">
167+
<rule><include state="ws"/></rule>
168+
<rule pattern="[a-zA-Z_]\w*"><token type="NameClass"/></rule>
169+
<rule pattern="&lt;"><token type="Punctuation"/><push state="closing_gt_type"/></rule>
170+
<rule pattern="="><token type="Punctuation"/><push state="#pop" state="type_end" state="type_start"/></rule>
171+
</state>
172+
</rules>
173+
</lexer>

‎lexers/testdata/luau.actual‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
-- we can define our own structural data types
2+
type Point = { x: number, y: number }
3+
-- and we can annotate variables (and function arguments) with them)
4+
local p: Point = { x = 1, y = 2 }
5+
6+
print(p.x, p.y)
7+
8+
-- and we can see that accessing an invalid field from a table raises a type error!
9+
print(p.z)

‎lexers/testdata/luau.expected‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
[
2+
{"type":"CommentSingle","value":"-- we can define our own structural data types"},
3+
{"type":"TextWhitespace","value":"\n"},
4+
{"type":"KeywordReserved","value":"type"},
5+
{"type":"TextWhitespace","value":" "},
6+
{"type":"NameClass","value":"Point"},
7+
{"type":"TextWhitespace","value":" "},
8+
{"type":"Punctuation","value":"="},
9+
{"type":"TextWhitespace","value":" "},
10+
{"type":"Punctuation","value":"{"},
11+
{"type":"TextWhitespace","value":" "},
12+
{"type":"NameClass","value":"x"},
13+
{"type":"Punctuation","value":":"},
14+
{"type":"TextWhitespace","value":" "},
15+
{"type":"NameClass","value":"number"},
16+
{"type":"Punctuation","value":","},
17+
{"type":"TextWhitespace","value":" "},
18+
{"type":"NameClass","value":"y"},
19+
{"type":"Punctuation","value":":"},
20+
{"type":"TextWhitespace","value":" "},
21+
{"type":"NameClass","value":"number"},
22+
{"type":"TextWhitespace","value":" "},
23+
{"type":"Punctuation","value":"}"},
24+
{"type":"TextWhitespace","value":"\n"},
25+
{"type":"CommentSingle","value":"-- and we can annotate variables (and function arguments) with them)"},
26+
{"type":"TextWhitespace","value":"\n"},
27+
{"type":"KeywordDeclaration","value":"local"},
28+
{"type":"TextWhitespace","value":" "},
29+
{"type":"Name","value":"p"},
30+
{"type":"Punctuation","value":":"},
31+
{"type":"TextWhitespace","value":" "},
32+
{"type":"NameClass","value":"Point"},
33+
{"type":"TextWhitespace","value":" "},
34+
{"type":"Operator","value":"="},
35+
{"type":"TextWhitespace","value":" "},
36+
{"type":"Punctuation","value":"{"},
37+
{"type":"TextWhitespace","value":" "},
38+
{"type":"Name","value":"x"},
39+
{"type":"TextWhitespace","value":" "},
40+
{"type":"Operator","value":"="},
41+
{"type":"TextWhitespace","value":" "},
42+
{"type":"LiteralNumberFloat","value":"1"},
43+
{"type":"Punctuation","value":","},
44+
{"type":"TextWhitespace","value":" "},
45+
{"type":"Name","value":"y"},
46+
{"type":"TextWhitespace","value":" "},
47+
{"type":"Operator","value":"="},
48+
{"type":"TextWhitespace","value":" "},
49+
{"type":"LiteralNumberFloat","value":"2"},
50+
{"type":"TextWhitespace","value":" "},
51+
{"type":"Punctuation","value":"}"},
52+
{"type":"TextWhitespace","value":"\n\n"},
53+
{"type":"NameOther","value":"print"},
54+
{"type":"Punctuation","value":"("},
55+
{"type":"Name","value":"p.x"},
56+
{"type":"Punctuation","value":","},
57+
{"type":"TextWhitespace","value":" "},
58+
{"type":"Name","value":"p.y"},
59+
{"type":"Punctuation","value":")"},
60+
{"type":"TextWhitespace","value":"\n\n"},
61+
{"type":"CommentSingle","value":"-- and we can see that accessing an invalid field from a table raises a type error!"},
62+
{"type":"TextWhitespace","value":"\n"},
63+
{"type":"NameOther","value":"print"},
64+
{"type":"Punctuation","value":"("},
65+
{"type":"Name","value":"p.z"},
66+
{"type":"Punctuation","value":")"},
67+
{"type":"TextWhitespace","value":"\n"}
68+
]

0 commit comments

Comments
 (0)