#7242
#7238
#7241

Xmake utf8 Module Interface

The utf8 module provides comprehensive support for UTF-8 string manipulation, extending standard Lua functionality with additional helpers and convenient character-based indexing for common operations.

Overview

Mixed Indexing Modes:

• Byte Indices: Used by len, codepoint, offset, and codes (consistent with Lua 5.3+ utf8 standard) for efficient low-level iteration.
• Character Indices: Used by sub, find, lastof, and byte for convenient high-level text processing (1-based, treating each UTF-8 sequence as one unit).

Interface Description

utf8.len(s [, i [, j [, lax]]])

Returns the number of UTF-8 characters in string s that start between byte positions i and j (both inclusive).

• Parameters:
  • s (string): The input string.
  • i (number, optional): Start byte position (default: 1).
  • j (number, optional): End byte position (default: -1).
  • lax (boolean, optional): If true, ignores invalid sequences.
• Returns: (number) The count of characters, or nil + error position if invalid.

utf8.sub(s, i [, j])

Returns the substring of s that starts at character index i and ends at character index j.

• Parameters:
  • s (string): The input string.
  • i (number): Start character index.
  • j (number, optional): End character index (default: -1).
• Returns: (string) The substring.

utf8.find(s, pattern [, init [, plain]])

Looks for the first match of pattern in the string s.

• Parameters:
  • s (string): The input string.
  • pattern (string): The pattern to search for.
  • init (number, optional): Start character index for search (default: 1).
  • plain (boolean, optional): If true, treats pattern as a plain string.
• Returns: (number, number) Start and end character indices of the match, or nil.

utf8.lastof(s, pattern [, plain])

Finds the last occurrence of pattern in the string s.

• Parameters:
  • s (string): The input string.
  • pattern (string): The pattern to search for.
  • plain (boolean, optional): If true, treats pattern as a plain string.
• Returns: (number) The start character index of the last match, or nil.

utf8.byte(s [, i [, j]])

Returns the internal numeric codes (codepoints) of the characters s[i], s[i+1], ..., s[j].

• Parameters:
  • s (string): The input string.
  • i (number, optional): Start character index (default: 1).
  • j (number, optional): End character index (default: i).
• Returns: (number...) The codepoints.

utf8.codepoint(s [, i [, j [, lax]]])

Returns the codepoints (as integers) from all characters in s that start between byte position i and j.

• Parameters:
  • s (string): The input string.
  • i (number, optional): Start byte position (default: 1).
  • j (number, optional): End byte position (default: i).
  • lax (boolean, optional): If true, ignores invalid sequences.
• Returns: (number...) The codepoints.

utf8.char(...)

Receives zero or more integers, converts each one to its corresponding UTF-8 character, and returns a string with the concatenation of all these characters.

• Returns: (string) The resulting string.

utf8.offset(s, n [, i])

Returns the byte index within s of the n-th character after byte position i.

• Parameters:
  • s (string): The input string.
  • n (number): The number of characters to advance (can be negative).
  • i (number, optional): The starting byte position (default: 1).
• Returns: (number) The byte index.

utf8.width(s) or utf8.width(codepoint)

Returns the display width of a string or a single codepoint (handling full-width characters).

• Returns: (number) The width (usually 1 or 2 per character).

utf8.reverse(s)

Reverses the UTF-8 string s.

• Returns: (string) The reversed string.

utf8.codes(s [, lax])

Returns an iterator (byte_pos, codepoint) for the string.

Usage Examples

local s = "你好xmake"

-- Length
print(utf8.len(s)) -- 7 (2 Chinese + 5 English)

-- Substring (Character Index)
print(utf8.sub(s, 1, 2)) -- "你好"
print(utf8.sub(s, 3))    -- "xmake"

-- Find (Character Index)
local start, finish = utf8.find(s, "xmake")
print(start, finish) -- 3, 7

-- Byte/Codepoint
-- utf8.byte uses character index
print(utf8.byte(s, 1)) -- 20320 (codepoint for '你')
-- utf8.codepoint uses byte index
print(utf8.codepoint(s, 1, 3)) -- 20320 (only if the char starts at byte 1 and ends before/at byte 3)

-- Width
print(utf8.width("你好")) -- 4 (2 * 2)
print(utf8.width("xmake")) -- 5

-- Reverse
print(utf8.reverse("你好")) -- "好你"