Summary of Changes

Hello @luadebug, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the string manipulation capabilities of the system by integrating new lower and upper functions. These additions provide a standardized way to change the case of strings, with a focus on internationalization by correctly handling various character encodings, thereby improving the flexibility and global compatibility of string operations.

Highlights

• New String Case Conversion Functions: Introduced lower and upper functions for string objects, allowing conversion of strings to their lowercase and uppercase equivalents.
• Unicode Support: The new case conversion functions are implemented with robust support for multi-byte character sets, specifically handling UTF-8 input by converting to wide characters (UTF-16 on Windows, UTF-32 on other systems) for conversion, and then back to UTF-8.
• New Source File: A new C source file, core/src/xmake/string/case.c, has been added to encapsulate the logic for these string case conversions.
• Unit Tests: Comprehensive unit tests have been added in tests/modules/string/test.lua to verify the correctness of the lower and upper functions, including tests for both ASCII and non-ASCII (Cyrillic) characters.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

/gemini review

/gemini review

Please do not include third-party dependencies. xmake strictly controls the inclusion of third-party libraries and may gradually remove some existing third-party library dependencies in the future. tbox has built-in encoding handling and cross-platform support; we should improve tbox as much as possible to improve and implement these features.

Please do not include third-party dependencies. xmake strictly controls the inclusion of third-party libraries and may gradually remove some existing third-party library dependencies in the future. tbox has built-in encoding handling and cross-platform support; we should improve tbox as much as possible to improve and implement these features.

OK. What is a solution for this situation?

Supporting full character set conversion seems like a huge undertaking, potentially increasing the xmake binary by over 1-2MB.

https://github.com/JuliaStrings/utf8proc/blob/e5e799221b45bbb90f5fdc5c69b6b8dfbf017e78/utf8proc_data.c (2.25MB)

We should improve the charset module of tbox to provide tolower/toupper interfaces.

Then, for some system platforms, we can use existing system APIs to support it (e.g., Windows CharUpperBuffW/CharLowerBuffW, Apple NSString uppercaseString/lowercaseString, CFStringLowercase). If no existing system API support is available, we can fall back to our own simplified implementation, only providing conversions for some commonly used language character sets.

This might be quite complex and time-consuming. If you can't implement it in a short time, you can leave it for now. I'll try to support it when I have more time later.

/gemini review

wait for this patch. tboox/tbox#300

then we can use tb_charset_utf8_tolower/tb_charset_utf8_toupper, it will call tb_wcslwr/tb_wcsupr to do it.

It makes some optimizations. Since 90% of the time the strings are ASCII strings, executing the Unicode tolower/toupper completely every time would significantly impact performance, so we don't need to do that.

tb_long_t tb_charset_utf8_tolower(tb_char_t* s, tb_size_t n)
{
    // check
    tb_assert_and_check_return_val(s, -1);

    // try ascii tolower first
    tb_char_t* p = s;
    tb_char_t* e = s + n;
    while (p < e && *p)
    {
        if ((*p) & 0x80) break; 
        *p = tb_tolower(*p);
        p++;
    }
    if (p == e || !*p) return p - s;

    // convert the suffix to wchar_t
    tb_long_t   r = -1;
    tb_size_t   wn = e - p + 1;
    tb_wchar_t  wb[256];
    tb_wchar_t* w = (wn <= 256)? wb : (tb_wchar_t*)tb_malloc(wn * sizeof(tb_wchar_t));
    if (w)
    {
        // to lower
        if (tb_mbstowcs(w, p, wn) != -1)
        {
            tb_wcslwr(w);
            r = tb_wcstombs(p, w, wn);
            if (r != -1) r += (p - s);
        }

        // free it
        if (w != wb) tb_free(w);
    }
    return r;
}

Looks cool, would be nice to have it clean without windows.h or some appleframework?

I don't know if tb_wchar_t maps to wchar_t but beware on Windows wchar_t is UTF-16 and has surrogate pairs, which means a single wchar_t may be half a character.

I don't know if tb_wchar_t maps to wchar_t but beware on Windows wchar_t is UTF-16 and has surrogate pairs, which means a single wchar_t may be half a character.

Therefore, I would prioritize using the CharUpperW API to implement complete string conversion, as is the case on Apple's platform. A while-loop + towupper is the fallback solution. @luadebug

I don't know if tb_wchar_t maps to wchar_t but beware on Windows wchar_t is UTF-16 and has surrogate pairs, which means a single wchar_t may be half a character.

Therefore, I would prioritize using the CharUpperW API to implement complete string conversion, as is the case on Apple's platform. A while-loop + towupper is the fallback solution. @luadebug

Well sounds good as we have them included at any case.

We also need to update tbox.config.h for each platform to add detection results such as wcsupr and towlower. Usually only towlower and towupper exist.

wait for this patch #7237

rebase dev and try it again.

like this

/* string.upper
 *
 * @param str       the string
 *
 * @code
 *      local result = string.upper(str)
 * @endcode
 */
tb_int_t xm_string_toupper(lua_State *lua) {
    // check
    tb_assert_and_check_return_val(lua, 0);

    // get string
    size_t           size = 0;
    tb_char_t const* cstr = luaL_checklstring(lua, 1, &size);
    tb_check_return_val(cstr, 0);

    // empty?
    if (!size) {
        lua_pushstring(lua, "");
        return 1;
    }

    // copy string to buffer
    tb_char_t* buffer = tb_malloc_bytes(size + 1);
    if (buffer) {
        tb_memcpy(buffer, cstr, size);
        buffer[size] = '\0';

        // to upper
        tb_long_t real_size = tb_charset_utf8_toupper(buffer, size);
        
        // push result
        if (real_size >= 0) {
            lua_pushlstring(lua, buffer, real_size);
        } else {
            lua_pushlstring(lua, cstr, size);
        }
        tb_free(buffer);
    } else {
        lua_pushnil(lua);
    }
    return 1;
}