Summary

When tiktoken is unavailable, _estimate_tokens() falls back to len(text) // 3, which assumes ~3 characters per token. This is reasonable for Latin/ASCII text but severely underestimates token counts for CJK (Chinese, Japanese, Korean) text, where each character typically maps to 1–2 tokens.

The underestimation means CJK text that actually exceeds the 8192-token API limit gets an estimated count well below the threshold, so _chunk_text() never triggers chunking. The full text is then sent to the embedding API, which rejects it with a BadRequestError.

Fix

Replace:

return len(text) // 3

With:

return max(len(text) // 3, len(text.encode("utf-8")) // 4)

Each CJK character is 3 bytes in UTF-8, so len(text.encode("utf-8")) // 4 yields ~0.75 tokens per CJK character — much closer to the actual 1–2 range. The max() ensures whichever estimate is more conservative wins: the char-based estimate is still used for ASCII-heavy text, while the byte-based estimate kicks in for CJK-heavy text.

Fixes #616
Fixes #634