Here is a test specifically for #1920 for you to cherry-pick @mohtakver, in case you find it's useful: 3539ef3

The current blob data in typst-assets uses a patched ICU data generator. See #1009 and https://github.com/typst/typst-assets/blob/0bbb3f7ebe775b980a3ca4ae9c49d9ecdb7e021a/src/lib.rs#L14-L45 for background. I'm honestly not sure whether there was an attempt to upstream the patch in peng1999/icu4x@b9beb6c.

I fear that, as-is, this may regress Typst's CJK handling. Two CJK test references have been changed in this PR, which is probably related to that.

It would also be worth investigating how this affects Typst's binary size. I'm not sure how exactly the blob data provider avoids bloat (if at all). The data generation commands inspect the built binary to determine exactly what it references. It's possible that the blob data provider simply relies on dead code elimination, but I wouldn't bet on that working perfectly.

cc @peng1999 if you have the time, your input would be appreciated on the CJK topic. :)

I fear that, as-is, this may regress Typst's CJK handling. Two CJK test references have been changed in this PR, which is probably related to that.

Yes, that is a regression. The two tests shouldn’t be modified.

I'm honestly not sure whether there was an attempt to upstream the patch in peng1999/icu4x@b9beb6c.

See #1355 (comment) (you should expand the resolved review to show the comment). I think icu4x 2.0 supports locale in segmenter, so patched datagen is no longer needed.

I think icu4x 2.0 supports locale in segmenter, so patched datagen is no longer needed.

I created a minimal example of using icu_segmenter 2.1.2 in #6774 (comment).
Even if LineBreakOptions's content_locale is set to zh, the segementer without patch is still problematic. Specifically, ：“ should be allowed to break into ：\n“, but not.

So, what should we do now?
Request changes in icu4x, or update the patch to icu4x 2.1?

related issue: unicode-org/icu4x#5595

So, what should we do now?
Request changes in icu4x, or update the patch to icu4x 2.1?

I think ideally both. The first for the long-term and the second to unblock us.

I'll close this PR since OP does not seem to be involved anymore.

@YDX-2147483647 If you'd be interested in picking this up, then feel free to! Otherwise, I will try to take a look sometime soon-ish.

If you'd be interested in picking this up

The icu4x repo and its relationship with the Unicode line breaking algorithm are more complicated than I thought.
I haven't taken any substantive actions at present, and it's likely that I'm incapable of doing it in the near future.

Sure, no problem. Thanks for the quick response!