perf: use simdutf8 to validate UTF-8 when reading files#2040
perf: use simdutf8 to validate UTF-8 when reading files#2040IWANABETHATGUY merged 1 commit intorolldown:mainfrom
simdutf8 to validate UTF-8 when reading files#2040Conversation
✅ Deploy Preview for rolldown-rs canceled.
|
|
I assume this would break our wasm and wasi builds without and feature target gates. |
|
But it shouldn't break WASM anyway. |
928b189 to
4dbf02f
Compare
|
We can always test the effect of such functionalities in oxc first, oxlint in this case. |
|
Do oxlint's benchmarks include file IO? |
I need to profile locally. |
|
In monitor-oxc, validating 84190 files (before and after): 
Performance is greatly improved, but maybe M3 is too powerful to make a dent on the overall performance improvement. |
|
@Boshen since you assign to yourself, this pull request needs your approval to merge. |
|
I wasn't sure if this change also needs making elsewhere in Rolldown, if other parts of the code also read from file system. @IWANABETHATGUY can you advise? |
Description
I'm not familiar with Rolldown's codebase, so I'm not sure whether this function is the main one used to read source text from files - so this may be in wrong place/not enough places.
std::fs::read_to_stringinternally callsstd::str::from_utf8to check that the file is valid UTF-8.str::from_utf8is not so efficient as it doesn't use SIMD.simduft8should be faster.I assume that this difference won't show up on benchmarks as they likely won't include IO. But I'd hope it'd have an effect in the real world. How would we go about testing that assumption? Are there any benchmarks which do exercise this function?