Since changing from the text-encoding pollyfill to text-encoding-utf-8 (in #311), the performance of text decoding has dropped significantly.
In my testing, it was taking over 3 seconds to read a 175kb file and decode it into a JS string. It does depends on the size of the chunk you're giving to the text decoder.
Part of the problem is that the decoding routine in text-encoding-utf-8 examines each byte in the input buffer, it shifts the byte from the start (see the Stream class here) .
The original text-encoding does this better (see the Stream class here) by reversing the input buffer and operating on the end of the buffer rather than the start.
I've applied the reversing changes to text-encoding-utf-8 (in samuelbrian@99b895c) and it's seems on-par with the original version. But I'm not sure the best way to manage patching the node_modules dependencies, short of forking and maintaining a copy of text-encoding-utf-8.
Even better is using text encoding and decoding from QuickJS which I have done here samuelbrian@f308a6d, and that makes a file read that was taking >3 seconds into a <1 millisecond operation.
I've exposed these as tjs functions rather than implement them as TextEncoder/TextDecoder and remove the pollyfill, mainly because I didn't want to have to deal with the streaming option. But it might be viable to do.
Since changing from the
text-encodingpollyfill totext-encoding-utf-8(in #311), the performance of text decoding has dropped significantly.In my testing, it was taking over 3 seconds to read a 175kb file and decode it into a JS string. It does depends on the size of the chunk you're giving to the text decoder.
Part of the problem is that the decoding routine in
text-encoding-utf-8examines each byte in the input buffer, it shifts the byte from the start (see the Stream class here) .The original
text-encodingdoes this better (see the Stream class here) by reversing the input buffer and operating on the end of the buffer rather than the start.I've applied the reversing changes to
text-encoding-utf-8(in samuelbrian@99b895c) and it's seems on-par with the original version. But I'm not sure the best way to manage patching the node_modules dependencies, short of forking and maintaining a copy oftext-encoding-utf-8.Even better is using text encoding and decoding from QuickJS which I have done here samuelbrian@f308a6d, and that makes a file read that was taking >3 seconds into a <1 millisecond operation.
I've exposed these as tjs functions rather than implement them as TextEncoder/TextDecoder and remove the pollyfill, mainly because I didn't want to have to deal with the streaming option. But it might be viable to do.