Implement BOM stripping and ignoreBOM option in TextDecoder #4613
Description
Boa:
>> new TextDecoder().decode(Uint8Array.of(0xef, 0xbb, 0xbf)).length
1
>> new TextDecoder('utf-16le').decode(Uint8Array.of(0xff, 0xfe)).length
1
>> new TextDecoder('utf-16be').decode(Uint8Array.of(0xfe, 0xff)).length
1
>> new TextDecoder('utf-8', { ignoreBOM: true }).decode(Uint8Array.of(0xef, 0xbb, 0xbf)).length
1
>> new TextDecoder('utf-16le', { ignoreBOM: true }).decode(Uint8Array.of(0xff, 0xfe)).length
1
>> new TextDecoder('utf-16be', { ignoreBOM: true }).decode(Uint8Array.of(0xfe, 0xff)).length
1
>> new TextDecoder('utf-8', { ignoreBOM: false }).decode(Uint8Array.of(0xef, 0xbb, 0xbf)).length
1
>> new TextDecoder('utf-16le', { ignoreBOM: false }).decode(Uint8Array.of(0xff, 0xfe)).length
1
>> new TextDecoder('utf-16be', { ignoreBOM: false }).decode(Uint8Array.of(0xfe, 0xff)).length
1Correct behavior:
> new TextDecoder().decode(Uint8Array.of(0xef, 0xbb, 0xbf)).length
0
> new TextDecoder('utf-16le').decode(Uint8Array.of(0xff, 0xfe)).length
0
> new TextDecoder('utf-16be').decode(Uint8Array.of(0xfe, 0xff)).length
0
> new TextDecoder('utf-8', { ignoreBOM: true }).decode(Uint8Array.of(0xef, 0xbb, 0xbf)).length
1
> new TextDecoder('utf-16le', { ignoreBOM: true }).decode(Uint8Array.of(0xff, 0xfe)).length
1
> new TextDecoder('utf-16be', { ignoreBOM: true }).decode(Uint8Array.of(0xfe, 0xff)).length
1
> new TextDecoder('utf-8', { ignoreBOM: false }).decode(Uint8Array.of(0xef, 0xbb, 0xbf)).length
0
> new TextDecoder('utf-16le', { ignoreBOM: false }).decode(Uint8Array.of(0xff, 0xfe)).length
0
> new TextDecoder('utf-16be', { ignoreBOM: false }).decode(Uint8Array.of(0xfe, 0xff)).length
0