Incorrect parse with trim: true and utf16le encoding #349
Description
Describe the bug
When using the trim: true option with utf16le encoding, the content is modified.
csv@5.5.3 contain csv-parse@4.16.3
Also reproduced for csv@6.1.3, csv-parse@5.2.0.
To Reproduce
In csv-parse@5.2.0:
const assert = require('assert')
const {parse: csvParse} = require('csv-parse/sync')
const data = 'ф, ы'
const encoding = 'utf16le'
const delimiter = ','
const record_delimiter = '\n'
const records = csvParse(Buffer.from(data, encoding), {
//delimiter: Buffer.from(delimiter, encoding),
//record_delimiter: Buffer.from(record_delimiter, encoding)
encoding,
trim: true
})
const correctRecords = data
.split(record_delimiter)
.map(
record => record
.split(delimiter)
.map(str => str.trim())
)
assert.deepStrictEqual(records, correctRecords)Additional context
I need to parse utf16le encoded csv file with custom delimiter and record_delimiter. Please release bugfix also for csv-parse@4.16.3.