Skip to content

Use UTF8 when reading json/msgpack bytes #464

Merged
lihaoyi merged 2 commits intomainfrom
utf8-read
Mar 12, 2023
Merged

Use UTF8 when reading json/msgpack bytes #464
lihaoyi merged 2 commits intomainfrom
utf8-read

Conversation

@lihaoyi
Copy link
Copy Markdown
Member

@lihaoyi lihaoyi commented Mar 12, 2023

Fixes #413

We already hard-code support for UTF-8 when rendering JSON and MessagePack

final def escapeByte(unicodeCharBuilder: upickle.core.CharBuilder,
val strBytes = s.toString.getBytes(java.nio.charset.StandardCharsets.UTF_8)
, so the only thing missing is to ensure we use UTF-8 when reading

Added some simple tests under upickle.jvm[_].testNonUtf8 running with -Dfile.encoding=US-ASCII exercise these code paths

@lihaoyi lihaoyi changed the title Use UTF8 when reading json bytes Use UTF8 when reading json/msgpack bytes Mar 12, 2023
@lihaoyi lihaoyi marked this pull request as ready for review March 12, 2023 07:29
@lihaoyi lihaoyi merged commit a6fea69 into main Mar 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

read/write uses defaultCharset to create String

1 participant