Description

Motivation

When outputting a C++ string to a human-readable (printable) format, we must ensure the string is UTF8 valid, and escape special characters such as the new line character \n or the quote mark ". This is required if we want to serialize string data into a JSON format.
There are third-party libraries that already implement such a validation + escaping mechanism. For example, nlohmann JSON provides string dump to execute the validation and escape. However, using external libraries have the following problems:

1. Run time overhead. We need to convert a string view into their representation. In our previous benchmark, using nlohmann JSON for escaping purpose can introduce significant overhead.
2. Library dependencies in ffi libraries: We need extra dependencies on our ffi libraries.

Therefore, we decide to implement our own implementation for validating and escaping UTF8 strings.

Implementation

The functionalities can be breakdown as follows:

• Validate UTF8 encoding:

  • Validate UTF8 byte sequence
  • Validate code point ranges

• Escape special characters

  • Escape printable characters by add ''
  • Escape control characters in the form of \\u00xx

  This PR adds a util function to implement the functionalities listed above.

Validation performed

• Passed clang-tidy checks
• Added unit tests to cover the functionalities discussed above. Details can be found in the unit test implementations.