Skip to content

Support for certain unicode literals depends on sys.maxunicode #350

@anishathalye

Description

@anishathalye

It seems that support for unicode literals over code point 0xffff, introduced in cf1c86c, depends on sys.maxunicode. So for example, it'll work with standard Python 3 builds, but won't work with standard Python 2 builds, where sys.maxunicode is 2**16-1 (built with UCS-2).

Here's a simple test program that works on Python 3, but not on a standard Python 2 build:

# coding=utf8
import yaml

s = '''
"😢"
'''

assert yaml.safe_load(s) == '😢'

It produces the following error on Python 2:

Traceback (most recent call last):
  File "pyyaml_unicode.py", line 8, in <module>
    assert yaml.safe_load(s) == '😢'
  File "/usr/local/lib/python2.7/site-packages/yaml/__init__.py", line 162, in safe_load
    return load(stream, SafeLoader)
  File "/usr/local/lib/python2.7/site-packages/yaml/__init__.py", line 112, in load
    loader = Loader(stream)
  File "/usr/local/lib/python2.7/site-packages/yaml/loader.py", line 34, in __init__
    Reader.__init__(self, stream)
  File "/usr/local/lib/python2.7/site-packages/yaml/reader.py", line 81, in __init__
    self.determine_encoding()
  File "/usr/local/lib/python2.7/site-packages/yaml/reader.py", line 137, in determine_encoding
    self.update(1)
  File "/usr/local/lib/python2.7/site-packages/yaml/reader.py", line 174, in update
    self.check_printable(data)
  File "/usr/local/lib/python2.7/site-packages/yaml/reader.py", line 149, in check_printable
    'unicode', "special characters are not allowed")
yaml.reader.ReaderError: unacceptable character #xd83d: special characters are not allowed
  in "<string>", position 2

Is there any reasonable way to support this functionality on Python 2 as well?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions