Accessing elements of an array with dtype="U" may raise UnicodeDecodeError #15363
Description
Reproducing code example:
import sys
import numpy as np
arr = np.zeros(shape=1, dtype="U1")
for i in range(1, sys.maxunicode + 1):
arr[0] = chr(i)
arr[0]Error message:
In [35]: arr = np.zeros(shape=1, dtype="U1")
...: for i in range(1, sys.maxunicode + 1):
...: arr[0] = chr(i)
...: arr[0]
---------------------------------------------------------------------------
UnicodeDecodeError Traceback (most recent call last)
<ipython-input-35-5a43f3fdb19c> in <module>
2 for i in range(1, sys.maxunicode + 1):
3 arr[0] = chr(i)
----> 4 arr[0]
UnicodeDecodeError: 'utf-32-le' codec can't decode bytes in position 0-3:
code point in surrogate code point range(0xd800, 0xe000)For some later codepoints the message is instead code point not in range(0x110000) (false, as the problematic codepoints are all <= 0xdfff). For extra bonus fun some affected code points are < 0xd800, and not all surrogate characters are affected.
Numpy/Python version information:
Numpy 1.17.4
Python 3.7.3 (default, Apr 24 2019, 15:29:51) [MSC v.1915 64 bit (AMD64)]