python / cpython

…nsistently (gh-20038)

raw_data_manager (default for EmailPolicy, EmailMessage)
does correct wrapping of 'text' parts as long as the message contains
characters outside of 7bit US-ASCII set: base64 or qp
Content-Transfer-Encoding is applied if the lines would be too long
without it.  It did not, however, do this for ascii-only text,
which could result in lines that were longer than
policy.max_line_length or even the rfc 998  maximum.

This changeset fixes the heuristic so that if lines are longer than
policy.max_line_length, it will always apply a
content-transfer-encoding so that the lines are wrapped correctly.

Move PyInterpreterState.fs_codec into a new
PyInterpreterState.unicode structure.

Give a name to the fs_codec structure and use this structure in
unicodeobject.c.

The previous commits on bpo-29587 got exception chaining working
with gen.throw() in the `yield` case. This patch also gets the
`yield from` case working.

As a consequence, implicit exception chaining now also works in
the asyncio scenario of awaiting on a task when an exception is
already active.

Tests are included for both the asyncio case and the pure
generator-only case.

Remove the _PyUnicode_ClearStaticStrings() function from the C API.
Make the function fully private (declare it with "static").

…8007)

The crash that this fixes occurs when using traceback and other modules from multiple threads; 
del cache[filename] can raise a KeyError.

Don't hardcode defining_class parameter name to "cls":

* Define CConverter.set_template_dict(): do nothing by default
* CLanguage.render_function() now calls set_template_dict() on all
  converters.

This commit fixes SyntaxError locations when the caret is not displayed,
by doing the following:

- `col_number` always gets set to the location of the offending
  node/expr. When no caret is to be displayed, this gets achieved
  by setting the object holding the error line to None.

- Introduce a new function `_PyPegen_raise_error_known_location`,
  which can be called, when an arbitrary `lineno`/`col_offset`
  needs to be passed. This function then gets used in the grammar
  (through some new macros and inline functions) so that SyntaxError
  locations of the new parser match that of the old.

…yncio.wait`'s documentation (#20008)

…20069)

_Py_hashtable_get_entry_ptr() avoids comparing the entry hash:
compare directly keys.

Move _Py_hashtable_get_entry_ptr() just after
_Py_hashtable_get_entry_generic().

_Py_hashtable_t values become regular "void *" pointers.

* Add _Py_hashtable_entry_t.data member
* Remove _Py_hashtable_t.data_size member
* Remove _Py_hashtable_t.get_func member. It is no longer needed
  to specialize _Py_hashtable_get() for a specific value size, since
  all entries now have the same size (void*).
* Remove the following macros:

  * _Py_HASHTABLE_GET()
  * _Py_HASHTABLE_SET()
  * _Py_HASHTABLE_SET_NODATA()
  * _Py_HASHTABLE_POP()

* Rename _Py_hashtable_pop() to _Py_hashtable_steal()
* _Py_hashtable_foreach() callback now gets key and value rather than
  entry.
* Remove _Py_hashtable_value_destroy_func type. value_destroy_func
  callback now only has a single parameter: data (void*).

Rewrite _tracemalloc to store "trace_t*" rather than directly
"trace_t" in traces hash tables. Traces are now allocated on the heap
memory, outside the hash table.

Add tracemalloc_copy_traces() and tracemalloc_copy_domains() helper
functions.

Remove _Py_hashtable_copy() function since there is no API to copy a
key or a value.

Remove also _Py_hashtable_delete() function which was commented.

Add key_destroy_func and value_destroy_func parameters to
_Py_hashtable_new_full().

marshal.c and _tracemalloc.c use these destroy functions.

Rewrite _Py_hashtable_t type to always store the key as
a "const void *" pointer. Add an explicit "key" member to
_Py_hashtable_entry_t.

Remove _Py_hashtable_t.key_size member.

hash and compare functions drop their hash table parameter, and their
'key' parameter type becomes "const void *".

Rewrite how the _tracemalloc module stores traces of other domains.
Rather than storing the domain inside the key, it now uses a new hash
table with the domain as the key, and the data is a per-domain traces
hash table.

* Add tracemalloc_domain hash table.
* Remove _Py_tracemalloc_config.use_domain.
* Remove pointer_t and related functions.

Add a new _Py_HashPointerRaw() function which avoids replacing -1
with -2 to micro-optimize hash table using pointer keys: using
_Py_hashtable_hash_ptr() hash function.

…H-20024)

… now that we allow subtypes of PyCFunction.

Also add PyCMethod_CheckExact() and PyCMethod_Check() for checks against the PyCMethod subtype.

…ining non-BMP characters on Windows. (GH-20053)

Optimize _Py_hashtable_get() and _Py_hashtable_get_entry() for
pointer keys:

* key_size == sizeof(void*)
* hash_func == _Py_hashtable_hash_ptr
* compare_func == _Py_hashtable_compare_direct

Changes:

* Add get_func and get_entry_func members to _Py_hashtable_t
* Convert _Py_hashtable_get() and _Py_hashtable_get_entry() functions
  to static nline functions.
* Add specialized get and get entry for pointer keys.

…e. (GH-20033)

These were added by mistake (see https://bugs.python.org/issue39481#msg366288).

)

In translate(), generate unique group names across calls.

The restores the undocumented ability to get a valid regexp
by joining multiple translate() results via `|`.

_Py_hashtable_new() now uses PyMem_Malloc/PyMem_Free allocator by
default, rather than PyMem_RawMalloc/PyMem_RawFree.

PyMem_Malloc is faster than PyMem_RawMalloc for memory blocks smaller
than or equal to 512 bytes.

* Move Modules/hashtable.h to Include/internal/pycore_hashtable.h
* Move Modules/hashtable.c to Python/hashtable.c
* Python is now linked to hashtable.c. _tracemalloc is no longer
  linked to hashtable.c. Previously, marshal.c got hashtable.c via
  _tracemalloc.c which is built as a builtin module.

* Replace id->object with _PyUnicode_FromId(&id)
* Use _Py_static_string_init(str) macro to initialize statically
  name_op in typeobject.c.

…_offset (GH-20026)

Co-authored-by: Brad Solomon <brsolomon@deloitte.com>
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>

* Py_UNICODE -> wchar_t
* Py_UNICODE -> unicode in Argument Clinic
* PyUnicode_AsUnicode -> PyUnicode_AsWideCharString
* Don't use "u#" format.

Co-authored-by: Victor Stinner <vstinner@python.org>

…GH-20018)

Avoid unnecessary overhead in _PyDict_GetItemIdWithError() by calling
_PyDict_GetItem_KnownHash() instead of the more generic PyDict_GetItemWithError(),
since we already know the hash of interned strings.