[C API] Remove token.h header file #92651
Labels
Comments
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
May 11, 2022
Move Include/token.h to Include/internal/pycore_token.h. Including this header file now requires that the Py_BUILD_CORE macro is defined. It no longer checks for the Py_LIMITED_API macro. Rename functions: * PyToken_OneChar() => _PyToken_OneChar() * PyToken_TwoChars() => _PyToken_TwoChars() * PyToken_ThreeChars() => _PyToken_ThreeChars() These functions and _PyParser_TokenNames symbols are no longer exported. There was never no public tokenizer C API. The <token.h> header file was only designed to be used by Python internals.
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
May 11, 2022
Remove the token.h header file. There was never any public tokenizer C API. The token.h header file was only designed to be used by Python internals. Move Include/token.h to Include/internal/pycore_token.h. Including this header file now requires that the Py_BUILD_CORE macro is defined. It no longer checks for the Py_LIMITED_API macro. Rename functions: * PyToken_OneChar() => _PyToken_OneChar() * PyToken_TwoChars() => _PyToken_TwoChars() * PyToken_ThreeChars() => _PyToken_ThreeChars() These functions and _PyParser_TokenNames symbols are no longer exported.
vstinner
added a commit
to vstinner/cpython
that referenced
this issue
May 11, 2022
Remove the token.h header file. There was never any public tokenizer C API. The token.h header file was only designed to be used by Python internals. Move Include/token.h to Include/internal/pycore_token.h. Including this header file now requires that the Py_BUILD_CORE macro is defined. It no longer checks for the Py_LIMITED_API macro. Rename functions: * PyToken_OneChar() => _PyToken_OneChar() * PyToken_TwoChars() => _PyToken_TwoChars() * PyToken_ThreeChars() => _PyToken_ThreeChars()
vstinner
added a commit
that referenced
this issue
May 11, 2022
Remove the token.h header file. There was never any public tokenizer C API. The token.h header file was only designed to be used by Python internals. Move Include/token.h to Include/internal/pycore_token.h. Including this header file now requires that the Py_BUILD_CORE macro is defined. It no longer checks for the Py_LIMITED_API macro. Rename functions: * PyToken_OneChar() => _PyToken_OneChar() * PyToken_TwoChars() => _PyToken_TwoChars() * PyToken_ThreeChars() => _PyToken_ThreeChars()
|
Fixed by da5727a |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Symbols and functions exported by the
<token.h>header file cannot be used outside Python internals. There is no public C API to tokenize a Python source file. I propose to simply remove this header file. It's not included byPython.h.In Python 3.10, I removed the C API related to AST and the parser, since it was not possible to use it outside Python internals. See: https://docs.python.org/dev/whatsnew/3.10.html#id4
The text was updated successfully, but these errors were encountered: