Speed up sequence subscripting using Py_TPFLAGS_SEQUENCE#7432
Merged
scoder merged 37 commits intocython:masterfrom Jan 22, 2026
Merged
Speed up sequence subscripting using Py_TPFLAGS_SEQUENCE#7432scoder merged 37 commits intocython:masterfrom
scoder merged 37 commits intocython:masterfrom
Conversation
…st benchmark instead of the slowest. Scale back the timings of sub-benchmarks to the outer scale count to report comparable timings.
Contributor
|
I think this is reasonable although it's arguably a slight abuse of the flag (which is really only documented as being for pattern matching). I think a type type that gives a different answer if you go through |
…ed, and also time the subscripting from Python.
da-woods
reviewed
Jan 2, 2026
Co-authored-by: da-woods <dw-git@d-woods.co.uk>
…ed, and also time the subscripting from Python.
…e should be covered separately by the dedicated list subscript functions.
… run. This gives quicker insights long before the complete results are printed at the end of the run.
Contributor
Author
|
NumPy will set the sequence flag in the future: numpy/numpy#30519 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Python semantics dictate that we first try the mapping protocol and then the sequence protocol for subscripting. When the index is a C integer, we can optimise perfectly for list/tuple, but all other sequences suffer from having to build a Python
intobject for the index to pass it through the mapping lookup if they implement that (e.g. to support extended slicing, like NumPy arrays).Python 3.10 added type markers (for pattern matching) for explicitly declaring a type as sequence or mapping, called
Py_TPFLAGS_SEQUENCEandPy_TPFLAGS_MAPPING, which can now be checked for quite quickly. If a type is marked as sequence but still implements mapping lookups for slicing, and it supports sequence subscripting, we can avoid the Pythonintcreation of the mapping protocol and go straight through the sequence index lookup.With this change, indexing into Python's
array.arrayandmemoryviewtypes is ~60% faster in a micro-benchmark.Using a C integer as dict key got slightly slower but is resolved by adding a separate up-front special case.
Benchmark is based on #7431
See
https://docs.python.org/3/c-api/typeobj.html#c.Py_TPFLAGS_SEQUENCE
#1807
pandas-dev/pandas#55915
pandas-dev/pandas#55179 (comment)