Skip to content

Speed up ichunked#793

Merged
bbayles merged 7 commits intomore-itertools:masterfrom
james-wasson:ichunked
Feb 15, 2024
Merged

Speed up ichunked#793
bbayles merged 7 commits intomore-itertools:masterfrom
james-wasson:ichunked

Conversation

@james-wasson
Copy link
Copy Markdown
Contributor

@james-wasson james-wasson commented Jan 9, 2024

Issue reference

#797

Changes

I was trying to create my own ichunked method and found this library and thought I might contribute. This new method implementation runs about 2x faster than the current implementation due to it not using any user created iterators (eg. classes with the next() method).

There should be no changes visible to the end user other than the speed difference unless the user is relying on the private class _IChunk.

This implementation uses a scoped variables within a wrapping method scope to modify the state of the generator on the fly.

I have tested and timed this method extensively using both calls to next() and just looping. I have also tried modifying the state part way through iteration and I have not run across any issues.

A simple timing test:

  • Before
    image
  • After
    image

@james-wasson james-wasson mentioned this pull request Jan 9, 2024
@james-wasson
Copy link
Copy Markdown
Contributor Author

All checks passed locally.

Had to modify a test to work around the non exposure of the _cache attribute. I believe all the same functionality is tested.

@james-wasson
Copy link
Copy Markdown
Contributor Author

For timing reference:

from more_itertools import ichunked, consume
%timeit consume(consume(c) for c in ichunked([0] * 10000, 107))

@bbayles
Copy link
Copy Markdown
Collaborator

bbayles commented Feb 15, 2024

I took a few shots at making a simpler version of this, but yours is undeniably faster. Thanks!

@bbayles bbayles merged commit 399df10 into more-itertools:master Feb 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants