Skip to content

Drop python 3.8, add python 3.14#896

Merged
ddelange merged 4 commits intodevelopfrom
drop-3.8
Oct 20, 2025
Merged

Drop python 3.8, add python 3.14#896
ddelange merged 4 commits intodevelopfrom
drop-3.8

Conversation

@ddelange
Copy link
Collaborator

@ddelange ddelange commented Oct 20, 2025 •
edited
Loading

Please pick a concise, informative and complete title for your PR.

The title is important because it will appear in our change log.

Motivation

Please explain the motivation behind this PR.

If you're fixing a bug, link to the issue using a supported keyword like "Fixes #{issue_number}".

If you're adding a new feature, then consider opening a ticket and discussing it with the maintainers before you actually do the hard work.

Preparation for #895

Refactor iter_bucket using multithreading instead of multiprocessing: #896 (comment)

Tests

If you're fixing a bug, consider test-driven development:

  1. Create a unit test that demonstrates the bug. The test should fail.
  2. Implement your bug fix.
  3. The test you created should now pass.

If you're implementing a new feature, include unit tests for it.

Make sure all existing unit tests pass.
You can run them locally using:

pytest tests

If there are any failures, please fix them before creating the PR (or mark it as WIP, see below).

Work in progress

If you're still working on your PR, mark the PR as draft PR.

We'll skip reviewing it for the time being.

Once it's ready, mark the PR as "ready for review", and ping one of the maintainers (e.g. mpenkov).

Checklist

Before you mark the PR as "ready for review", please make sure you have:

  • Picked a concise, informative and complete title
  • Clearly explained the motivation behind the PR
  • Linked to any existing issues that your PR will be solving
  • Included tests for any new functionality
  • Run python update_helptext.py in case there are API changes
  • Checked that all unit tests pass

Workflow

Please avoid rebasing and force-pushing to the branch of the PR once a review is in progress.

Rebasing can make your commits look a bit cleaner, but it also makes life more difficult from the reviewer, because they are no longer able to distinguish between code that has already been reviewed, and unreviewed code.

@ddelange ddelange force-pushed the drop-3.8 branch 10 times, most recently from d474409 to f123451 Compare October 20, 2025 11:32
ddelange
ddelange commented Oct 20, 2025
View reviewed changes
tests/test_s3.py
Comment on lines -973 to -977
@pytest.mark.skipif(condition=sys.platform == 'win32', reason="does not run on windows")
@pytest.mark.xfail(
condition=sys.platform == 'darwin',
reason="MacOS uses spawn rather than fork for multiprocessing",
)
Copy link
Collaborator Author

@ddelange ddelange Oct 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3.14 changes default start methods (docs)

since iter_bucket is network bound and not CPU bound, this PR refactors iter_bucket to always use a single shared concurrent.futures.ThreadPoolExecutor with a single shared thread-safe boto S3.Client.

This makes the code simpler, more predictable, easier to maintain, and cross-platform compatible.

The default of 16 * 4 = global maximum 64 threads is tested to be stable on an EC2 c-family machine downloading from an S3 bucket in the same region.

@ddelange ddelange merged commit 1d08a57 into develop Oct 20, 2025
6 checks passed
@ddelange ddelange deleted the drop-3.8 branch October 20, 2025 11:45
ddelange added a commit that referenced this pull request Oct 20, 2025
@ddelange
Merge branch 'develop' into master
cefa547 
* develop:
  Update CHANGELOG.md
  Use compression.zstd (PEP-784) (#895)
  Drop python 3.8, add python 3.14 (#896)
  [s3] Add range_chunk_size param to read using multiple GET requests (#887)
  Run tests in parallel (#893)
  Optimize forward seeks within buffered data to avoid redundant GET (#892)
  Add macos to CI (#891)
  Simplify CI, use uv (#890)
  [s3] Improve handling of InvalidRange and seek on empty file (#889)
  Protect against hanging tests (#888)
  Bump the github-actions group with 2 updates (#886)
  build: fix invalid `fallback_version` when builing with `uv` (#884)
  Remove travis leftover (#881)
  Disambiguate URI examples in README.rst (#879)
@github-actions
Copy link

github-actions bot commented Oct 20, 2025

Released v7.4.0

This was referenced Nov 8, 2025
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ddelange