Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make hashlib related modules thread-safe without the GIL #111916

Closed
Tracked by #108219
, #116738
...
colesbury opened this issue Nov 9, 2023 · 2 comments · Fixed by #111981
Closed
Tracked by #108219
, #116738
...

Make hashlib related modules thread-safe without the GIL #111916

colesbury opened this issue Nov 9, 2023 · 2 comments · Fixed by #111981
Assignees
@tomasr8
Labels
3.13 new features, bugs and security fixes extension-modules C modules in the Modules dir topic-free-threading type-feature A feature request or enhancement

Comments

@colesbury
Copy link
Contributor

colesbury commented Nov 9, 2023 •
edited by bedevere-app bot

Feature or enhancement

The hashlib based modules already have some locking to make some operations thread-safe (with the GIL), but the logic isn't sufficient if running with the GIL disabled.

Relevant files:

  • Modules/_blake2/blake2b_impl.c
  • Modules/_blake2/blake2s_impl.c
  • Modules/_hashopenssl.c
  • Modules/hashlib.h
  • Modules/md5module.c
  • Modules/sha1module.c
  • Modules/sha2module.c
  • Modules/sha3module.c

Basic idea:

  1. Replace PyThread_type_lock lock with PyMutex. This should be both simpler and faster in general and avoid the need for dynamically assigning a lock, which can pose thread-safety issues without the GIL
  2. Add a field bool use_mutex to indicate if the code should lock the mutex. This should always be set to true in Py_NOGIL. In the default build, we should dynamically set it to true in places where we previously allocated self->lock
  3. Update ENTER_HASHLIB and EXIT_HASHLIB macros.

Linked PRs
@colesbury colesbury added type-feature A feature request or enhancement 3.13 new features, bugs and security fixes topic-free-threading labels Nov 9, 2023
@colesbury colesbury mentioned this issue Nov 9, 2023
Open
@AlexWaygood AlexWaygood added the extension-modules C modules in the Modules dir label Nov 10, 2023
@tomasr8
Copy link
Contributor

tomasr8 commented Nov 10, 2023

Hey! I'd like to give this a try. I have a limited experience with cpython internals but I had a quick look at the code and it seems feasible to me 😅

@colesbury
Copy link
Contributor Author

Great, thanks! Let me know if you have any questions or there is anything I can help with.

@bedevere-app bedevere-app bot mentioned this issue Nov 11, 2023
Merged
gpshead pushed a commit that referenced this issue Nov 15, 2023
@tomasr8
gh-111916: Make hashlib related modules thread-safe without the GIL (#…
a646560 
…111981)

Always use an individual lock on hash objects when in free-threaded builds.

Fixes #111916
aisk pushed a commit to aisk/cpython that referenced this issue Feb 11, 2024
@tomasr8 @aisk
pythongh-111916: Make hashlib related modules thread-safe without the…
fcc7a7e 
… GIL (python#111981)

Always use an individual lock on hash objects when in free-threaded builds.

Fixes python#111916
@swtaarrs swtaarrs mentioned this issue Mar 13, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tomasr8 tomasr8

Labels
3.13 new features, bugs and security fixes extension-modules C modules in the Modules dir topic-free-threading type-feature A feature request or enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

gh-111916: Make hashlib related modules thread-safe without the GIL tomasr8/cpython
3 participants
@colesbury @tomasr8 @AlexWaygood