Skip to content

py_wheel: always generate zip64-capable wheels#2711

Merged
rickeylev merged 9 commits intobazel-contrib:mainfrom
devtbi:main
May 3, 2025
Merged

py_wheel: always generate zip64-capable wheels#2711
rickeylev merged 9 commits intobazel-contrib:mainfrom
devtbi:main

Conversation

@devtbi
Copy link
Copy Markdown
Contributor

@devtbi devtbi commented Mar 31, 2025 •
edited by rickeylev
Loading

Currently, there is no possibility to pass the force zip64 option to the wheel creation. This hinders creation of packages that contain >2Gb files (e.g. large projects with debug symbols).

To fix, always generate zip64 capable wheels. zip64 support is wide spread.

Fixes #2852

@devtbi devtbi requested review from aignas and rickeylev as code owners March 31, 2025 12:15
@devtbi devtbi marked this pull request as draft March 31, 2025 12:16
@rickeylev
Copy link
Copy Markdown
Collaborator

rickeylev commented Apr 1, 2025

zip64 is enabled by default, so shouldn't that obviate the need for forcing it to be enabled?

@devtbi
Copy link
Copy Markdown
Contributor Author

devtbi commented Apr 1, 2025

@rickeylev for my case it doesnt seem to suffice.. this is the output of the failure for the current main (7d431d8), which then works with forcing zip64:

~/<...>-mlir$ bazel build --config debug //packages/...
INFO: Analyzed 178 targets (331 packages loaded, 23354 targets configured).
ERROR: <...>/packages/<...>-iree/compiler/BUILD.bazel:668:9: Building wheel @@//packages/<...>-iree/compiler:compiler_python_wheel failed: (Exit 1): wheelmaker failed: error executing PyWheel command (from target //packages/<...>-iree/compiler:compiler_python_wheel) bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/rules_python~/tools/wheelmaker --name iree_compile_bazel --version 0.0.1 --python_tag py3 --abi none --platform any --out ... (remaining 12 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
Traceback (most recent call last):
  File "/<...>/.cache/bazel/_bazel_devtbi/ab6f2c0bfbd923053b3e2b5e324ae76c/sandbox/processwrapper-sandbox/3/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/rules_python~/tools/wheelmaker.runfiles/_main/../rules_python~/tools/wheelmaker.py", line 631, in <module>
    main()
  File "/<...>/.cache/bazel/_bazel_devtbi/ab6f2c0bfbd923053b3e2b5e324ae76c/sandbox/processwrapper-sandbox/3/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/rules_python~/tools/wheelmaker.runfiles/_main/../rules_python~/tools/wheelmaker.py", line 541, in main
    maker.add_file(package_filename, real_filename)
  File "/<...>/.cache/bazel/_bazel_devtbi/ab6f2c0bfbd923053b3e2b5e324ae76c/sandbox/processwrapper-sandbox/3/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/rules_python~/tools/wheelmaker.runfiles/_main/../rules_python~/tools/wheelmaker.py", line 304, in add_file
    self._whlfile.add_file(package_filename, real_filename)
  File "/<...>/.cache/bazel/_bazel_devtbi/ab6f2c0bfbd923053b3e2b5e324ae76c/sandbox/processwrapper-sandbox/3/execroot/_main/bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/rules_python~/tools/wheelmaker.runfiles/_main/../rules_python~/tools/wheelmaker.py", line 157, in add_file
    with self.open(zinfo, "w") as fdst:
  File "/<...>/.cache/bazel/_bazel_devtbi/ab6f2c0bfbd923053b3e2b5e324ae76c/execroot/_main/external/python_3_11_x86_64-unknown-linux-gnu/lib/python3.11/zipfile.py", line 1201, in close
    raise RuntimeError("File size too large, try using force_zip64")
    ```

@rickeylev
Copy link
Copy Markdown
Collaborator

rickeylev commented Apr 1, 2025

Oh hm. I think the usage of zip64 is inconsistent, depending on which zip APIs is used.

ZipFile() says it defaults to allow_zip64=True. Meanwhile, ZipFile.open (the API used in this code), defaults to force_zip64=False

rickeylev
rickeylev requested changes Apr 1, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator

@rickeylev rickeylev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Direction LGTM, but see comment

@rickeylev rickeylev changed the title feat: Add force_zip64 option feat: always generate zip64-capable wheels Apr 20, 2025
@rickeylev rickeylev marked this pull request as ready for review April 20, 2025 06:32
@rickeylev rickeylev changed the title feat: always generate zip64-capable wheels py_wheel: always generate zip64-capable wheels Apr 20, 2025
@rickeylev rickeylev enabled auto-merge April 20, 2025 06:33
@rickeylev rickeylev disabled auto-merge April 20, 2025 06:33
@rickeylev
Copy link
Copy Markdown
Collaborator

rickeylev commented Apr 20, 2025

I updated it to always set zip64.

There's a wheel test with a hard coded hash, though, so that needs to be updated.

@rickeylev rickeylev mentioned this pull request May 2, 2025
Closed
@rickeylev rickeylev enabled auto-merge May 3, 2025 03:19
@rickeylev rickeylev added this pull request to the merge queue May 3, 2025
Merged via the queue into bazel-contrib:main with commit ccbe5dc May 3, 2025
2 checks passed
@tbarrella tbarrella mentioned this pull request May 3, 2025
Closed
@devtbi
Copy link
Copy Markdown
Contributor Author

devtbi commented May 6, 2025

thanks @rickeylev for getting this merged :) ... had to prioritize some other work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rickeylev rickeylev rickeylev approved these changes

@aignas aignas Awaiting requested review from aignas aignas is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

"File size too large" error when making wheel

2 participants

@devtbi @rickeylev