URL fragments generate invalid "wheel" --hash arguments #1194
Description
🐞 bug report
Affected Rule
python/pip_install/requirements_parser.bzl:parse
Is this a regression?
This change in behavior was introduced in 0.19.0.
Description
The requirements_parser produces bad output when it encounters the # symbol in a dependency URL.
It appends the --hash= directive that follows to extra_pip_args, which subsequently breaks all pip wheel commands.
The reference for the requirements file format defines comments as:
- Lines that begin with
#. - Content preceded by
#(aspacefollowed by#).
Example breaking build file:
#requirements_lock.txt
requests @ https://github.com/psf/requests/releases/download/v2.29.0/requests-2.29.0.tar.gz#sha1=3897c249b51a1a405d615a8c9cb92e5fdbf0dd49 \
--hash=sha256:eca58eb564b134e4ff521a02aa6f566c653835753e1fc8a50a20cb6bee4673cd
# via -r requirements.txt
setuptools==67.4.0 \
--hash=sha256:f106dee1b506dee5102cc3f3e9e68137bbad6d47b616be7991714b0c62204251
# via -r requirements.txt
The problem ultimately resides in this line of code:
I started work on a patch, but realized that _handleConsumeSpace discards spaces, making it difficult to determine whether the previous consumed character was a space.
To conform with the spec, I think we'd need to either modify the way we manage the buffer or track whether the last action was _handleConsumeSpace.
When evaluating whether the current parser is "correct enough", I suppose some questions to answer are:
Do#shaX=...in URLs even provide value given the explicit--hasharguments? (My guess: no).Is it still worth fixing given that#in a URL is parse-able bypip?
Edit: I was mistaken about the buffer. This is easier to implement than I thought. I can open a supporting PR.
🔬 Minimal Reproduction
This patch creates a test for the case.
diff --git a/python/pip_install/private/test/requirements_parser_tests.bzl b/python/pip_install/private/test/requirements_parser_tests.bzl
index c13ec20..9f98d4a 100644
--- a/python/pip_install/private/test/requirements_parser_tests.bzl
+++ b/python/pip_install/private/test/requirements_parser_tests.bzl
@@ -62,6 +62,9 @@ SomeProject == 1.3 # This is a comment
FooProject==1.0.0
# Comment
BarProject==2.0.0 #Comment
+""").requirements)
+ asserts.equals(env, [("requests", "requests @ https://github.com/psf/requests/releases/download/v2.29.0/requests-2.29.0.tar.gz#sha1=3897c249b51a1a405d615a8c9cb92e5fdbf0dd49")], parse("""\
+requests @ https://github.com/psf/requests/releases/download/v2.29.0/requests-2.29.0.tar.gz#sha1=3897c249b51a1a405d615a8c9cb92e5fdbf0dd49
""").requirements)
# Multiline🔥 Exception or Error
The --hash argument from the dependency defined via URL is passed to all wheel commands.
no such option: --hash
Traceback (most recent call last):
File ".../external/python39_x86_64-unknown-linux-gnu/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File ".../external/python39_x86_64-unknown-linux-gnu/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File ".../external/rules_python/python/pip_install/tools/wheel_installer/wheel_installer.py", line 454, in <module>
main()
File ".../external/rules_python/python/pip_install/tools/wheel_installer/wheel_installer.py", line 431, in main
subprocess.run(pip_args, check=True, env=env)
File ".../external/python39_x86_64-unknown-linux-gnu/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['.../python39_x86_64-unknown-linux-gnu/bin/python3', '-m', 'pip', '--isolated', 'wheel', '--no-deps', '--hash=sha256:eca58eb564b134e4ff521a02aa6f566c653835753e1fc8a50a20cb6bee4673cd', '-r', '/tmp/tmpka3n51xj']' returned non-zero exit status 2.
🌍 Your Environment
Operating System:
Ubuntu 20.04.6 LTS
Output of bazel version:
6.1.2
Rules_python version:
0.21.0