Skip to content

(bzlmod) Index responses containing relative paths for wheels causes bazel downloader to fail. #2150

@WillMorrisonEnspi

Description

@WillMorrisonEnspi

🐞 bug report

Affected Rule

The issue is caused by: _create_wheel_repos passing index-relative paths to whl_library instead of full URLs.

Is this a regression?

Probably not.

Description

The current implementation of whl_library passes URLs parsed from simpleapi_download verbatim to rctx.download. However, some indexes (like AWS CodeArtifact) return relative URLs. This causes the bazel downloader to choke when experimental_index_url is set to an index that returns relative URLs.

Expand to show GET response for private index vs pypi

curl --basic -u "aws:$auth_token" 'https://<redacted>.d.codeartifact.eu-central-1.amazonaws.com/pypi/packages-prod/simple/mypy-extensions/'

<!DOCTYPE html>
<html>
  <head>
    <title>Links for mypy-extensions</title>
  </head>
  <body>
    <h1>Links for mypy-extensions</h1>
      <a href="0.1.0/mypy_extensions-0.1.0.tar.gz#sha256=6942c0429e505a0cdcd0468fe897960fbd8113f84cb9bde1540120bc8b68eca9"  data-gpg-sig="false" >mypy_extensions-0.1.0.tar.gz</a>
      <br/>
      <a href="0.2.0/mypy_extensions-0.2.0.tar.gz#sha256=b8f40b3bca27843f6ca911d63550799aa2586804d40da3ab087bab8130da3ebf"  data-gpg-sig="false" >mypy_extensions-0.2.0.tar.gz</a>
      <br/>
      <a href="0.3.0/mypy_extensions-0.3.0.tar.gz#sha256=c4553f2475996c912658d3c2f398766cd47a65e0c2c6bb27ac8dea94a21640f4"  data-gpg-sig="false" >mypy_extensions-0.3.0.tar.gz</a>
      <br/>
      <a href="0.4.0/mypy_extensions-0.4.0.tar.gz#sha256=118ec27adb66c682dc842bc2ad48d6d5218867df5f90ca713f4deb3760324fd3"  data-gpg-sig="false" >mypy_extensions-0.4.0.tar.gz</a>
      <br/>
      <a href="0.4.1/mypy_extensions-0.4.1-py2.py3-none-any.whl#sha256=b16cabe759f55e3409a7d231ebd2841378fb0c27a5d1994719e340e4f429ac3e"  data-gpg-sig="false" >mypy_extensions-0.4.1-py2.py3-none-any.whl</a>
      <br/>
      <a href="0.4.1/mypy_extensions-0.4.1.tar.gz#sha256=37e0e956f41369209a3d5f34580150bcacfabaa57b33a15c0b25f4b5725e0812"  data-gpg-sig="false" >mypy_extensions-0.4.1.tar.gz</a>
      <br/>
      <a href="0.4.2/mypy_extensions-0.4.2-py2.py3-none-any.whl#sha256=c77e34350c01523cd63b6dc30b36515107b2c3cbacc8392d51d33ea18d59f8f1"  data-gpg-sig="false" >mypy_extensions-0.4.2-py2.py3-none-any.whl</a>
      <br/>
      <a href="0.4.2/mypy_extensions-0.4.2.tar.gz#sha256=a161e3b917053de87dbe469987e173e49fb454eca10ef28b48b384538cc11458"  data-gpg-sig="false" >mypy_extensions-0.4.2.tar.gz</a>
      <br/>
      <a href="0.4.3/mypy_extensions-0.4.3-py2.py3-none-any.whl#sha256=090fedd75945a69ae91ce1303b5824f428daf5a028d2f6ab8a299250a846f15d"  data-gpg-sig="false" >mypy_extensions-0.4.3-py2.py3-none-any.whl</a>
      <br/>
      <a href="0.4.3/mypy_extensions-0.4.3.tar.gz#sha256=2d82818f5bb3e369420cb3c4060a7970edba416647068eb4c5343488a6c604a8"  data-gpg-sig="false" >mypy_extensions-0.4.3.tar.gz</a>
      <br/>
      <a href="0.4.4/mypy_extensions-0.4.4.tar.gz#sha256=c8b707883a96efe9b4bb3aaf0dcc07e7e217d7d8368eec4db4049ee9e142f4fd" data-requires-python="&gt;=2.7" data-gpg-sig="false" >mypy_extensions-0.4.4.tar.gz</a>
      <br/>
      <a href="1.0.0/mypy_extensions-1.0.0-py3-none-any.whl#sha256=4392f6c0eb8a5668a69e23d168ffa70f0be9ccfd32b5cc2d26a34ae5b844552d" data-requires-python="&gt;=3.5" data-gpg-sig="false" >mypy_extensions-1.0.0-py3-none-any.whl</a>
      <br/>
      <a href="1.0.0/mypy_extensions-1.0.0.tar.gz#sha256=75dbf8955dc00442a438fc4d0666508a9a97b6bd41aa2f0ffe9d2f2725af0782" data-requires-python="&gt;=3.5" data-gpg-sig="false" >mypy_extensions-1.0.0.tar.gz</a>
      <br/>
  </body>
</html>

curl 'https://pypi.org/simple/mypy-extensions/'

<!DOCTYPE html>
<html>
  <head>
    <meta name="pypi:repository-version" content="1.1">
    <title>Links for mypy-extensions</title>
  </head>
  <body>
    <h1>Links for mypy-extensions</h1>
<a href="https://files.pythonhosted.org/packages/77/9c/cad8e95677e005fc6d223f850e59a1751657a314e7f33aa06ffd5e209f28/mypy_extensions-0.1.0.tar.gz#sha256=6942c0429e505a0cdcd0468fe897960fbd8113f84cb9bde1540120bc8b68eca9" >mypy_extensions-0.1.0.tar.gz</a><br />
<a href="https://files.pythonhosted.org/packages/aa/61/2c140b126809291baeb5642b626310c12373c0fa978fb7639ffeb5feaf5d/mypy_extensions-0.2.0.tar.gz#sha256=b8f40b3bca27843f6ca911d63550799aa2586804d40da3ab087bab8130da3ebf" >mypy_extensions-0.2.0.tar.gz</a><br />
<a href="https://files.pythonhosted.org/packages/f7/51/a55d2c14e37821e6938b2e150afd50f0ed9ee4beaccb5d71b1014be63e6c/mypy_extensions-0.3.0.tar.gz#sha256=c4553f2475996c912658d3c2f398766cd47a65e0c2c6bb27ac8dea94a21640f4" >mypy_extensions-0.3.0.tar.gz</a><br />
<a href="https://files.pythonhosted.org/packages/42/c8/3503632780b856dfd1ee4b3f6c61733833f9f58fe1a608dc8406054de99e/mypy_extensions-0.4.0.tar.gz#sha256=118ec27adb66c682dc842bc2ad48d6d5218867df5f90ca713f4deb3760324fd3" >mypy_extensions-0.4.0.tar.gz</a><br />
<a href="https://files.pythonhosted.org/packages/4d/72/8d54e2b296631b9b14961d583e56e90d9d7fba8a240d5ce7f1113cc5e887/mypy_extensions-0.4.1-py2.py3-none-any.whl#sha256=b16cabe759f55e3409a7d231ebd2841378fb0c27a5d1994719e340e4f429ac3e" data-dist-info-metadata="sha256=15da51178c8af2bed1a429804dd2887b68ad35fdd404898e5dd429a99fa010e7" data-core-metadata="sha256=15da51178c8af2bed1a429804dd2887b68ad35fdd404898e5dd429a99fa010e7">mypy_extensions-0.4.1-py2.py3-none-any.whl</a><br />
<a href="https://files.pythonhosted.org/packages/c2/92/3cc05d1206237d54db7b2565a58080a909445330b4f90a6436302a49f0f8/mypy_extensions-0.4.1.tar.gz#sha256=37e0e956f41369209a3d5f34580150bcacfabaa57b33a15c0b25f4b5725e0812" >mypy_extensions-0.4.1.tar.gz</a><br />
<a href="https://files.pythonhosted.org/packages/55/aa/2df16f9c6215b10c472fd7564b5f2c5644c13ab9c26d83b1c9373c11d97d/mypy_extensions-0.4.2-py2.py3-none-any.whl#sha256=c77e34350c01523cd63b6dc30b36515107b2c3cbacc8392d51d33ea18d59f8f1" data-dist-info-metadata="sha256=753214ffa231c23df663310f4d19d0d9fd75d729ff29de61335aa8b5cc3c1b5b" data-core-metadata="sha256=753214ffa231c23df663310f4d19d0d9fd75d729ff29de61335aa8b5cc3c1b5b">mypy_extensions-0.4.2-py2.py3-none-any.whl</a><br />
<a href="https://files.pythonhosted.org/packages/1e/2f/7bba47eb58f62a473387cd7658dedd0bedb4b0fa9d530bbbfa0a6d23034a/mypy_extensions-0.4.2.tar.gz#sha256=a161e3b917053de87dbe469987e173e49fb454eca10ef28b48b384538cc11458" >mypy_extensions-0.4.2.tar.gz</a><br />
<a href="https://files.pythonhosted.org/packages/5c/eb/975c7c080f3223a5cdaff09612f3a5221e4ba534f7039db34c35d95fa6a5/mypy_extensions-0.4.3-py2.py3-none-any.whl#sha256=090fedd75945a69ae91ce1303b5824f428daf5a028d2f6ab8a299250a846f15d" data-dist-info-metadata="sha256=e801077942edd68f705e69e548d4aedd0790731e18c2af0e2a107af3d486d29e" data-core-metadata="sha256=e801077942edd68f705e69e548d4aedd0790731e18c2af0e2a107af3d486d29e">mypy_extensions-0.4.3-py2.py3-none-any.whl</a><br />
<a href="https://files.pythonhosted.org/packages/63/60/0582ce2eaced55f65a4406fc97beba256de4b7a95a0034c6576458c6519f/mypy_extensions-0.4.3.tar.gz#sha256=2d82818f5bb3e369420cb3c4060a7970edba416647068eb4c5343488a6c604a8" >mypy_extensions-0.4.3.tar.gz</a><br />
<a href="https://files.pythonhosted.org/packages/d6/c6/7809c27b5c5dacb9f6537e9489d969b266f7091204c75a03048edcb4daf9/mypy_extensions-0.4.4.tar.gz#sha256=c8b707883a96efe9b4bb3aaf0dcc07e7e217d7d8368eec4db4049ee9e142f4fd" data-requires-python="&gt;=2.7" >mypy_extensions-0.4.4.tar.gz</a><br />
<a href="https://files.pythonhosted.org/packages/2a/e2/5d3f6ada4297caebe1a2add3b126fe800c96f56dbe5d1988a2cbe0b267aa/mypy_extensions-1.0.0-py3-none-any.whl#sha256=4392f6c0eb8a5668a69e23d168ffa70f0be9ccfd32b5cc2d26a34ae5b844552d" data-requires-python="&gt;=3.5" data-dist-info-metadata="sha256=617d60dffb00826a22f5662114d8111d02fa29ffb1ec67e39d6814084ef75486" data-core-metadata="sha256=617d60dffb00826a22f5662114d8111d02fa29ffb1ec67e39d6814084ef75486">mypy_extensions-1.0.0-py3-none-any.whl</a><br />
<a href="https://files.pythonhosted.org/packages/98/a4/1ab47638b92648243faf97a5aeb6ea83059cc3624972ab6b8d2316078d3f/mypy_extensions-1.0.0.tar.gz#sha256=75dbf8955dc00442a438fc4d0666508a9a97b6bd41aa2f0ffe9d2f2725af0782" data-requires-python="&gt;=3.5" >mypy_extensions-1.0.0.tar.gz</a><br />
</body>
</html>
<!--SERIAL 16697020-->

🔬 Minimal Reproduction

MODULE.bazel

bazel_dep(name = "rules_python", version = "0.35.0")

python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    is_default = True,
    python_version = "3.11.5",
)

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    hub_name = "reqs",
    experimental_index_url = "https://<redacted>.d.codeartifact.eu-central-1.amazonaws.com/pypi/packages-prod/simple",
    python_version = "3.11.5",
    requirements_lock = "//:requirements.txt",
)
use_repo(pip, "reqs")

requirements.txt

mypy-extensions==1.0.0 \
    --hash=sha256:4392f6c0eb8a5668a69e23d168ffa70f0be9ccfd32b5cc2d26a34ae5b844552d \
    --hash=sha256:75dbf8955dc00442a438fc4d0666508a9a97b6bd41aa2f0ffe9d2f2725af0782

Running bazel fetch @reqs//mypy_extensions with the above files should reproduce the error below.

🔥 Exception or Error


ERROR: An error occurred during the fetch of repository 'rules_python~~pip~reqs_311_mypy_extensions_py3_none_any_4392f6c0':
   Traceback (most recent call last):
        File "/home/wmorrison/.cache/bazel/_bazel_wmorrison/0c20006a190ede26aadee6a87e8d620f/external/rules_python~/python/private/pypi/whl_library.bzl", line 216, column 31, in _whl_library_impl
                result = rctx.download(
Error in download: java.io.IOException: Bad URL: 1.0.0/mypy_extensions-1.0.0-py3-none-any.whl

🌍 Your Environment

Operating System:

Ubuntu Linux

Output of bazel version:


Bazelisk version: v1.20.0
Build label: 7.3.1
Build target: @@//src/main/java/com/google/devtools/build/lib/bazel:BazelServer
Build time: Mon Aug 19 16:12:50 2024 (1724083970)
Build timestamp: 1724083970
Build timestamp as int: 1724083970

Rules_python version:

  0.35.0

Anything else relevant?

Related to #1357

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions