Support http/https URLs in `uv python --python-downloads-json-url` #16542

MeitarR · 2025-10-31T21:51:37Z

continuation PR based on #14687

MeitarR · 2025-10-31T23:17:25Z

I successfully rebased the old PR on the updated main. Later, I will also try to fix the CR comments from there.

MeitarR · 2025-11-07T19:30:40Z

@konstin, I added a WireMock test and fixed the confusing comment. It should be ready for review.

Tell me if the test is what you expected

Gankra

Wow this is great! I have a few nits but this seems solid.

docs/reference/environment.md

crates/uv-static/src/env_vars.rs

crates/uv-python/src/installation.rs

Gankra · 2025-11-10T22:22:24Z

crates/uv-python/src/downloads.rs

+            Source::Path(ref path) => fs_err::read(path.as_ref())?.into(),
+            Source::Http(ref url) => {
+                let (mut reader, size) = read_url(url, client).await?;
+                let capacity = size.and_then(|s| s.try_into().ok()).unwrap_or(1_048_576);
+                let mut buf = Vec::with_capacity(capacity);
+                reader.read_to_end(&mut buf).await?;
+                buf.into()
+            }


Should we be adding context to these errors..?

How would we do that?

I see that read_url already has different error types.

(BTW, I see that there is another usage of read_url later in the file that does the same)

It's probably fine.

You can test this by running with an arbitrary large file (so you can abort before it finished), and turning the network off mid-download:

$ UV_HTTP_TIMEOUT=1 cargo run python install 3.14 --python-downloads-json-url https://files.pythonhosted.org/packages/a9/8c/3da60787bcf70add986c4ad485993026ac0ca74f2fc21410bc4eb1bb7695/torch-2.9.1-cp314-cp314t-manylinux_2_28_x86_64.whl error: error decoding response body Caused by: request or response body error Caused by: operation timed out

We should make sure that we show at least the URL in the error message.

For example, this is the error message when network isn't available when starting the download:

$ UV_HTTP_TIMEOUT=1 cargo run python install 3.14 --python-downloads-json-url https://files.pythonhosted.org/packages/a9/8c/3da60787bcf70add986c4ad485993026ac0ca74f2fc21410bc4eb1bb7695/torch-2.9.1-cp314-cp314t-manylinux_2_28_x86_64.whl error: Failed to download https://files.pythonhosted.org/packages/a9/8c/3da60787bcf70add986c4ad485993026ac0ca74f2fc21410bc4eb1bb7695/torch-2.9.1-cp314-cp314t-manylinux_2_28_x86_64.whl Caused by: error sending request for url (https://files.pythonhosted.org/packages/a9/8c/3da60787bcf70add986c4ad485993026ac0ca74f2fc21410bc4eb1bb7695/torch-2.9.1-cp314-cp314t-manylinux_2_28_x86_64.whl) Caused by: client error (Connect) Caused by: dns error Caused by: failed to lookup address information: Name or service not known

We're also not retrying network failures as we do else, because we're using the python download client that has automatic retries deactivated, but don't implement our own retry loop, and don't do any caching. read_url fits badly here, CachedClient has the features we want: Retries, serde integration and caching.

Regarding the retrying, can you add a test to network.rs for --python-downloads-json-url?

Added context for the errors there

$ UV_HTTP_TIMEOUT=1 cargo run python install 3.14 --python-downloads-json-url https://files.pythonhosted.org/packages/a9/8c/3da60787bcf70add986c4ad485993026ac0ca74f2fc21410bc4eb1bb7695/torch-2.9.1-cp314-cp314t-manylinux_2_28_x86_64.whl error: Error while fetching remote python downloads json from 'https://files.pythonhosted.org/packages/a9/8c/3da60787bcf70add986c4ad485993026ac0ca74f2fc21410bc4eb1bb7695/torch-2.9.1-cp314-cp314t-manylinux_2_28_x86_64.whl' Caused by: error decoding response body Caused by: request or response body error Caused by: operation timed out

About the other comments, I will address them in a later PR, as this PR is very conflict-prone

(Agreed to have this as a followup)

Gankra · 2025-11-10T22:23:57Z

crates/uv-python/src/downloads.rs

+            Source::Path(ref path) => fs_err::read(path.as_ref())?.into(),
+            Source::Http(ref url) => {
+                let (mut reader, size) = read_url(url, client).await?;
+                let capacity = size.and_then(|s| s.try_into().ok()).unwrap_or(1_048_576);


Preallocating a 1MB buffer seems a bit aggressive? Is that a pattern copied from somewhere else?

From reading the code (which @geofft wrote), I see that the 1MB is only relevant when we can't get the size of the buffer, and the only real case (if I understand correctly) is when in the HTTP response there where no content-length header.

Maybe it is somehow related to #14687 (comment)

.bytes(), the non-streaming version, has a better collector for this, but it doesn't really fit with the current read_url

Gankra · 2025-11-10T22:24:58Z

crates/uv-python/src/downloads.rs

+                // this by parsing into a Map<String, IgnoredAny> which allows any valid JSON on the
+                // value side. (Because it's zero-sized, Clippy suggests Set<String>, but that won't
+                // have the same parsing effect.)
+                #[allow(clippy::zero_sized_map_values)]


Interesting nod to future-compat, sure.

Gankra · 2025-11-10T22:25:13Z

crates/uv-python/src/downloads.rs

+                #[allow(clippy::zero_sized_map_values)]
+                |e| {
+                    let source = match json_source {
+                        Source::BuiltIn => "EMBEDDED IN THE BINARY".to_owned(),


Gankra

Should be mergeable, but this has a non-trivial rebase to do :(

In preparation for supporting download sources over HTTP, change the implicit creation of the download list to an explicit async constructor. Also pass it a BaseClient which will be used in the next commit.

…wnload list If the JSON does not deserialize, check to see if any key is named "version" and print a different error message hinting at upgrading uv. The intention is that we will add this field on backwards-incompatible changes (and set it to an integer or string or something else that does not deserialize into a JsonPythonDownload).

Co-authored-by: Aria Desires <aria.desires@gmail.com>

This MR contains the following updates: | Package | Update | Change | |---|---|---| | [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.9.9` -> `0.9.10` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>astral-sh/uv (astral-sh/uv)</summary> ### [`v0.9.10`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0910) [Compare Source](astral-sh/uv@0.9.9...0.9.10) Released on 2025-11-17. ##### Enhancements - Add support for `SSL_CERT_DIR` ([#16473](astral-sh/uv#16473)) - Enforce UTF‑8-encoded license files during `uv build` ([#16699](astral-sh/uv#16699)) - Error when a `project.license-files` glob matches nothing ([#16697](astral-sh/uv#16697)) - `pip install --target` (and `sync`) install Python if necessary ([#16694](astral-sh/uv#16694)) - Account for `python_downloads_json_url` in pre-release Python version warnings ([#16737](astral-sh/uv#16737)) - Support HTTP/HTTPS URLs in `uv python --python-downloads-json-url` ([#16542](astral-sh/uv#16542)) ##### Preview features - Add support for `--upgrade` in `uv python install` ([#16676](astral-sh/uv#16676)) - Fix handling of `python install --default` for pre-release Python versions ([#16706](astral-sh/uv#16706)) - Add `uv workspace list` to list workspace members ([#16691](astral-sh/uv#16691)) ##### Bug fixes - Don't check file URLs for ambiguously parsed credentials ([#16759](astral-sh/uv#16759)) ##### Documentation - Add a "storage" reference document ([#15954](astral-sh/uv#15954)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

MeitarR force-pushed the remote-python-downloads-json-url branch 2 times, most recently from 26d3bb9 to 259bc5f Compare October 31, 2025 22:44

MeitarR mentioned this pull request Nov 1, 2025

improvements for creating and using offline mirror for uv-python #12838

Open

3 tasks

MeitarR force-pushed the remote-python-downloads-json-url branch 3 times, most recently from 9273a33 to d4d65a2 Compare November 7, 2025 19:12

MeitarR marked this pull request as ready for review November 7, 2025 19:28

Gankra self-requested a review November 10, 2025 20:16

Gankra approved these changes Nov 10, 2025

View reviewed changes

MeitarR force-pushed the remote-python-downloads-json-url branch 2 times, most recently from 3ab69a6 to be31b12 Compare November 11, 2025 20:38

Gankra approved these changes Nov 12, 2025

View reviewed changes

MeitarR force-pushed the remote-python-downloads-json-url branch 2 times, most recently from e2c9b66 to 1d5a0a4 Compare November 14, 2025 18:37

geofft and others added 10 commits November 14, 2025 23:13

uv-python: Refactor ManagedPythonDownload state management

4127a64

In preparation for supporting download sources over HTTP, change the implicit creation of the download list to an explicit async constructor. Also pass it a BaseClient which will be used in the next commit.

update docs

91ab8f7

comments from Konsti

efa1917

Support http/https URLs in uv python --python-downloads-json-url

79aac36

Rename ManagedPythonDownloadList methods for readability

94a2756

review comment

44402c2

removed the (confusing) comment about the URL parsing ability

27d993d

add wiremock test

24823f6

Apply docs suggestions from code review

c101699

Co-authored-by: Aria Desires <aria.desires@gmail.com>

MeitarR force-pushed the remote-python-downloads-json-url branch from 1d5a0a4 to c101699 Compare November 14, 2025 21:14

add context for errors

5ec9b6f

MeitarR force-pushed the remote-python-downloads-json-url branch from 62a645f to 5ec9b6f Compare November 14, 2025 22:15

Gankra merged commit b982677 into astral-sh:main Nov 14, 2025
161 checks passed

MeitarR mentioned this pull request Nov 15, 2025

Follow-up from #16542 #16749

Draft

3 tasks

BrewTestBot mentioned this pull request Nov 17, 2025

uv 0.9.10 Homebrew/homebrew-core#254745

Merged

Support http/https URLs in uv python --python-downloads-json-url #16542

Support http/https URLs in uv python --python-downloads-json-url #16542

Uh oh!

Conversation

MeitarR commented Oct 31, 2025

Uh oh!

MeitarR commented Oct 31, 2025

Uh oh!

MeitarR commented Nov 7, 2025

Uh oh!

Gankra left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Gankra left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Support http/https URLs in `uv python --python-downloads-json-url` #16542

Support http/https URLs in `uv python --python-downloads-json-url` #16542