Support http/https URLs in `uv python --python-downloads-json-url` #14687

geofft · 2025-07-17T18:54:32Z

No description provided.

geofft · 2025-07-17T19:11:47Z

I'll fix up docs and I can add some test coverage.

This lets you do things like

target/debug/uv python list --python-downloads-json-url https://github.com/astral-sh/uv/raw/0.7.12/crates/uv-python/download-metadata.json

The high-level goal here is making it a little bit easier for users to roll back if we unexpectedly break something in python-build-standalone. I also intend to add, likely in separate PRs,

a --uv-tag 0.7.12 option that is syntax sugar for the above
maybe some form of explicit versioning to the JSON, retroactively calling the current format version 1 but printing a nice message if we see {"version": "2", ...} (we do not necessarily have to ever move to version 2, but having the error is nice)
something to astral-sh/setup-uv to let you use this option

zanieb · 2025-07-17T22:30:06Z

crates/uv-python/src/downloads.rs

+        &self,
+        downloader: &'a ManagedPythonDownloader,
+    ) -> impl Iterator<Item = &'a ManagedPythonDownload> {
+        let s = self.clone();


Why do we need to clone here? That seems weird?

Huh, I needed this to get past some sort of lifetiime issue because the iterator captures self, but it seems fine now. I must have fixed something else better. I'll get rid of it, thanks!

zanieb · 2025-07-17T22:32:01Z

I might need a couple sentence summary in the PR description about the high-level approach here.

geofft · 2025-07-17T23:18:03Z

I might need a couple sentence summary in the PR description about the high-level approach here.

In terms of implementation, or something more like design/intended semantics? In terms of implementation, this is broken down into two commits which you might prefer for reviewing, but the change is basically:

Instead of having static methods on ManagedPythonDownload that populate a once_cell, make an explicit ManagedPythonDownloader object to hold that state, and adjust a handful of 'static lifetimes as needed
Make the ManagedPythonDownloader constructor async, so we can do HTTP requests from it
Also have its callers pass in a client
Actually do the HTTP request

Oh, one note which I forgot to mention is that per serde::from_reader's docs, you definitely shouldn't call it on an unbuffered reader (I ran strace and confirmed it's read(2)ing one byte at a time) and you might still get better results calling it on a big string than on a buffered reader. So I'm reading the whole thing into memory regardless of source, which should be fine because it's under two megabytes at the moment.

In terms of semantics, this breaks the exciting corner case of parsing irc://localhost/path/to/file.json as the local file /path/to/file.json and instead treats that as a filename. An http/https URL is handled (hopefully with the normal set of network configuration for uv?), a valid file URL is handled, an invalid file URL (with a hostname, except on Windows where you get a UNC path back, or with %00 or something similarly exciting) is an error, and everything else is treated as a path. There should be no other changes to semantics.

Reviewing #14687, I noticed that we had implemented a `Url::from_url_or_path`-like function, but it wasn't reusable. This change `Verbatim::from_url_or_path` so we can use it in other places too. The PEP 508 parser is an odd place for this, but that's where `VerbatimUrl` and `Scheme` are already living.

konstin

Needs a test as mentioned, the code is looking good already.

konstin · 2025-07-18T10:12:23Z

crates/uv/src/commands/python/install.rs

            })
            .into_iter()
-            .map(|a| InstallRequest::new(a, python_downloads_json_url.as_deref()))
+            .map(|a| InstallRequest::new(a, &downloader))


Not your fault but I'm only now seeing it.

Suggested change

.map(|a| InstallRequest::new(a, &downloader))

.map(|request| InstallRequest::new(request, &downloader))

konstin · 2025-07-18T10:12:44Z

crates/uv/src/commands/python/install.rs

            .iter()
            .map(|target| PythonRequest::parse(target.as_str()))
-            .map(|a| InstallRequest::new(a, python_downloads_json_url.as_deref()))
+            .map(|a| InstallRequest::new(a, &downloader))


Suggested change

.map(|a| InstallRequest::new(a, &downloader))

.map(|request| InstallRequest::new(request, &downloader))

konstin · 2025-07-18T10:16:21Z

crates/uv-python/src/downloads.rs

+        let s = self.clone();
+        downloader
+            .iter_all()
+            .filter(move |download| s.satisfied_by_download(download))


Suggested change

let s = self.clone();

downloader

.iter_all()

.filter(move |download| s.satisfied_by_download(download))

downloader

.iter_all()

.filter(move |download| self.satisfied_by_download(download))

konstin · 2025-07-18T10:18:38Z

crates/uv/src/commands/python/list.rs

            .as_ref()
-            .map(|a| PythonDownloadRequest::iter_downloads(a, python_downloads_json_url.as_deref()))
-            .transpose()?
+            .map(|a| PythonDownloadRequest::iter_downloads(a, &downloader))


Suggested change

.map(|a| PythonDownloadRequest::iter_downloads(a, &downloader))

.map(|request| request.iter_downloads(&downloader))

konstin · 2025-07-18T10:19:27Z

crates/uv-python/src/downloads.rs

-    ) -> Result<&'static ManagedPythonDownload, Error> {
-        if let Some(download) = request.iter_downloads(python_downloads_json_url)?.next() {
+    ) -> Result<&ManagedPythonDownload, Error> {
+        if let Some(download) = request.iter_downloads(self).next() {


This call looks backwards, flipping the method so that the call is self.iter_downloads(request) would look more natural

konstin · 2025-07-18T11:06:23Z

crates/uv-python/src/downloads.rs

-
-                let file = fs_err::File::open(json_source.as_ref())?;
+    ) -> Result<Self, Error> {
+        // Although read_url() handles file:// URLs and converts them to local file reads, here we


I tried improving the URL parsing and reusability situation in #14712, but I didn't get anything that's noticeable better than the current logic

Reviewing #14687, I noticed that we had implemented a `Url::from_url_or_path`-like function, but it wasn't reusable. This change `Verbatim::from_url_or_path` so we can use it in other places too. The PEP 508 parser is an odd place for this, but that's where `VerbatimUrl` and `Scheme` are already living.

zanieb · 2025-07-18T12:24:27Z

crates/uv-python/src/downloads.rs

    ///
    /// If there is no stable version matching the request, a compatible pre-release version will
    /// be searched for — even if a pre-release was not explicitly requested.
    pub fn from_request(


With the rename, this is a little weird? Like ManagedPythonDownloader::from_request -> ManagedPythonDownload when I'd expect from_.. to return Self.

Yeah, for this and Konsti's comment on "this call looks backwards" I was trying to minimize the amount of textual churn, but probably better to not optimize for that (especially now that reviewers have seen the commits so far :) ), I can rework things a bit.

Would be resolved by #14687 (comment)

Makes sense. In general, don't be afraid to refactor in uv.

zanieb · 2025-07-18T12:26:04Z

crates/uv-python/src/downloads.rs

 }

-impl ManagedPythonDownload {
+impl ManagedPythonDownloader {


I might rename this to ManagedPythonDownloads then have download_for_requestand iter_downloads methods (or just find and iter)? This doesn't actually download the versions, which is a little misleading.

I think the rename is also a little confusing because the client you pass to this struct won't be used by ManagedPythonDownload for the actual download

Yes, that occurred to me too. Maybe ManagedPythonDownloadList for a little more visual distinction than the plural?

Yeah I initially suggested that too and edited it out. I don't mind either way — ManagedPythonDownloads is a little closer to our typical naming in the project.

zanieb · 2025-07-18T12:27:55Z

crates/uv-python/src/downloads.rs

-    /// Note: The list is generated on the first call to this function.
-    /// so `python_downloads_json_url` is only used in the first call to this function.
-    pub fn iter_all(
+    pub async fn new(


This should have a docstring. What's the tldr on why this can't just take a BaseClientBuilder? I'm not worried about the performance, but it'd be a little clearer that it doesn't necessarily need a materialized client.

I looked at the callsites and it seems like you have a BaseClientBuilder you could provide instead?

It could, the only reason is that two of its callers construct a client shortly after calling this, and so if we passed a builder, we'd be calling .build() twice.. If that's not a problem I can definitely change that (and lazy-construct the client if we're in the HTTP(S) case only). But I was worried I'd have to do something like pass a &mut Option<BaseClient> to pass the constructed client back to the caller to reuse it, or something.

I think it's probably fine if we build twice.

zanieb · 2025-07-18T12:28:53Z

crates/uv-python/src/downloads.rs

+                        if let Ok(path) = url.to_file_path() {
+                            Source::Path(Cow::Owned(path))
+                        } else {
+                            return Err(Error::InvalidUrlFormat(url));
+                        }


nit: I'd probably write this as a let Ok(...) else return instead of if let Ok(...) else return so you can early return

In preparation for supporting download sources over HTTP, change the implicit creation of the download list to an explicit async constructor. Also pass it a BaseClient which will be used in the next commit.

…wnload list If the JSON does not deserialize, check to see if any key is named "version" and print a different error message hinting at upgrading uv. The intention is that we will add this field on backwards-incompatible changes (and set it to an integer or string or something else that does not deserialize into a JsonPythonDownload).

konstin

we need a (wiremock) test, otherwise r+ me on the code.

crates/uv-python/src/downloads.rs

konstin · 2025-08-06T10:23:16Z

crates/uv-settings/src/settings.rs

    /// URL pointing to JSON of custom Python installations.
    ///
-    /// Note that currently, only local paths are supported.
+    /// This can be a local path or file://, http://, or https:// URL.


This reads like there is a particle missing

From reviewing #14687, I was reminded that every base client operation requires manually destructuring the `NetworkSettings`. We can avoid that by passing the global network settings directly.

MeitarR · 2025-10-14T11:56:24Z

Hi @geofft , this feature is something I'm very interested in as well.

I noticed there hasn't been any recent activity - are you still planning to work on it? If not, I'd be happy to help move it forward or open a continuation PR based on your work.

…16542) continuation PR based on #14687 --------- Co-authored-by: Geoffrey Thomas <geofft@ldpreload.com> Co-authored-by: Aria Desires <aria.desires@gmail.com>

geofft requested review from jtfmumm, konstin and zanieb July 17, 2025 18:54

geofft force-pushed the geofft/python-download-old-versions branch from 05f956e to af86992 Compare July 17, 2025 18:58

geofft temporarily deployed to uv-test-registries July 17, 2025 19:01 — with GitHub Actions Inactive

geofft mentioned this pull request Jul 17, 2025

Add UV_PYTHON_DOWNLOADS_JSON_URL to set custom managed python sources #10939

Merged

zanieb reviewed Jul 17, 2025

View reviewed changes

konstin mentioned this pull request Jul 18, 2025

Add a reusable path-or-URL parser #14712

Merged

konstin reviewed Jul 18, 2025

View reviewed changes

zanieb reviewed Jul 18, 2025

View reviewed changes

konstin added the enhancement New feature or improvement to existing functionality label Jul 18, 2025

MeitarR mentioned this pull request Apr 30, 2025

improvements for creating and using offline mirror for uv-python #12838

Open

3 tasks

geofft added 7 commits July 18, 2025 23:30

uv-python: Refactor ManagedPythonDownload state management

5f42c7c

In preparation for supporting download sources over HTTP, change the implicit creation of the download list to an explicit async constructor. Also pass it a BaseClient which will be used in the next commit.

update docs

dbeca98

comments from Konsti

01998f4

Support http/https URLs in uv python --python-downloads-json-url

e20103f

Rename ManagedPythonDownloadList methods for readability

66937e3

review comment

ecebc91

geofft force-pushed the geofft/python-download-old-versions branch 2 times, most recently from e7ab6bc to 25dd584 Compare August 4, 2025 17:50

konstin changed the title ~~Support http/https URLs in uv python --python-downloads-json-url~~ Support http/https URLs in uv python --python-downloads-json-url Aug 4, 2025

konstin reviewed Aug 6, 2025

View reviewed changes

konstin mentioned this pull request Aug 6, 2025

Don't destructure global network settings #15105

Closed

konstin removed the request for review from jtfmumm October 14, 2025 11:59

MeitarR mentioned this pull request Oct 31, 2025

Support http/https URLs in uv python --python-downloads-json-url #16542

Merged

	.map(\|a\| InstallRequest::new(a, &downloader))
	.map(\|request\| InstallRequest::new(request, &downloader))

	.map(\|a\| PythonDownloadRequest::iter_downloads(a, &downloader))
	.map(\|request\| request.iter_downloads(&downloader))

Support http/https URLs in uv python --python-downloads-json-url #14687

Are you sure you want to change the base?

Support http/https URLs in uv python --python-downloads-json-url #14687

Uh oh!

Conversation

geofft commented Jul 17, 2025

Uh oh!

geofft commented Jul 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zanieb commented Jul 17, 2025

Uh oh!

geofft commented Jul 17, 2025

Uh oh!

konstin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zanieb Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zanieb Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

konstin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MeitarR commented Oct 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Support http/https URLs in `uv python --python-downloads-json-url` #14687

Support http/https URLs in `uv python --python-downloads-json-url` #14687

zanieb Jul 18, 2025 •

edited

Loading

zanieb Jul 18, 2025 •

edited

Loading