Preserve absolute/relative paths in lockfiles#18176
Conversation
ca82125 to
c0c3e75
Compare
da28d9b to
dd390c1
Compare
|
I consider changing the lockfiles fine - we shuffled those around before in regular releases - though the |
| pub fn try_relative_to_if( | ||
| path: impl AsRef<Path>, | ||
| base: impl AsRef<Path>, | ||
| should_relativize: bool, |
There was a problem hiding this comment.
Should we pass the verbatim URL here instead so it can capture the logic about was_given_absolute?(probably needs to move to verbatim URL then)
I also noticed install_path and url: VerbatimUrl always come as a pair, do we want to bundle them?
There was a problem hiding this comment.
I think if we bundled install_path and url together then it may make sense to have that type handle this.
An initial version of this PR had try_relative_to_if take the VerbatimUrl but I didn't feel like it made sense that way. Until we merge those, I feel like it strictly makes more sense to have them split up.
But I agree that all the code basically having the same combination of install_path and url.was_given_absolute() is an indication that there's an abstraction missing here.
I'll investigate the combination option.
There was a problem hiding this comment.
I did investigate this, and the issue I am seeing is with RegistrySourceDist and RegistryBuiltWheel where there is no install_path but rather there is an IndexUrl which could be a file.
But the approach could work for PathBuiltDist, PathSourceDist, and DirectorySourceDist.
For the time being I've pushed a commit which reverts the uv add changes for this PR and created another PR which reverts the revert and is stacked. I've marked that other one breaking,
I've requested your re-review for this one just to double check I have actually reverted all the "breaking" changes.
52ea916 to
f1b84c2
Compare
Previously it would treat paths with `<drive letter>:/` as URLs pre-emptively instead of parsing the scheme like other similar code does.
f1b84c2 to
31ca0ea
Compare
8c38f1a to
666ae61
Compare
|
I think this broke a use case I have: I used to specify some dependencies that live in the same repo and provide the same python package (and are included in mutually exclusive dependency groups) as Up to now, this would generate lock files with relative paths that can be shared between systems where the repo lives in different places. This no longer works. As a workaround, using seems to do the trick, but overall this does seem to be a bit more breaking than anticipated (producing lockfiles that aren't usable on other systems than the one that produced them, from pyproject configs that worked before)? |
|
Can you describe in more detail in what case the |
File URL dependencies containing environment variable references (e.g.,
`file:///${PROJECT_ROOT}/a`) should produce relative paths in lockfiles,
since the user is parameterizing the path for portability. Previously,
`was_given_absolute()` treated all file URLs as absolute, causing these
paths to be stored as absolute in the lockfile after PR astral-sh#18176.
https://claude.ai/code/session_018jGTUmqEdwKfaWo1zQ9jWW
|
It works fine in all the cases, but it breaks locking from older |
…les as relative (#18680) ## Summary Fix a regression caused by and reported in #18176. PEP 508 doesn't actually permit variables to be specified within these URLs but we support this probably due to needing to handle it for requirements files. To avoid a breaking change in a patch release, any `VerbatimUrl` that was parsed as a PEP 508 URL that contained variables that were expanded is always treated as relative. The determination of if a `VerbatimUrl` qualifies has to be done at creation time because otherwise we would incorrectly treat a non PEP 508 URL which contained something which looks like a variable reference as a relative path in cases where this wouldn't be correct. ## Test Plan Existing test coverage covers the non-regressed case, added a test for the regressed case.
|
Thanks for the report! This should be fixed in 0.11.0 |
…les as relative (#18680) ## Summary Fix a regression caused by and reported in #18176. PEP 508 doesn't actually permit variables to be specified within these URLs but we support this probably due to needing to handle it for requirements files. To avoid a breaking change in a patch release, any `VerbatimUrl` that was parsed as a PEP 508 URL that contained variables that were expanded is always treated as relative. The determination of if a `VerbatimUrl` qualifies has to be done at creation time because otherwise we would incorrectly treat a non PEP 508 URL which contained something which looks like a variable reference as a relative path in cases where this wouldn't be correct. ## Test Plan Existing test coverage covers the non-regressed case, added a test for the regressed case.
This MR contains the following updates: | Package | Update | Change | |---|---|---| | [uv](https://github.com/astral-sh/uv) | minor | `0.10.9` → `0.11.3` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>astral-sh/uv (uv)</summary> ### [`v0.11.3`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0113) [Compare Source](astral-sh/uv@0.11.2...0.11.3) Released on 2026-04-01. ##### Enhancements - Add progress bar for hashing phase in uv publish ([#​18752](astral-sh/uv#18752)) - Add support for ROCm 7.2 ([#​18730](astral-sh/uv#18730)) - Emit abi3t tags for every abi3 version ([#​18777](astral-sh/uv#18777)) - Expand `uv workspace metadata` with dependency information from the lock ([#​18356](astral-sh/uv#18356)) - Implement support for PEP 803 ([#​18767](astral-sh/uv#18767)) - Pretty-print platform in built wheel errors ([#​18738](astral-sh/uv#18738)) - Publish installers to `/installers/uv/latest` on the mirror ([#​18725](astral-sh/uv#18725)) - Show free-threaded Python in built-wheel errors ([#​18740](astral-sh/uv#18740)) ##### Preview features - Add `--ignore` and `--ignore-until-fixed` to `uv audit` ([#​18737](astral-sh/uv#18737)) ##### Bug fixes - Bump simple API cache ([#​18797](astral-sh/uv#18797)) - Don't drop `blake2b` hashes ([#​18794](astral-sh/uv#18794)) - Handle broken range request implementations ([#​18780](astral-sh/uv#18780)) - Remove `powerpc64-unknown-linux-gnu` from release build targets ([#​18800](astral-sh/uv#18800)) - Respect dependency metadata overrides in `uv pip check` ([#​18742](astral-sh/uv#18742)) - Support debug CPython ABI tags in environment compatibility ([#​18739](astral-sh/uv#18739)) ##### Documentation - Document `false` opt-out for `exclude-newer-package` ([#​18768](astral-sh/uv#18768), [#​18803](astral-sh/uv#18803)) ### [`v0.11.2`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0112) [Compare Source](astral-sh/uv@0.11.1...0.11.2) Released on 2026-03-26. ##### Enhancements - Add a dedicated Windows PE editing error ([#​18710](astral-sh/uv#18710)) - Make `uv self update` fetch the manifest from the mirror first ([#​18679](astral-sh/uv#18679)) - Use uv reqwest client for self update ([#​17982](astral-sh/uv#17982)) - Show `uv self update` success and failure messages with `--quiet` ([#​18645](astral-sh/uv#18645)) ##### Preview features - Evaluate extras and groups when determining auditable packages ([#​18511](astral-sh/uv#18511)) ##### Bug fixes - Skip redundant project configuration parsing for `uv run` ([#​17890](astral-sh/uv#17890)) ### [`v0.11.1`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0111) [Compare Source](astral-sh/uv@0.11.0...0.11.1) Released on 2026-03-24. ##### Bug fixes - Add missing hash verification for `riscv64gc-unknown-linux-musl` ([#​18686](astral-sh/uv#18686)) - Fallback to direct download when direct URL streaming is unsupported ([#​18688](astral-sh/uv#18688)) - Revert treating 'Dynamic' values as case-insensitive ([#​18692](astral-sh/uv#18692)) - Remove torchdata from list of packages to source from the PyTorch index ([#​18703](astral-sh/uv#18703)) - Special-case `==` Python version request ranges ([#​9697](astral-sh/uv#9697)) ##### Documentation - Cover `--python <dir>` in "Using arbitrary Python environments" ([#​6457](astral-sh/uv#6457)) - Fix version annotations for `PS_MODULE_PATH` and `UV_WORKING_DIR` ([#​18691](astral-sh/uv#18691)) ### [`v0.11.0`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0110) [Compare Source](astral-sh/uv@0.10.12...0.11.0) Released on 2026-03-23. ##### Breaking changes This release includes changes to the networking stack used by uv. While we think that breakage will be rare, it is possible that these changes will result in the rejection of certificates previously trusted by uv so we have marked the change as breaking out of an abundance of caution. The changes are largely driven by the upgrade of reqwest, which powers uv's HTTP clients, to [v0.13](https://seanmonstar.com/blog/reqwest-v013-rustls-default/) which included some breaking changes to TLS certificate verification. The following changes are included: - [`rustls-platform-verifier`](https://github.com/rustls/rustls-platform-verifier) is used instead of [`rustls-native-certs`](https://github.com/rustls/rustls-native-certs) and [`webpki`](https://github.com/rustls/webpki) for certificate verification **This change should have no effect unless you are using the `native-tls` option to enable reading system certificates.** `rustls-platform-verifier` delegates to the system for certificate validation (e.g., `Security.framework` on macOS) instead of eagerly loading certificates from the system and verifying them via `webpki`. The effects of this change will vary based on the operating system. In general, uv's certificate validation should now be more consistent with browsers and other native applications. However, this is the most likely cause of breaking changes in this release. Some previously failing certificate chains may succeed, and some previously accepted certificate chains may fail. In either case, we expect the validation to be more correct and welcome reports of regressions. In particular, because more responsibility for validating the certificate is transferred to your system's security library, some features like [CA constraints](https://support.apple.com/en-us/103255) or [revocation of certificates](https://en.wikipedia.org/wiki/Certificate_revocation) via OCSP and CRLs may now be used. This change should improve performance when using system certificate on macOS, as uv no longer needs to load all certificates from the keychain at startup. - [`aws-lc`](https://github.com/aws/aws-lc) is used instead of `ring` for a cryptography backend There should not be breaking changes from this change. We expect this to expand support for certificate signature algorithms. - `--native-tls` is deprecated in favor of a new `--system-certs` flag The `--native-tls` flag is still usable and has identical behavior to `--system-certs.` This change was made to reduce confusion about the TLS implementation uv uses. uv always uses `rustls` not `native-tls`. - Building uv on x86-64 and i686 Windows requires NASM NASM is required by `aws-lc`. If not found on the system, a prebuilt blob provided by `aws-lc-sys` will be used. If you are not building uv from source, this change has no effect. See the [CONTRIBUTING](https://github.com/astral-sh/uv/blob/b6854d77bfd0cb78157fecaf8b30126c6f16bc11/CONTRIBUTING.md#setup) guide for details. - Empty `SSL_CERT_FILE` values are ignored (for consistency with `SSL_CERT_DIR`) See [#​18550](astral-sh/uv#18550) for details. ##### Python - Enable frame pointers for improved profiling on Linux x86-64 and aarch64 See the [python-build-standalone release notes](https://github.com/astral-sh/python-build-standalone/releases/20260320) for details. ##### Enhancements - Treat 'Dynamic' values as case-insensitive ([#​18669](astral-sh/uv#18669)) - Use a dedicated error for invalid cache control headers ([#​18657](astral-sh/uv#18657)) - Enable checksum verification in the generated installer script ([#​18625](astral-sh/uv#18625)) ##### Preview features - Add `--service-format` and `--service-url` to `uv audit` ([#​18571](astral-sh/uv#18571)) ##### Performance - Avoid holding flat index lock across indexes ([#​18659](astral-sh/uv#18659)) ##### Bug fixes - Find the dynamic linker on the file system when sniffing binaries fails ([#​18457](astral-sh/uv#18457)) - Fix export of conflicting workspace members with dependencies ([#​18666](astral-sh/uv#18666)) - Respect installed settings in `uv tool list --outdated` ([#​18586](astral-sh/uv#18586)) - Treat paths originating as PEP 508 URLs which contain expanded variables as relative ([#​18680](astral-sh/uv#18680)) - Fix `uv export` for workspace member packages with conflicts ([#​18635](astral-sh/uv#18635)) - Continue to alternative authentication providers when the pyx store has no token ([#​18425](astral-sh/uv#18425)) - Use redacted URLs for log messages in cached client ([#​18599](astral-sh/uv#18599)) ##### Documentation - Add details on Linux versions to the platform policy ([#​18574](astral-sh/uv#18574)) - Clarify `FLASH_ATTENTION_SKIP_CUDA_BUILD` guidance for `flash-attn` installs ([#​18473](astral-sh/uv#18473)) - Split the dependency bots page into two separate pages ([#​18597](astral-sh/uv#18597)) - Split the alternative indexes page into separate pages ([#​18607](astral-sh/uv#18607)) ### [`v0.10.12`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#01012) [Compare Source](astral-sh/uv@0.10.11...0.10.12) Released on 2026-03-19. ##### Python - Add pypy 3.11.15 ([#​18468](astral-sh/uv#18468)) - Add support for using Python 3.6 interpreters ([#​18454](astral-sh/uv#18454)) ##### Enhancements - Include uv's target triple in version report ([#​18520](astral-sh/uv#18520)) - Allow comma separated values in `--no-emit-package` ([#​18565](astral-sh/uv#18565)) ##### Preview features - Show `uv audit` in the CLI help ([#​18540](astral-sh/uv#18540)) ##### Bug fixes - Improve reporting of managed interpreter symlinks in `uv python list` ([#​18459](astral-sh/uv#18459)) - Preserve end-of-line comments on previous entries when removing dependencies ([#​18557](astral-sh/uv#18557)) - Treat abi3 wheel Python version as a lower bound ([#​18536](astral-sh/uv#18536)) - Detect hard-float support on aarch64 kernels running armv7 userspace ([#​18530](astral-sh/uv#18530)) ##### Documentation - Add Python 3.15 to supported versions ([#​18552](astral-sh/uv#18552)) - Adjust the PyPy note ([#​18548](astral-sh/uv#18548)) - Move Pyodide to Tier 2 in the Python support policy ([#​18561](astral-sh/uv#18561)) - Move Rust and Python version support out of the Platform support policy ([#​18535](astral-sh/uv#18535)) - Update Docker guide with changes from `uv-docker-example` ([#​18558](astral-sh/uv#18558)) - Update the Python version policy ([#​18559](astral-sh/uv#18559)) ### [`v0.10.11`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#01011) [Compare Source](astral-sh/uv@0.10.10...0.10.11) Released on 2026-03-16. ##### Enhancements - Fetch Ruff release metadata from an Astral mirror ([#​18358](astral-sh/uv#18358)) - Use PEP 639 license metadata for uv itself ([#​16477](astral-sh/uv#16477)) ##### Performance - Improve distribution id performance ([#​18486](astral-sh/uv#18486)) ##### Bug fixes - Allow `--project` to refer to a `pyproject.toml` directly and reduce to a warning on other files ([#​18513](astral-sh/uv#18513)) - Disable `SYSTEM_VERSION_COMPAT` when querying interpreters on macOS ([#​18452](astral-sh/uv#18452)) - Enforce available distributions for supported environments ([#​18451](astral-sh/uv#18451)) - Fix `uv sync --active` recreating active environments when `UV_PYTHON_INSTALL_DIR` is relative ([#​18398](astral-sh/uv#18398)) ##### Documentation - Add missing `-o requirements.txt` in `uv pip compile` example ([#​12308](astral-sh/uv#12308)) - Link to organization security policy ([#​18449](astral-sh/uv#18449)) - Link to the AI policy in the contributing guide ([#​18448](astral-sh/uv#18448)) ### [`v0.10.10`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#01010) [Compare Source](astral-sh/uv@0.10.9...0.10.10) Released on 2026-03-13. ##### Python - Add CPython 3.15.0a7 ([#​18403](astral-sh/uv#18403)) ##### Enhancements - Add `--outdated` flag to `uv tool list` ([#​18318](astral-sh/uv#18318)) - Add riscv64 musl target to build-release-binaries workflow ([#​18228](astral-sh/uv#18228)) - Fetch Ruff from an Astral mirror ([#​18286](astral-sh/uv#18286)) - Improve error handling for platform detection in Python downloads ([#​18453](astral-sh/uv#18453)) - Warn if `--project` directory does not exist ([#​17714](astral-sh/uv#17714)) - Warn when workspace member scripts are skipped due to missing build system ([#​18389](astral-sh/uv#18389)) - Update build backend versions used in `uv init` ([#​18417](astral-sh/uv#18417)) - Log explicit config file path in verbose output ([#​18353](astral-sh/uv#18353)) - Make `uv cache clear` an alias of `uv cache clean` ([#​18420](astral-sh/uv#18420)) - Reject invalid classifiers, warn on license classifiers in `uv_build` ([#​18419](astral-sh/uv#18419)) ##### Preview features - Add links to `uv audit` output ([#​18392](astral-sh/uv#18392)) - Output/report formatting for `uv audit` ([#​18193](astral-sh/uv#18193)) - Switch to batched OSV queries for `uv audit` ([#​18394](astral-sh/uv#18394)) ##### Bug fixes - Avoid sharing version metadata across indexes ([#​18373](astral-sh/uv#18373)) - Bump zlib-rs to 0.6.2 to fix panic on decompression of large wheels on Windows ([#​18362](astral-sh/uv#18362)) - Filter out unsupported environment wheels ([#​18445](astral-sh/uv#18445)) - Preserve absolute/relative paths in lockfiles ([#​18176](astral-sh/uv#18176)) - Recreate Python environments under `uv tool install --force` ([#​18399](astral-sh/uv#18399)) - Respect timestamp and other cache keys in cached environments ([#​18396](astral-sh/uv#18396)) - Simplify selected extra markers in `uv export` ([#​18433](astral-sh/uv#18433)) - Send pyx mint-token requests with a proper `Content-Type` ([#​18334](astral-sh/uv#18334)) - Fix Windows operating system and version reporting ([#​18383](astral-sh/uv#18383)) ##### Documentation - Update the platform support policy with a tier 3 section including freebsd and 32-bit windows ([#​18345](astral-sh/uv#18345)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My45Mi4xIiwidXBkYXRlZEluVmVyIjoiNDMuMTAyLjIiLCJ0YXJnZXRCcmFuY2giOiJtYWluIiwibGFiZWxzIjpbIlJlbm92YXRlIEJvdCIsImF1dG9tYXRpb246Ym90LWF1dGhvcmVkIiwiZGVwZW5kZW5jeS10eXBlOjptaW5vciJdfQ==-->
Summary
Attempt to track and preserve relative/absolute paths when read from files.
File URLs are treated as absolute. Synthetic VerbatimUrls shouldn't have a
given, and are treated as relative.This means that paths passed as absolute will be output as absolute, although they may get normalized. Paths passed as relative will be output as relative but they may be relative to a different location (so that they continue to work going forwards). Previously in various places we'd either make things absolute unconditionally or relative unconditionally.
Cases which should now be fixed:
Also noteworthy is the bugfix for a windows misbehaviour. See the commit message for some more information.
Note: For now the
uv addside of this has been split off as a breaking change.Test Plan
Added missing tests, updated existing.
I believe all the changed tests are all now correct and were previously demonstrating buggy behaviour. Well, at least if you are on board with the idea that we should keep relative paths relative and absolute paths and / file URLs absolute.
Related Issues/PRs
find-linksorpath =causes absolute wheel path to become relative path when creating lock file #16602pylock.tomlexports #16514