Expand environment variables in user-provided index URLs by msabramo · Pull Request #13879 · astral-sh/uv

msabramo · 2025-06-06T04:05:54Z

Summary

This PR adds support for environment variable expansion in [[tool.uv.index]] configuration URLs, enabling dynamic and secure index configuration.

What it does:

Expands environment variables in url and publish-url fields using ${VARIABLE} and ${VARIABLE:-default} syntax
Supports tilde (~) expansion for home directory paths
Enables secure credential injection without storing sensitive information in plaintext configuration files
Allows for environment-specific configuration (dev/staging/prod) using the same pyproject.toml

Implementation details:

Added shellexpand crate dependency for robust environment variable and tilde expansion
Implemented custom Deserialize for the Index struct to perform expansion during TOML parsing
Added comprehensive error handling with descriptive messages for missing variables or expansion failures
Maintains backward compatibility with existing configurations

Example usage:

[[tool.uv.index]]
name = "artifactory"
url = "https://${ARTIFACTORY_USER}:${ARTIFACTORY_API_TOKEN}@artifactory.company.com/simple"

Test Plan

Unit tests:

test_index_environment_variable_expansion: Verifies ${VAR} syntax expansion with set environment variables
test_index_without_environment_variables: Ensures existing configurations without env vars continue to work
test_index_missing_environment_variable: Tests error handling when required environment variables are missing
test_index_tilde_expansion: Verifies ~ home directory expansion works correctly

Integration testing:

Ran /Users/abramowi/Code/OpenSource/uv/target/release/uv sync in a project that had something like:

[[tool.uv.index]]
name = "artifactory"
url = "https://${ARTIFACTORY_USER}:${ARTIFACTORY_API_TOKEN}@artifactory.company.com/artifactory/api/pypi/pypi-colorado-tools-snapshot/simple"

Ran a normal uv sync which failed:

$ uv sync -v | grep artifactory
...
  × Failed to download `vortex-sdk==0.0.35`
  ├─▶ Failed to fetch:
  │   `https://artifactory.company.com/artifactory/api/pypi/pypi-colorado-tools-snapshot/vortex-sdk/0.0.35/vortex_sdk-0.0.35-py3-none-any.whl`
  ╰─▶ HTTP status client error (401 Unauthorized) for url
      (https://artifactory.company.com/artifactory/api/pypi/pypi-colorado-tools-snapshot/vortex-sdk/0.0.35/vortex_sdk-0.0.35-py3-none-any.whl)

Built uv from this branch with cargo build --release
Ran $ /Users/abramowi/Code/OpenSource/uv/target/release/uv sync -v | grep artifactory and got:

DEBUG No cache entry for: https://artifactory.company.com/artifactory/api/pypi/pypi-colorado-tools-snapshot/vortex-sdk/0.0.35/vortex_sdk-0.0.35-py3-none-any.whl
Prepared 1 package in 469ms
Installed 1 package in 3ms
 + vortex-sdk==0.0.35

Compatibility testing:

All existing cargo test -p uv-distribution-types tests pass

abramowi at marcs-mbp-3 in ~/Code/OpenSource/uv (env-vars-in-index-urls●)
$ cargo test -p uv-distribution-types
...
test result: ok. 19 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

No breaking changes to existing index configurations
Error messages are clear and actionable when environment variables are missing

Documentation:

Updated docs/concepts/indexes.md with environment variable expansion section and examples
Enhanced docs/guides/integration/alternative-indexes.md with env var examples for Azure, GCP, and AWS
Updated docs/reference/settings.md with comprehensive examples and feature description

Fixes: astral-shGH-5734

by running `cargo dev generate-options-reference`

msabramo · 2025-06-06T12:32:34Z

Windows test failed with what looks to be an intermittent failure downloading Python:

    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Snapshot Summary ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    Snapshot: python_install_preview-10
    Source: D:\uv:463
    ───────────────────────────────────────────────────────────────────────────────
    Expression: snapshot
    ───────────────────────────────────────────────────────────────────────────────
    -old snapshot
    +new results
    ────────────┬──────────────────────────────────────────────────────────────────
        0       │-success: true
        1       │-exit_code: 0
              0 │+success: false
              1 │+exit_code: 1
        2     2 │ ----- stdout -----
        3     3 │ 
        4     4 │ ----- stderr -----
        5       │-Installed 2 versions in [TIME]
        6       │- + cpython-3.12.6-[PLATFORM]
        7       │- + cpython-3.12.8-[PLATFORM] (python3.12)
              5 │+Installed Python 3.12.6 in [TIME]
              6 │+ + cpython-3.12.6-[PLATFORM] (python3.12)
              7 │+error: Failed to install cpython-3.12.8-[PLATFORM]
              8 │+  Caused by: Failed to download https://github.com/astral-sh/python-build-standalone/releases/download/20250115/cpython-3.12.8%2B20250115-x86_64-pc-windows-msvc-install_only_stripped.tar.gz
              9 │+  Caused by: HTTP status server error (500 Internal Server Error) for url (https://github.com/astral-sh/python-build-standalone/releases/download/20250115/cpython-3.12.8%2B20250115-x86_64-pc-windows-msvc-install_only_stripped.tar.gz)
    ────────────┴──────────────────────────────────────────────────────────────────
    Stopped on the first failure. Run `cargo insta test` to run all snapshots.
    test python_install::python_install_preview ... FAILED

to see if failing Windows GitHub action passes this time.

msabramo · 2025-06-06T12:58:03Z

Windows test failed with what looks to be an intermittent failure downloading Python:

    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Snapshot Summary ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
    Snapshot: python_install_preview-10
    Source: D:\uv:463
    ───────────────────────────────────────────────────────────────────────────────
    Expression: snapshot
    ───────────────────────────────────────────────────────────────────────────────
    -old snapshot
    +new results
    ────────────┬──────────────────────────────────────────────────────────────────
        0       │-success: true
        1       │-exit_code: 0
              0 │+success: false
              1 │+exit_code: 1
        2     2 │ ----- stdout -----
        3     3 │ 
        4     4 │ ----- stderr -----
        5       │-Installed 2 versions in [TIME]
        6       │- + cpython-3.12.6-[PLATFORM]
        7       │- + cpython-3.12.8-[PLATFORM] (python3.12)
              5 │+Installed Python 3.12.6 in [TIME]
              6 │+ + cpython-3.12.6-[PLATFORM] (python3.12)
              7 │+error: Failed to install cpython-3.12.8-[PLATFORM]
              8 │+  Caused by: Failed to download https://github.com/astral-sh/python-build-standalone/releases/download/20250115/cpython-3.12.8%2B20250115-x86_64-pc-windows-msvc-install_only_stripped.tar.gz
              9 │+  Caused by: HTTP status server error (500 Internal Server Error) for url (https://github.com/astral-sh/python-build-standalone/releases/download/20250115/cpython-3.12.8%2B20250115-x86_64-pc-windows-msvc-install_only_stripped.tar.gz)
    ────────────┴──────────────────────────────────────────────────────────────────
    Stopped on the first failure. Run `cargo insta test` to run all snapshots.
    test python_install::python_install_preview ... FAILED

Adding an empty commit to force a rerun fixed this temporary issue, though maybe it might be good to add retry logic for this?

Anyway, all tests passing now! 🎉

zanieb · 2025-06-06T13:47:38Z

Thanks for the pull request.

As a brief note, we do not have consensus on a design for this and are very unlikely to accept a pull request without reaching that consensus first. We may be able to use this proposal to help drive consensus forward, but I want to set the right expectations here.

msabramo · 2025-06-06T14:11:03Z

Thanks for the pull request.

As a brief note, we do not have consensus on a design for this and are very unlikely to accept a pull request without reaching that consensus first. We may be able to use this proposal to help drive consensus forward, but I want to set the right expectations here.

@zanieb

Yeah understood, I know this is kind of controversial. I was just composing a comment on the issue with a proposal about how to solve some of the concerns. That proposal is not implemented in this PR, but if by discussion we come to some solution that people are happy with, then I could add it to this PR.

msabramo temporarily deployed to release June 6, 2025 04:06 — with GitHub Actions Inactive

msabramo marked this pull request as draft June 6, 2025 05:11

msabramo force-pushed the env-vars-in-index-urls branch from e2f68b3 to 1dbd9ec Compare June 6, 2025 05:12

msabramo temporarily deployed to release June 6, 2025 05:12 — with GitHub Actions Inactive

msabramo marked this pull request as ready for review June 6, 2025 05:12

msabramo force-pushed the env-vars-in-index-urls branch from 1dbd9ec to 65ad2bd Compare June 6, 2025 05:14

msabramo temporarily deployed to release June 6, 2025 05:14 — with GitHub Actions Inactive

msabramo marked this pull request as draft June 6, 2025 05:16

msabramo force-pushed the env-vars-in-index-urls branch from 65ad2bd to acf7189 Compare June 6, 2025 05:18

msabramo temporarily deployed to release June 6, 2025 05:18 — with GitHub Actions Inactive

Expand environment variables in user-provided index URLs

c938a55

Fixes: astral-shGH-5734

msabramo force-pushed the env-vars-in-index-urls branch from acf7189 to c938a55 Compare June 6, 2025 05:50

msabramo temporarily deployed to release June 6, 2025 05:50 — with GitHub Actions Inactive

Update settings.md

775196a

by running `cargo dev generate-options-reference`

msabramo temporarily deployed to release June 6, 2025 11:54 — with GitHub Actions Inactive

rustfmt crates/uv-distribution-types/src/index.rs

3074194

msabramo temporarily deployed to release June 6, 2025 11:56 — with GitHub Actions Inactive

npx prettier --write --prose-wrap always --check "**/*.md"

2cabc5d

msabramo temporarily deployed to release June 6, 2025 12:17 — with GitHub Actions Inactive

Empty commit to retrigger CI

1386f72

to see if failing Windows GitHub action passes this time.

msabramo temporarily deployed to release June 6, 2025 12:33 — with GitHub Actions Inactive

msabramo marked this pull request as ready for review June 6, 2025 12:56

zanieb added the needs-decision Undecided if this should be done label Jun 6, 2025

msabramo mentioned this pull request Jun 6, 2025

Expand environment variables in user-provided index URLs #5734

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expand environment variables in user-provided index URLs#13879

Expand environment variables in user-provided index URLs#13879
msabramo wants to merge 5 commits intoastral-sh:mainfrom
msabramo:env-vars-in-index-urls

msabramo commented Jun 6, 2025 •

edited

Loading

Uh oh!

msabramo commented Jun 6, 2025

Uh oh!

msabramo commented Jun 6, 2025

Uh oh!

zanieb commented Jun 6, 2025

Uh oh!

msabramo commented Jun 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

msabramo commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Plan

Uh oh!

msabramo commented Jun 6, 2025

Uh oh!

msabramo commented Jun 6, 2025

Uh oh!

zanieb commented Jun 6, 2025

Uh oh!

msabramo commented Jun 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

msabramo commented Jun 6, 2025 •

edited

Loading