Use uv as hatch installer for faster test environment setup#23497
Conversation
- ddev hatch plugin: set installer = 'uv' on every collected env (default, test, e2e, benchmark, latest, lint) and switch _dd-install-packages scripts to 'uv pip install'. - Bump min hatch dependency to >=1.13 (first version with the uv installer). - CI: add astral-sh/setup-uv and migrate the shared dependency cache from ~/.cache/pip to ~/.cache/uv (plus pip path for the Py2 branch). Workflows updated: cache-shared-deps, test-target, test-fips-e2e, docs, build-ddev.
Modern hatch (1.10+, where the uv installer lives) requires virtualenv>=20.26.1, but datadog_checks_dev[cli] was capping it at <20.22.0 — a constraint added in 2023 to keep Py2 testing alive. Core no longer tests Py2, and the [cli] extras already pull tools that don't support Py2 anyway, so the upper bound was dead weight. Also adds changelog entries for both packages.
uv-managed venvs don't seed pip the way virtualenv does. This breaks two classes of in-env consumers: - mypy --install-types in lint envs (auto-installs missing type stubs) - python -m pip subprocess calls in tests / hatch.toml pre/post-install commands Fixes: - Add pip to lint env deps (for mypy) and to test/e2e env deps (defensively, for any test that subprocesses pip). - Migrate the four hatch.toml files that explicitly used python -m pip (ddev, datadog_checks_dev, root docs env, kafka_consumer) to uv pip.
|
✨ Fix all issues with BitsAI or with Cursor
|
…elper uv builds this legacy setup.py-only package via setuptools.build_meta:__legacy__, which writes the name verbatim into METADATA. With the underscored form 'datadog_checks_tests_helper', get_datadog_wheels() in datadog_checks_base (filters by 'datadog-' prefix) didn't pick it up, breaking test_get_datadog_wheels. Distribution names should use hyphens per PEP 503/625 anyway.
Codecov Report❌ Patch coverage is Additional details and impacted files🚀 New features to boost your workflow:
|
uv defaults to hardlink mode on Windows. When backports.zstd._zstd.pyd (transitive of hatch>=1.13) is loaded by Python during the test run, Windows refuses to delete the venv's hardlinked copy while the cache copy is still pinned. This breaks hatch env remove invoked by ddev test --compat / --recreate. See astral-sh/uv#7918
Regular Windows test jobs do not invoke 'hatch env remove' at the end, so they keep uv's hardlink speed-up. Only minimum-base-package jobs (which run with --compat) need the copy mode to allow venv cleanup.
The application_name field in HANA audit logs reflects the client process name (argv[0]). uv-created venvs launch pytest as 'python3', virtualenv-created venvs launch it as 'python'. Both are valid; the integration just forwards whatever HANA reports. Use ANY instead of asserting on the runner's binary name.
uv-created venvs (which is what hatch>=1.13 uses with installer=uv) launch pytest as python3, so HANA records the audit-log application_name as python3 rather than python. Update the test to match the new stable layout.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ee47f0f0c3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| def finalize_config(self, config): | ||
| for env_name, env_config in config.items(): | ||
| env_config.setdefault('installer', 'uv') |
There was a problem hiding this comment.
Keep the pip installer for Python 2 envs
The reusable test workflow still exposes the test-py2 path for non-core/community repos, but this unconditional default makes every collected Hatch env use uv. Hatch's installer = "uv" uses uv for environment creation and package installation, and uv's Python support policy excludes Python 2, so any integration matrix that still has a python = "2.7" env will fail before tests run. Gate this on the resolved env Python version or leave the installer unset for Python 2 environments.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
integrations-extras: no python = "2.x" entries in any hatch.toml or pyproject.toml matrix. Not affected.
marketplace: three integrations still declare py2.7 in their matrix — avmconsulting_workday,
mulesoft_anypoint,
oracle_timesten — all with:
[[envs.default.matrix]]
python = ["2.7", "3.11"]None of them actually build a py2 env in CI:
-
marketplace/.github/workflows/test-all.ymldefaultstest-py2: falseandtest-py3: true. -
test-target.yml'ssetup-test-env.shtranslates that into
SKIP_ENV_NAME=py2.*, so hatch filters the py2 envs out before the installer is invoked:if [[ "${INPUT_TEST_PY2:-}" == 'true' && "${INPUT_TEST_PY3:-}" != 'true' ]]; then SKIP_ENV_NAME="py3.*" elif [[ "${INPUT_TEST_PY2:-}" != 'true' && "${INPUT_TEST_PY3:-}" == 'true' ]]; then SKIP_ENV_NAME="py2.*" else SKIP_ENV_NAME="" fi
-
Verified on the 2026-04-29 nightly minimum-base-package run for
oracle_timesten(job 73524721025): resolved
test-py2: false, only py3.11 envs were built.
The failure mode flagged (uv refusing py2 because of its Python support policy) is real but unreachable from any current caller. It would
require explicitly invoking the workflow with test-py2: true, which is not exercised anywhere today.
Decision: not gating the installer on env Python version. Carrying that conditional in the plugin to cover three already-dead matrix entries puts the workaround in the wrong layer. The
intended follow-up is to drop "2.7" from those three marketplace hatch.toml files and remove the test-py2 plumbing from the reusable workflows once we confirm no external caller relies on
it.
Pin to specific versions for deterministic builds and stable cache reuse. virtualenv stays on the 20.x line (latest 20.39.1) due to hatch issue #2193.
Review from iliakur is dismissed. Related teams and files:
- agent-integrations
- .github/workflows/build-ddev.yml
|
The following files, which will be shipped with the agent, were modified in this PR and You can ignore this if you are sure the changes in this PR do not require QA. Otherwise, List of modified files that will be shipped with the agent |
hatch 1.16.5 declares virtualenv>=21 as a hard requirement (the 2026-02-25 virtualenv 21.0.0 release moved interpreter discovery to the python-discovery package, fixed in hatch by pypa/hatch#2196). The earlier 20.39.1 pin would have failed dependency resolution.
Review from NouemanKHAL is dismissed. Related teams and files:
- agent-integrations
- .github/workflows/build-ddev.yml
Validation ReportAll 20 validations passed. Show details
|
What does this PR do?
Switches the ddev hatch plugin to use uv as the installer for every hatch environment it collects (default, test, e2e, benchmark, latest, lint), and migrates the CI shared-dependency cache from pip's wheel cache to uv's wheel cache.
Motivation
ddev testis slow because pip spends a lot of time on metadata resolution even with wheels cached. Swapping pip for uv as the hatch installer makes environment setup substantially faster, both locally and in CI. uv was already partially used in the repo (resolve-build-deps.yaml,.gitlab/tagger/Dockerfile), so this rounds it out for the test path.Core changes
ddev/src/ddev/plugin/external/hatch/environment_collector.pypip_install_command→uv_install_command(returnsuv pip install)finalize_configsetsinstaller = 'uv'on every collected env (setdefault, so individual integrations can override back to pip if they ever need to)get_initial_configsets the same on the lint envpipadded to lint env and test/e2e env deps (uv-managed venvs don't seed pip; mypy--install-typesand a couple of in-env scripts still need it)ddev/pyproject.toml: bumphatch>=1.8.1→hatch>=1.13.0(first hatch version with the uv installer)datadog_checks_dev/pyproject.toml: bumpvirtualenv<20.22.0→virtualenv>=20.26.1in the[cli]extras. The old upper bound was added in 2023 to keep Py2 testing alive; core no longer tests Py2 and the cli extras already pull Py3-only tools, so the cap was dead weight blocking modern hatch.hatch.tomlfiles migrated frompython -m pip installtouv pip installin pre/post-install commands:ddev/hatch.toml,datadog_checks_dev/hatch.toml, roothatch.toml(docs env),kafka_consumer/hatch.toml.astral-sh/setup-uvand cache~/.cache/uv:.github/workflows/cache-shared-deps.yml.github/workflows/test-target.yml.github/workflows/test-fips-e2e.yml.github/workflows/docs.yml.github/workflows/build-ddev.ymlv01-python-...→v02-uv-...so the new cache populates cleanly without colliding with the old pip cache.No per-integration
hatch.tomlfiles needed touching beyond those four — the plugin injectsinstaller = 'uv'into all envs collected via[env.collectors.datadog-checks].Cache layout: why both uv AND pip paths are cached
The
cache-shared-deps,test-target,test-fips-e2eworkflows all cache two directories under a single key:Each line is a per-platform path (Linux / macOS / Windows) for one of the two wheel caches. Both directories are saved and restored together under one cache entry.
Why both:
~/.cache/uv) is what hatch envs and the system-leveluv pip install --systemwrite to. This is the cache that drives the speedup.~/.cache/pip) is still used by the Python 2.7 branch incache-shared-deps.yml(uv requires Py3.8+, so the Py2 step has to stay on pip). Without caching this path, the Py2 install step would re-download every wheel each run.Caching both under one key with one
actions/cachestep keeps the workflow simple — single restore, single save, both ecosystems covered.Snowflakes handled along the way
Switching the installer surfaced several second-order issues unrelated to uv itself but exposed by it. Documenting them so future debugging starts with context.
1. Windows venv cleanup blocked by uv hardlinks (
_zstd.pydlock)Symptom: Windows minimum-base-package jobs failed during post-test cleanup with
PermissionError: [WinError 5] Access is deniedonbackports/zstd/_zstd.cp313-win_amd64.pyd.Cause: uv defaults to
hardlinkmode on Windows. Whenhatch>=1.13(whose transitivebackports-zstdships a.pyd) is installed into a uv-managed venv, the.pydis a hardlink to the uv cache.hatch env remove(invoked at the end ofddev test --compat, used by the minimum-base-package matrix) then can'tshutil.rmtreethe venv because the loaded extension keeps the file open, and Windows refuses to delete any hardlink to an open file. Whether this manifests on a given run depends on Defender activity / shell / Windows build — see the astral-sh/uv#7918 discussion.Fix: Set
UV_LINK_MODE=copyonly on Windows minimum-base-package jobs in.github/workflows/test-target.yml. Regular Windows test jobs never runhatch env remove, so they keep the hardlink fast path. Refs: uv link-mode setting.2.
datadog-checks-tests-helperdistribution nameSymptom:
tests/cli/release/test_get_datadog_wheels.py::test_get_datadog_wheelsfailed in theddevtest job with the helper package now reported asdatadog_checks_tests_helper(underscores) instead ofdatadog-checks-tests-helper(hyphens).Cause:
datadog_checks_tests_helper/setup.pyhadname='datadog_checks_tests_helper'. The legacypython setup.py bdist_wheelinvocation that older pip+setuptools defaulted to silently normalized this to hyphens. Modern pip+setuptools (and uv) use the PEP 517setuptools.build_meta:__legacy__backend, which writes the literalname=value verbatim intoMETADATA. Thevirtualenv<20.22.0bump alone (which brings modern pip+setuptools) would have broken this test even without the uv switch.Fix: Change
name='datadog_checks_tests_helper'→name='datadog-checks-tests-helper'indatadog_checks_tests_helper/setup.py. The package is now distributed under the same canonical hyphenated name on PyPI conventions.3. SAP HANA
application_namein audit logsSymptom:
sap_hana/tests/test_integration.py::test_checkfailed with the audit log carryingapplication_name='python3'while the test asserted'python'.Cause: The
application_namefield that the SAP HANA server records is the connecting client's process name (argv[0]//proc/self/common Linux). Withinstaller = "uv"set on hatch envs, hatch ≥1.13 creates the venv viauv venv, whose canonical interpreter symlink ispython3; the previousvirtualenv-created venvs usedpython. So pytest is now launched aspython3and HANA recordspython3.Fix: Update the three assertions in
sap_hana/tests/test_integration.pyfrom'application_name': 'python'to'application_name': 'python3'to reflect the new (and now stable) venv layout.Review checklist (to be filled by reviewers)
qa/skip-qalabel if the PR doesn't need to be tested during QA.backport/<branch-name>label to the PR and it will automatically open a backport PR once this one is merged