Skip to content

Workspaces and monorepo support (add sync --all-packages) #6935

@carderne

Description

@carderne

I've put a decent amount of effort trying to figure out a workable "monorepo" solution with pip-tools/Rye/etc and now uv. What I mean by a monorepo:

  1. 2+ packages with interdependencies.
  2. The ability to lock dependencies across packages (where not needed, split into multiple workspaces). More sophisticated multi-version handling would be great but out of scope.
  3. Multiple entrypoints. Packages are peers and there is no "root" package.
  4. Probably want to distribute the packages in a Dockerfile or similar.

I'm packaging a few thoughts into this issue as I think they're all related, but happy to split things out if any portions of this are more likely to be worked on than others.

Should uv support this?

I think yes. Pants/Bazel/etc are a big step up in complexity and lose a lot of nice UX. uv is shaping up as the defacto Python tool and I think this is a common pattern for medium-sized teams that are trying to move past multirepo but don't want more sophisticated tooling. If you (uv maintainers) are unconvinced (but convince-able), I'm happy to spend more time doing so!

Issues

1. Multiple packages with single lockfile

Unfortunately, uv v0.4.0 seems to be a step back for this. It's no longer possible to uv sync for the whole workspace (related #6874), and the root project being "virtual" is not really supported. The docs make it clear that uv workspaces aren't (currently) meant for this, but I think that's a mistake. Have separate uv packages isn't a great solution, as you lose the global version locks (which makes housekeeping 10x easier), so you have multiple venvs, multiple pyright/pytest installs/configs etc.

For clarity, I'm talking about the structure below. I think adding a tool.uv.virtual: bool flag (like Rye has) would be a great step. In that case the root is not a package and can't be built.

.
├── pyproject.toml                 # virtual
├── uv.lock
└── packages
    ├── myserver
    │   ├── pyproject.toml         # depends on mylib
    │   └── myserver
    │       └── __init__.py
    └── mylib
        ├── pyproject.toml
        └── mylib
            └── __init__.py

2. Distributing in Dockerfiles etc

This is I think orthogonal to the issue above. (And much less important, as it's possible to work around it with plugins.) Currently, there's no good way to get an efficient (cacheable) Docker build in a uv workspace. You'd like to do something like the Dockerfile below, but you can't (related #6867).

FROM python:3.12.5-slim-bookworm
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv

WORKDIR /app
COPY uv.lock pyproject.toml /app/

# NB: doesn't work as the server package isn't there!
RUN uv sync --locked --no-install-project --package=server

COPY packages /app/packages
RUN uv sync --locked --package=server
ENV PATH="/app/.venv/bin:$PATH"

If that gets resolved, there's another issue, but this is very likely to be outside the scope of uv. Just sharing it for context.

  • Either you have to copy the entire packages/ directory into every Dockerfile (regardless of what they actually need), forcing tons of unnecessary rebuilds.
  • OR you have custom COPY lines in each Dockerfile, which is a mess to maintain with more than a couple of packages, and has to be constantly updated to match the dependency graph.

My own solution has been to build wheels that include any dependencies so you can just do this:

# uv is nice enough to resolve transitive dependencies of server
uv export --format=requirements-txt --package=server > reqs.txt

Then in Dockerfile:

COPY reqs.txt reqs.txt
RUN uv pip install -r reqs.txt
# add --no-index to prevent internet access to ensure only the
# hash-locked versions in reqs.txt are downloaded
RUN uv pip install server.whl --no-deps --no-index

I've written a tiny Hatch plugin here that injects all the required workspace code into the wheel. This won't work for many use-cases (local dev hot reload) but is one way around the problem of COPYing the entire workspace into the Dockerfile. I don't think there's any solution that solves both together, and at least this way permits efficient Docker builds and simple Dockerfiles. (Note: since uv v0.4.0 the plugin seems to break uv's editable builds, haven't yet looked into why.)

Metadata

Metadata

Assignees

Labels

questionAsking for clarification or support

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions