Development environments

As a user I want an environment that contains the dev(build/host/run) dependencies of all packages in my workspace, so that I can connect my IDE, use the build tools, and "play" with my setup.

What does the workflow look like?

Assume you have the following two manifests
./pixi.toml

[dependencies]
python = "*"

# Proposed new table:
[dev]
package_a = { path = "src/package_a" } 

./src/package_a/pixi.toml

[package.build.backend]
name = "pixi-build-cmake"

[package.build.config]
compilers = ["c", "cxx"]

[package.build-dependencies]
openssl = "*"

pixi install creates an environment that will contain:

• Not package_a, this is up to the user to run a build of.
• python from dependencies
• c-compiler from compilers in package_a
• cxx-compiler from compilers in package_a
• openssl from build-dependencies in package_a
• cmake from a default in backend in package_a

Manifest definition

Add a new table dev

[dev]
foo = { path = "src/package-foo" }
python = "*" # Error as not a path dependency

# It should also work with a `feature` table.
[feature.test.dev]
foo = { path = "src/package-foo" }

This table can contain a normal style source dependency.

Questions

• Q: How do we deal with run-exports?
  A: Ignore them for now
• Q: Does the dev workflow install the package itself by running the build script?
  A: In the first iteration, we completely ignore the build scripts, this might be fixed with [package.tasks] in the future (needs a design)
• Q: How do I specify extra dependencies not part of the packaging steps.
  A: We are thinking of addingtest/dev-dependencies in the package table.
• Q: How to deal with recursive source dependencies. E.g. package a is a dev package and requires package b. What happens when b is a dev package, what happens if it is not.
  A: If a package is specified in dev section we just take its dev(host/build/run) dependencies. If a source dependencies is part of that, we recursively take their dependencies.
• Q: How are dependencies reevaluated? E.g. how do we know if the dependencies of a package changed.
  A: We add a special field to the lockfile for this dev dependency.
• Q: What is stored in the lock-file?
  A: Metadata input globs, and the resulting devopment dependencies.
• Q: Does the build backend return a set of dependencies for a particular output in "dev" mode or are they derived from the host/build of the output.
  A: Yes they will be derived from the package dependencies.
• Q: How do you run something in this environment, with this I mean (tasks) or something like cargo c or make?
  A: pixi run do-your-thing
• Q: How do we deal with solve-groups?
  A: We don't do anything special right now. We see that this might cause more issues, but we'll have to see that first.

Multiple packages conflict

A concern would be that build dependencies conflict. This is a natural issue that users could solve by modifying the dependencies. Or they use the current feature set of Pixi-build where you use the non-dev workflow where Pixi does create an isolated environment.

[dependencies]
# Isolated build, only install package and run-deps
python = { path = "../python"}

[dev]
# Non isolated only install build/host dependencies
my_lib = { path = "src/my_lib" }

Role of the build-backend

The build backend should understand that the user wants a development environment, possibly having better defaults available for the dependencies of even an extended list with e.g. test-dependencies or dev-dependencies.

Use cases

Pixi itself

For Pixi development we need an environment with rust cargo openssl and a separate test env with pytest python etc.

This proposal would allow us to use a package definition in Pixi and reusing the rust dependency in the default environment.

ROS workspaces

In a ROS workspace, you are often dealing with 10+ packages in one workspace. You want to combine all build dependencies into one environment and be able to call build on it with easy tooling.

This is the environment you connect the IDE to because you need to python interpreter.

Problem: you still want to avoid the use of colcon as pixi does have all the information to deal with building sequence and the ability to use them. (#4640). This is an item we have to solve in a next proposal for running [package.tasks] which the backend could prepare for you.

[workspace]
channels = ["conda-forge", "robostack-jazzy"]
platforms = ["linux-64", "osx-arm64"]

[dev]
ros-jazzy-nav-msgs = { path = "src/nav-msgs"}
ros-jazzy-nav = { path = "src/nav"}
ros-jazzy-cam-msgs = { path = "src/cam-msgs"}
ros-jazzy-cam = { path = "src/cam"}
ros-jazzy-ade = { path = "src/ade"}
ros-jazzy-std-msgs = { path = "../std_msgs"}

[dependencies]
ros-jazzy-plotjuggler = ">=12.3"
ros-jazzy-rviz = "~=1.2.3"
colcon-ros = "*"

[feature.test.dependencies]
pytest = "*"

[tasks]
build = "colcon build"

[feature.test.tasks]
test = "colcon test"

[environments]
test = ["test"]

Follow up: locked build/host/test environments

As a user, I want to use the same version of Python in my workspace as I want to use in my pixi build environments.

Since this proposal would add the build/host/test dependencies to an environment in your workspace, these items are locked in the lockfile.

I (Ruben) think it would be great if these dependencies could be fetched from the pixi.lock and be used in the build host and test environments that are created when running pixi build.

Questions:

• What does this do with variants?
• What does this do with cross-compile?
• Could this not give a solvable environment for the build and host?
• How to not use this and get a "latest/greatest" build.
• How to deal with overrides proposed in: R&D: Build/Host dependencies from source if they're from source in the workspace #4640
  This deals with the question, if a develop package is also a source dependency of a different source package that is not "dev" what do you do?

Follow up: build a package?

Users can use the normal tasks in this environment, so there would be an unlimited amount of flexibility.

Future improvement might add the reuse of pixi build.
Where pixi build currently creates an isolated build/host environment, we could reuse the command to allow the user to run it in the build environment.

So, pixi build in the workspace would build the package directly into the workspace environment, instead of in the isolated environment. The current pixi build command could be renamed to pixi package to disconnect these commands. Or we add a pixi dev command like maturin has.

This idea has to be extended in a different proposal, but this would be something along the lines of:

[dev]
self = { path = "." }

[package]
name = "self"

[package.tasks]
build = "cargo build" # Or a default that the backend could provide

pixi dev build

[workspace]
channels = "bla"
platforms = ["linux-64"]

[dependencies]
python = "*"
ruff = "*"
cargo-insta = "*"

[dev]
self = { path = "." }

[feature.develop.dev]
self = { path = "." }

[package]
name = "self"
version = "1.2.3"

[package.build.backend]
name = "pixi-build-cmake"

[package.build.config]
compilers = ["c", "cxx"]

[package.build-dependencies]
openssl = "*"

[package.dev-dependencies]
ruff = "*"
cargo-insta = "*"

pixi install
pixi run cargo
pixi run cargo-insta review
pixi run ruff

[package.tasks]
build = "cargo b"
lint = "cargo clippy"
test = "cargo t"

pixi run self build
# Or
pixi dev build

[package.build.backend]
name = "pixi-build-cmake"

[package.build.config]
compilers = ["c", "cxx"]

[package.build-dependencies]
openssl = "*"

[package.build.backend]
name = "pixi-build-python"

[package.build.config]
compilers = ["rust"]

[package.host-dependencies]
maturin = "*"
# Source dep!
package_b = { path = "../package_b" }

[package.run-dependencies]
numpy = "*"

[workspace]
channels = "cf"
platforms = ["linux-64"]

[dependencies]
python = "*"
ruff = "*"
cargo-insta = "*"

[dev]
package_a = { path = "package_a" }
package_b = { path = "package_b" }


[feature.develop.dev]
package_b = {}
[feature.dev.dependencies]
package_a = {}

[feature.prod.dependencies]
package_a = { path = "package_a" }
package_b = { path = "package_b" }

[environments]
default = { solve-group = "default"}
prod = { features = ["prod"], solve-group = "default"}

https://hackmd.io/@ECkB7Uj1RPmzh9FM3agRAw/SyOAZq8ael/edit