Add `pixi` to run codegen in a portable way by jleibs · Pull Request #3707 · rerun-io/rerun

jleibs · 2023-10-05T20:52:10Z

What

Part of Use pixi to replace our bash scripts and many of the Dockerfile layers #3717

Add a proof-of-concept pixi.toml file for the project.

This is like Just but also takes care of dependency management for us. No need to run setup_dev.sh or pip install -r scripts/requirements-dev.txt.

This even manages the installation of arrow-cpp for us.

Just:

cargo install pixi

Then have fun:

pixi run codegen
pixi run py-test
pixi run cpp-test

Verified this works in a clean Ubuntu-20.04 docker container.

Checklist

I have read and agree to Contributor Guide and the Code of Conduct
I've included a screenshot or gif (if applicable)
I have tested demo.rerun.io (if applicable)

emilk · 2023-10-06T06:22:21Z

abey79 · 2023-10-06T07:03:50Z

This looks great.

I have two concerns though:

yet another place were we pin deps version (in addition to pyproject.toml, requirements*.txt, CI yamls, etc.)
further increase the distance between the dev (pixi/conda) vs. (main?) production workflow (pip/pypi)

It feels that is a great tool for end-user/"binary" projects, but doesn't really help with the complexity of "library" projects which are by definition more exposed to the world's complexity.

jleibs · 2023-10-06T11:28:59Z

yet another place were we pin deps version (in addition to pyproject.toml, requirements*.txt, CI yamls, etc.)

pyproject.toml needs to stay since it specifies the actual library deps

My proposal though is that all of the other rerun_py/requirements-*.txt all of the package listings in our Docker containers and CI scripts would just use the pixi environment. Which I think makes this a net win on places to capture things.

pixi.toml

emilk · 2023-10-09T10:48:31Z

I've converted this into a draft until it is a bit more than just a proof-of-concept

emilk · 2023-10-13T13:03:57Z

Changes

For now we will stick to using just as our main task runner, but we'll be porting more and more tasks to pixi, and have just forward these calls. For instance, just codegen now calls pixi run codegen.

To that end I have deprecated cargo codegen etc, and crated just version of them instead.

The immediate benefit of this PR is that just codegen now produces predictable results, because of it using fixed clang-format and black versions. This is a huge benefit for me personally when doing C++ dev, so I think this minimal proof-of-concept is already useful.

emilk · 2023-10-13T13:26:57Z

I've also added a bunch of TODO:s to #3717

Wumpf

works like a charm on my mac for the most part (well except some of the cpp build). Exited to try it out on Windows later today!

pixi.toml

Wumpf · 2023-10-13T13:22:37Z

pixi.toml

+platforms = ["linux-64", "osx-arm64", "osx-64", "win-64"]
+readme = "README.md"
+repository = "https://github.com/rerun-io/rerun"
+version = "0.1.0"                                                              # TODO(emilk): sync version with `Cargo.toml` with help from `crates.py`


todo is fine, but we might as well just start with 0.10.0-alpha

I think this is more clearly wrong. 0.10.0-alpha will just bitrot quickly. We're not planning on publishing rerun on pixi soon.

Wumpf · 2023-10-13T13:24:34Z

pixi.toml

+  "py-build",
+] }
+
+cpp-prepare = "cmake -G Ninja -B build -S . -DCMAKE_BUILD_TYPE=Debug"


note to self: Check if this still works fine on windows (since pixi should download ninja for windows it might as well work fine, opening the door to more unified build experience than what I had in mind! \o/)

There's this in CMakeLists.txt right now but I think we still need it for the time being on CI

# Use makefiles on linux, otherwise it might use Ninja which isn't installed by default. if(NOT DEFINED CMAKE_GENERATOR AND UNIX) set(CMAKE_GENERATOR "Unix Makefiles") endif()

is to be removed!

Wumpf · 2023-10-13T13:31:34Z

pixi.toml

+] }
+
+cpp-prepare = "cmake -G Ninja -B build -S . -DCMAKE_BUILD_TYPE=Debug"
+cpp-build = { cmd = "cmake --build build --config Debug --target rerun_sdk_tests", depends_on = [


this does not yet work on my Mac. Gives me a ton of warnings from Arrow and eventually fails building. Needs some more love. But can wait, after all we haven't put this in the just file yet

Wumpf · 2023-10-13T13:33:38Z

forgot to try before. pixi run py-build fails for me:

pixi run py-build
💥 maturin failed
  Caused by: Both VIRTUAL_ENV and CONDA_PREFIX are set. Please unset one of them

it doesn't like my venv apparently.

Again, not critical for the first step since this is now all about just getting codegen pixi'fied for everyone, everything else is fine if it works just on one machine

emilk · 2023-10-13T13:37:57Z

You should EITHER use venv or pixi. So run pixi run py-build from outside of a venv, or run pixi shell and run it from within the pixi environment

Wumpf · 2023-10-13T13:42:49Z

makes sense! With this in mind pixi run py-build works like a charm! <3

…ting differences

Wumpf · 2023-10-15T18:49:53Z

just codegen works on windows!

... but I had to push a bunch of fixes to make codegen actually do the same on windows

This sub-module was introduced likely by mistake with #3707

### What This sub-module was introduced likely by mistake with #3707. ### Checklist * [x] I have read and agree to [Contributor Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and the [Code of Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md) * [x] I've included a screenshot or gif (if applicable) * [x] I have tested the web demo (if applicable): * Full build: [app.rerun.io](https://app.rerun.io/pr/4432/index.html) * Partial build: [app.rerun.io](https://app.rerun.io/pr/4432/index.html?manifest_url=https://app.rerun.io/version/nightly/examples_manifest.json) - Useful for quick testing when changes do not affect examples in any way * [x] The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG - [PR Build Summary](https://build.rerun.io/pr/4432) - [Docs preview](https://rerun.io/preview/0800d294f0b7edbe91016137bbcde3b68ee49c43/docs)  - [Examples preview](https://rerun.io/preview/0800d294f0b7edbe91016137bbcde3b68ee49c43/examples)  - [Recent benchmark results](https://build.rerun.io/graphs/crates.html) - [Wasm size tracking](https://build.rerun.io/graphs/sizes.html)

Proof-of-concept: use pixi to manage build deps

be9448d

jleibs marked this pull request as ready for review October 5, 2023 20:52

jleibs added 🧑‍💻 dev experience developer experience (excluding CI) exclude from changelog PRs with this won't show up in CHANGELOG.md labels Oct 5, 2023

jleibs added 2 commits October 6, 2023 00:27

taplo

2b7e359

Pixi tasks for building and testing cpp

00ad344

jleibs mentioned this pull request Oct 5, 2023

Use pixi to replace our bash scripts and many of the Dockerfile layers #3717

Closed

9 tasks

jleibs added 2 commits October 5, 2023 19:43

Add more platforms

4f6be13

pixi.lock is another big file

52b04aa

jleibs added 2 commits October 6, 2023 13:37

Update lock

e35bcc2

Use the conda C compilers

1b98b50

emilk reviewed Oct 9, 2023

View reviewed changes

pixi.toml Outdated Show resolved Hide resolved

pixi.toml Outdated Show resolved Hide resolved

pixi.toml Outdated Show resolved Hide resolved

pixi.toml Show resolved Hide resolved

pixi.toml Show resolved Hide resolved

emilk marked this pull request as draft October 9, 2023 10:48

jleibs mentioned this pull request Oct 9, 2023

Investigate pixi package manager for installing build-dependencies #3336

Closed

emilk added 7 commits October 13, 2023 11:42

Merge branch 'main' into jleibs/pixi

5aae922

Add note that we are phasing out cargo aliases

4e0c85b

Add just rerun, just codegen, etc etc

142a807

Explain pixi codegen

21297bc

Install pixi in setup_dev.sh

28ff0c2

Cleanup

ef604e0

More cleanup

b02a30c

emilk marked this pull request as ready for review October 13, 2023 13:05

emilk added 2 commits October 13, 2023 15:08

Merge branch 'main' into jleibs/pixi

37e92dc

Mark .lock-files as linguist-generated=true

8b465c4

Wumpf self-requested a review October 13, 2023 13:11

Wumpf approved these changes Oct 13, 2023

View reviewed changes

Fix authors

b8e81ca

Wumpf added 2 commits October 15, 2023 20:49

Fix codegen producing different results on Windows due to path format…

0057290

…ting differences

use powershell for just file execution

983354d

emilk added 3 commits October 16, 2023 08:55

Add just cpp-clean, cpp-build, cpp-test

53d0b63

Call it format_path, use it more, fix clippy problems

0b4bafe

Enable Windows codegen in build.rs files

071e0d4

emilk changed the title ~~Proof-of-concept: use pixi to manage build deps~~ Add pixi to run codegen in a portable way Oct 16, 2023

emilk merged commit ca078fa into main Oct 16, 2023

emilk deleted the jleibs/pixi branch October 16, 2023 07:26

abey79 added a commit that referenced this pull request Dec 5, 2023

Remove spurious sub-module

0800d29

This sub-module was introduced likely by mistake with #3707

abey79 mentioned this pull request Dec 5, 2023

Remove spurious src/segment-anything submodule #4432

Merged

4 tasks

Conversation

jleibs commented Oct 5, 2023 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Checklist

Uh oh!

emilk commented Oct 6, 2023

Uh oh!

abey79 commented Oct 6, 2023

Uh oh!

jleibs commented Oct 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

emilk commented Oct 9, 2023

Uh oh!

emilk commented Oct 13, 2023

Changes

Uh oh!

emilk commented Oct 13, 2023

Uh oh!

Wumpf left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Wumpf Oct 13, 2023

Choose a reason for hiding this comment

Uh oh!

emilk Oct 13, 2023

Choose a reason for hiding this comment

Uh oh!

Wumpf Oct 13, 2023

Choose a reason for hiding this comment

Uh oh!

Wumpf Oct 13, 2023

Choose a reason for hiding this comment

Uh oh!

Wumpf Oct 13, 2023

Choose a reason for hiding this comment

Uh oh!

Wumpf commented Oct 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

emilk commented Oct 13, 2023

Uh oh!

Wumpf commented Oct 13, 2023

Uh oh!

Wumpf commented Oct 15, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jleibs commented Oct 5, 2023 •

edited by github-actions bot

Loading

jleibs commented Oct 6, 2023 •

edited

Loading

Wumpf left a comment •

edited

Loading

Wumpf commented Oct 13, 2023 •

edited

Loading