Convert Python examples to proper packages by abey79 · Pull Request #5966 · rerun-io/rerun

abey79 · 2024-04-12T15:46:11Z

What

This PR does the following:

Convert all Python examples to proper python packages, replacing the requirements.txt file with a pyproject.toml file.
Applies same conversion to tests/python/test_api.
Add WIP support for running example with pixi
Add a all_examples meta-project.
Fix dev-tools build-examples for the new example format.
Add a dev-tools build-examples install command to pip install the examples for the given channel (prerequisite to build-examples rrd).
Fix CI for the new example format.
Updates pixi to 0.19.1 in CI.

pixi support

This WIP pixi support allows running examples in the examples environment:

pixi run -e examples clock

Note that by default pixi builds the environment with a shim/empty/fake rerun package (from examples/python/_empty_rerun_sdk), which is a work-around to allow manual installation of rerun in the environment. As a result, at first (and whenever a rebuild is needed), this must be run:

pixi run -e examples py-build

`all_examples`

This is a meta-package that dynamically depends on all examples. The original hope was to be able to depend on it from pixi, avoiding the need to list example explicitly. However, this is current impossible for a number a reason:

doing so means that examples would not be installed as editable, causing issue with __file__ relative paths (Examples: download datasets into OS standard cache directory instead of relative to the python file #6054)
doing so we leak full path in the lock file
pixi currently has an issue where indirect dependency with git dependencies are broken

all_examples also has a very basic CLI tool that allows listing all examples in a format suitable for pixi: all_examples list.

Also, it should be relatively easy to cleanly reimplement the functionality of run_all.py, which is currently broken:

Replace run_all.py and nox with better alternatives #6057

As it stands, it allows for efficiently checking dependency conflicts, especially with uv:

cd /tmp
uv venv
source .venv/bin/activate
uv pip install -e path/to/rerun/examples/python/all_examples   # ok??

TODO

convert a few example to python package
initial version of a all_examples meta-package
fix re_build_example
RE-ENABLE LINK CHECKS!!!!
~~properly working pixi setup~~
~~python version matrix testing setup~~

Checklist

I have read and agree to Contributor Guide and the Code of Conduct
I've included a screenshot or gif (if applicable)
I have tested the web demo (if applicable):
- Using examples from latest main build: rerun.io/viewer
- Using full set of examples from nightly build: rerun.io/viewer
The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG
If applicable, add a new check to the release checklist!

To run all checks from main, comment on the PR with @rerun-bot full-check.

github-actions · 2024-04-12T15:46:45Z

Deployed docs

Commit	Link
`b73dc19`	https://landing-ea9s0unit-rerun.vercel.app/docs

jleibs · 2024-04-12T19:05:52Z

First big snag:
the pixi.lock file ends up with:

      - pypi: file:///Users/hhip/src/rerun/examples/python/blueprint_stocks
      - pypi: file:///Users/hhip/src/rerun/examples/python/clock
      - pypi: file:///Users/hhip/src/rerun/examples/python/redirect

which obviously isn't portable.

abey79 · 2024-04-12T19:16:11Z

Right. I specifically used full paths because something required them (was it uv?). I'll try to push a little more with relative path then.

jleibs · 2024-04-12T19:37:52Z

Removed those from the lock file and now I get an error:

thread 'main' panicked at src/install_pypi.rs:199:14:
could not convert path into uv dist: NotFound(Url { scheme: "file", cannot_be_a_base: false, username: "", password: None, host: None, port: None, path: "/home/jleibs/rerun/file:/home/jleibs/rerun/examples/python/blueprint_stocks", query: None, fragment: None })

abey79 · 2024-04-13T06:26:51Z

Yes, there seem to be an issue with pixi: prefix-dev/pixi#1186

abey79 · 2024-04-13T07:04:36Z

Some though I had over the night: instead of the dependency list to dynamically change based on the current python version, it's probably better to just stick the example's required-python to the dependency:

"click @ file://path/to/clock ; python_version <= '3.11'"

This would make the dependency list less dynamic (basically just on example existence and skip = true), and reduce the caching-related annoyances.

…g/excluding example from dependencies

# Conflicts: # pixi.lock

…nflict

# Conflicts: # examples/python/depth_guided_stable_diffusion/requirements.txt # pixi.lock

# Conflicts: # pixi.toml

teh-cmc · 2024-04-22T07:19:55Z

pixi support

This WIP pixi support allows running examples in the examples environment:
pixi run -e examples clock
Note that by default pixi builds the environment with a shim/empty/fake rerun package (from examples/python/_empty_rerun_sdk), which is a work-around to allow manual installation of rerun in the environment. As a result, at first (and whenever a rebuild is needed), this must be run:
pixi run -e examples py-build
all_examples

This is a meta-package that dynamically depends on all examples. The original hope was to be able to depend on it from pixi, avoiding the need to list example explicitly. However, this is current impossible for a number a reason:
* doing so means that examples would not be installed as editable, causing issue with `__file__` relative paths ([Examples: download datasets into OS standard cache directory instead of relative to the python file #6054](https://github.com/rerun-io/rerun/issues/6054))

* doing so we leak full path in the lock file

* pixi currently has an issue where indirect dependency with git dependencies are broken
all_examples also has a very basic CLI tool that allows listing all examples in a format suitable for pixi: all_examples list.

Also, it should be relatively easy to cleanly reimplement the functionality of run_all.py, which is currently broken:
* [Replace `run_all.py` and nox with better alternatives #6057](https://github.com/rerun-io/rerun/issues/6057)
As it stands, it allows for efficiently checking dependency conflicts, especially with uv:
cd /tmp
uv venv
source .venv/bin/activate
uv pip install -e path/to/rerun/examples/python/all_examples   # ok??

All of these explanations should appear in examples/python/README.md imo

teh-cmc · 2024-04-22T07:35:34Z

detect_and_track_objects doesn't work for me, is that known?

Failed to run `python3 -m detect_and_track_objects --save /tmp/snippets/detect_and_track_objects.rrd`:
stdout:

stderr:
DEV ENVIRONMENT DETECTED! Re-importing rerun from: /home/cmc/dev/rerun-io/rerun/rerun_py/rerun_sdk
/home/cmc/dev/rerun-io/rerun/.pixi/envs/default/lib/python3.11/site-packages/transformers/utils/hub.py:124: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
  warnings.warn(
[2024-04-22T07:33:00Z DEBUG re_log_types::data_table_batcher] creating new table batcher config=DataTableBatcherConfig { flush_tick: 1000000000s, flush_num_bytes: 131072, flush_num_rows: 18446744073709551615, max_commands_in_flight: None, max_tables_in_flight: None, hooks: BatcherHooks { on_insert: None, on_release: Some(ArrowChunkReleaseCallback("0x4f098020af0")) } }
[2024-04-22T07:33:00Z DEBUG re_sdk::recording_stream] setting recording info app_id=rerun_example_detect_and_track_objects rec_id=8fce6e69-bb27-432d-9020-e25dc4f7692b
[2024-04-22T07:33:00Z DEBUG re_log_encoding::file_sink] Saving file to "/tmp/snippets/detect_and_track_objects.rrd"…
[2024-04-22T07:33:00Z DEBUG re_sdk::recording_stream] setting recording info app_id=rerun_example_detect_and_track_objects rec_id=8fce6e69-bb27-432d-9020-e25dc4f7692b
/home/cmc/dev/rerun-io/rerun/.pixi/envs/default/lib/python3.11/site-packages/dataset/tracking_sequences/horses.mp4 already exists. No need to download
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/cmc/dev/rerun-io/rerun/.pixi/envs/default/lib/python3.11/site-packages/detect_and_track_objects.py", line 438, in <module>
    main()
  File "/home/cmc/dev/rerun-io/rerun/.pixi/envs/default/lib/python3.11/site-packages/detect_and_track_objects.py", line 432, in main
    track_objects(video_path, max_frame_count=args.max_frame)
  File "/home/cmc/dev/rerun-io/rerun/.pixi/envs/default/lib/python3.11/site-packages/detect_and_track_objects.py", line 330, in track_objects
    with open(COCO_CATEGORIES_PATH, encoding="utf8") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/home/cmc/dev/rerun-io/rerun/.pixi/envs/default/lib/python3.11/site-packages/panoptic_coco_categories.json'
[2024-04-22T07:33:00Z DEBUG rerun_bindings::python_bridge] Shutting down the Rerun SDK
[2024-04-22T07:33:00Z DEBUG re_sdk::recording_stream] setting recording info app_id=rerun_example_detect_and_track_objects rec_id=8fce6e69-bb27-432d-9020-e25dc4f7692b
[2024-04-22T07:33:00Z DEBUG re_log_encoding::file_sink] Log stream written to /tmp/snippets/detect_and_track_objects.rrd

abey79 · 2024-04-22T07:38:39Z

detect_and_track_objects doesn't work for me, is that known?

Probably #6054. You need to install in editable mode.

teh-cmc

All the pixi stuff just worked for me, I could install and save rrd files for all examples without any issue; that's pretty great.

I don't quite follow how it all works but I'm more concerned about the fact that it does.

I'm not sure the complexity added by the all_examples thing really is worth it though.

examples/python/blueprint/pyproject.toml

pixi.toml

abey79 · 2024-04-22T07:49:52Z

I'm not sure the complexity added by the all_examples thing really is worth it though.

I agree, to me it's 50/50 if this thing should/will survive. I do want to keep it around for a bit and see how things pan out with a bit of hindsight.

Also, there is this bit of TODO left where all_examples may or may not play a role:

Replace run_all.py and nox with better alternatives #6057

teh-cmc · 2024-04-22T07:57:26Z

detect_and_track_objects doesn't work for me, is that known?

Probably #6054. You need to install in editable mode.

Doesn't pixi run dev-tools build-examples install --channel nightly do that for me already? All others worked it seems

abey79 · 2024-04-22T08:01:33Z

detect_and_track_objects doesn't work for me, is that known?

Probably #6054. You need to install in editable mode.

Doesn't pixi run dev-tools build-examples install --channel nightly do that for me already? All others worked it seems

From your error log, clearly the example was not installed in editable mode. I checked and it's build-examples install's fault. That's an oversight on my part—fixing. Good catch!

… README.md

abey79 · 2024-04-22T08:43:06Z

All of these explanations should appear in examples/python/README.md imo

I added the pixi info, but skipped the all_examples one. Because #6054, running examples after pip install -e all_examples/. may be wonky/fail, so it's not best advertised too much. The speech about checking dependency conflict is present in all_examples/README.md though.

# Conflicts: # .github/workflows/contrib_checks.yml # .github/workflows/reusable_checks.yml # .github/workflows/reusable_checks_python.yml # .github/workflows/reusable_deploy_docs.yml # pixi.lock # pixi.toml

emilk · 2024-04-23T07:39:23Z

We have a lot of references to examples/python/…/main.py that this PR broke

### What While investigating recently broken links (because of #5966 (comment)) I decided to add a lint against linking to stuff on changing branches (e.g. `/blobs/main`). Unfortunately I cannot cover it all (see #6077 for more), but this is an OK start. ### Checklist * [x] I have read and agree to [Contributor Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and the [Code of Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md) * [x] I've included a screenshot or gif (if applicable) * [x] I have tested the web demo (if applicable): * Using examples from latest `main` build: [rerun.io/viewer](https://rerun.io/viewer/pr/6078?manifest_url=https://app.rerun.io/version/main/examples_manifest.json) * Using full set of examples from `nightly` build: [rerun.io/viewer](https://rerun.io/viewer/pr/6078?manifest_url=https://app.rerun.io/version/nightly/examples_manifest.json) * [x] The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG * [x] If applicable, add a new check to the [release checklist](https://github.com/rerun-io/rerun/blob/main/tests/python/release_checklist)! - [PR Build Summary](https://build.rerun.io/pr/6078) - [Recent benchmark results](https://build.rerun.io/graphs/crates.html) - [Wasm size tracking](https://build.rerun.io/graphs/sizes.html) To run all checks from `main`, comment on the PR with `@rerun-bot full-check`.

broken by #5966

abey79 added 3 commits April 12, 2024 17:40

Update a few examples to python package

fca1a81

Initial version of the all_examples meta-project

8d35203

[WIP] pixi support, doesn't work yet

b1d97cc

abey79 added examples Issues relating to the Rerun examples sdk-python Python logging API include in changelog labels Apr 12, 2024

abey79 added 15 commits April 15, 2024 10:24

Add python_version environment marker instead of dynamically includin…

5aaa970

…g/excluding example from dependencies

Merge branch 'main' into antoine/python-examples-2.0

ead9314

Add exclude-platform option to all_examples

557d194

[TEMP] remove all_examples from pixi.toml (this breaks the lockfile)

16f050a

Packagified all examples

6b02c31

Merge branch 'main' into antoine/python-examples-2.0

5558ced

# Conflicts: # pixi.lock

Some fixes

28d7fed

Temporarily disable depth_guided_stable_diff to address dependency co…

ae6ab12

…nflict

Merge branch 'main' into antoine/python-examples-2.0

c8a8a3d

Format toml

a2a6815

(wip) pixi.toml

357ec4f

Merge branch 'main' into antoine/python-examples-2.0

70768c7

# Conflicts: # examples/python/depth_guided_stable_diffusion/requirements.txt # pixi.lock

post merge fix

2644458

Merge branch 'main' into antoine/python-examples-2.0

1fd94bd

unrolled example list in pixi.toml

44412db

abey79 mentioned this pull request Apr 19, 2024

pixi run pip install is overridden by subsequent call to pixi prefix-dev/pixi#1233

Closed

2 tasks

abey79 added 2 commits April 19, 2024 09:26

Add fake rerun-sdk package to work around prefix-dev/pixi#1233

293a7fe

Merge branch 'main' into antoine/python-examples-2.0

517ca8f

# Conflicts: # pixi.toml

Update all_examples README.md

f7415d2

teh-cmc self-requested a review April 22, 2024 07:13

teh-cmc approved these changes Apr 22, 2024

View reviewed changes

examples/python/blueprint/pyproject.toml Outdated Show resolved Hide resolved

pixi.toml Show resolved Hide resolved

abey79 added 4 commits April 22, 2024 10:03

build-examples install now installs in editable mode

cb479fe

Remove dead config code and clarified all_examples configuration in…

0d3c485

… README.md

Merge branch 'main' into antoine/python-examples-2.0

4c8dbbd

Clarified env vars in pixi.toml

1089726

abey79 added 8 commits April 22, 2024 10:45

Added pixi stuff in exmaples/python/readme.md

5f762ea

Lint

cfe63cf

Moar lint

b108e58

Moar lint

7b8d4ba

🤦

ee908c2

Merge branch 'main' into antoine/python-examples-2.0

cc43d93

# Conflicts: # .github/workflows/contrib_checks.yml # .github/workflows/reusable_checks.yml # .github/workflows/reusable_checks_python.yml # .github/workflows/reusable_deploy_docs.yml # pixi.lock # pixi.toml

post merge

632ec00

lint

b73dc19

abey79 merged commit 5cafbe3 into main Apr 22, 2024

abey79 deleted the antoine/python-examples-2.0 branch April 22, 2024 09:30

emilk mentioned this pull request Apr 23, 2024

Add lint against linking into main branch on GitHub #6078

Merged

5 tasks

abey79 added a commit that referenced this pull request Apr 23, 2024

Fix links to Python examples

bdf386d

broken by #5966

abey79 mentioned this pull request Apr 23, 2024

Fix links to Python examples #6080

Merged

5 tasks

jprochazk mentioned this pull request Apr 25, 2024

Generate doc pages for blueprint views #6121

Merged

5 tasks

Conversation

abey79 commented Apr 12, 2024 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

pixi support

all_examples

TODO

Checklist

Uh oh!

github-actions bot commented Apr 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deployed docs

Uh oh!

jleibs commented Apr 12, 2024

Uh oh!

abey79 commented Apr 12, 2024

Uh oh!

jleibs commented Apr 12, 2024

Uh oh!

abey79 commented Apr 13, 2024

Uh oh!

abey79 commented Apr 13, 2024

Uh oh!

teh-cmc commented Apr 22, 2024

pixi support

all_examples

Uh oh!

teh-cmc commented Apr 22, 2024

Uh oh!

abey79 commented Apr 22, 2024

Uh oh!

teh-cmc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

abey79 commented Apr 22, 2024

Uh oh!

teh-cmc commented Apr 22, 2024

Uh oh!

abey79 commented Apr 22, 2024

Uh oh!

abey79 commented Apr 22, 2024

Uh oh!

emilk commented Apr 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

abey79 commented Apr 12, 2024 •

edited by github-actions bot

Loading

`all_examples`

github-actions bot commented Apr 12, 2024 •

edited

Loading

`all_examples`

emilk commented Apr 23, 2024 •

edited

Loading