Managing dependencies for our growing list of example is increasingly painful, eg. #5859

One way about it would be to pin everything in the requirements.txt. This is some drawbacks:

• Updating pins is a painful, manual process, especially to handle incompatibilities between individual dependencies...
• ...yet we must do it on a regular basis, if only to support newer version of Python every year.
• Our approach of "merging" all of our examples' requirements.txt files makes the dependency compatibility problem even worse.
• Even with the perfect pins, this still doesn't give us a lock file, so things can still break via updates to transitive dependencies.

When/how do/should examples run?

• Developer working on a single example
• Developer running all examples for testing purposes
• User wanting to try an example
• CI building .rrds for in-app examples
• CI testing our examples against all supported Python version

Some things to consider

Who do we optimize for? Use developers or potential users that may want to run an example?

High-level wants:

• Ability to run example with arbitrary (supported) Python version and Rerun either built from source, installed with local wheels, or installed with PyPI wheels
• All deps cached on CI
• Lockfile for all python deps (preferably one, or one per example)
• Very easy for us developers to use and maintain
  • I don't have to have different shells for running examples and pixi scripts
• Fast (installing deps, locally and on CI)

Less important wants:

• Easy for users to run and contribute examples
• Pythonic

Support conflicting dependencies per example?

• If yes
  • Per example dependency definition
  • Per example lock file
  • Per example isolated environment
  • Will probably be slower to install all deps
• If no (strongly leaning towards this)
  • Requires us to make the examples use the same version of each dependency (like we do for Rust)
  • Move complex examples outside of the repository (Move complex examples to own repositories #5661)
  • Probably a lot less work for us?
  • Need to fix existing examples first

Splitting examples python from rest of dev process?

I think it makes sense to treat our scripts/*.py helpers differently from our examples/python. That is, they can use different tooling if it makes sense.

Other

• Easy for external users to run and write examples

  • "Pythonic" (aka. compatible with pip, PEP 517, etc.) - or good docs
  • Probably needs a per-example requirements/pyproject file

• Handle well at least the following two configuration:

  • dev environment where rerun-sdk should be installed from the workplace
  • user checking out example and running against a "regular" install of rerun (e.g. with pipx, etc.)

• Plays nice with pixi (I can call pixi commands and python stuff from the same shell without changing venv)

• pixi run all-examples should work (maybe re_build_examples handles it?)