[MLflow Demo] Base implementation for demo framework by BenWilson2 · Pull Request #19994 · mlflow/mlflow

BenWilson2 · 2026-01-14T21:55:44Z

🥞 Stacked PR

Use this link to review incremental changes.

stack/demo/scaffold [Files changed]
- stack/demo/traces [Files changed]
  - stack/demo/eval [Files changed]
    - stack/demo/prompts [Files changed]
      - stack/demo/cli [Files changed]
        
        stack/demo/home [Files changed]
        
        stack/demo/docs [Files changed]
        
        stack/demo/scorers [Files changed]

Related Issues/PRs

#xxx

What changes are proposed in this pull request?

Adds the scaffolding framework for MLflow in-product demos.
Going with a template-based ABC approach here to make additions, modifications, and updates / fixes a bit more straightforward for maintenance and extension of these demos.
CI configuration with the first demo data generation (for traces) is added in #19995

How is this PR tested?

Existing unit/integration tests
New unit/integration tests
Manual tests

Does this PR require documentation update?

Release Notes

Is this a user-facing change?

No. You can skip the rest of this section.
Yes. Give a description of this change to be included in the release notes for MLflow users.

What component(s), interfaces, languages, and integrations does this PR affect?

Components

How should the PR be classified in the release notes? Choose one:

rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
rn/feature - A new user-facing feature worth mentioning in the release notes
rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
rn/documentation - A user-facing documentation change worth mentioning in the release notes

Should this PR be included in the next patch release?

Yes should be selected for bug fixes, documentation updates, and other small changes. No should be selected for new features and larger changes. If you're unsure about the release classification of this PR, leave this unchecked to let the maintainers decide.

What is a minor/patch release?

Minor release: a release that increments the second part of the version number (e.g., 1.2.0 -> 1.3.0).
Bug fixes, doc updates and new features usually go into minor releases.
Patch release: a release that increments the third part of the version number (e.g., 1.2.0 -> 1.2.1).
Bug fixes and doc updates usually go into patch releases.

Yes (this PR will be cherry-picked and included in the next patch release)
No (this PR will be included in the next minor release)

Copilot

Pull request overview

This PR introduces a base implementation for a demo framework that allows generating demo data for MLflow features. The framework provides a registry pattern for demo generators with versioning support to automatically regenerate demo data when the schema changes.

Changes:

Adds base classes (BaseDemoGenerator, DemoResult) for implementing demo data generators
Implements a registry pattern (DemoRegistry) for managing multiple demo generators
Includes version tracking to handle demo data migration on updates

Reviewed changes

Copilot reviewed 8 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
mlflow/demo/base.py	Core framework with abstract base class for generators, dataclass for results, and version management
mlflow/demo/registry.py	Registry implementation for discovering and managing demo generators
mlflow/demo/init.py	Public API with `generate_all_demos()` function
mlflow/demo/README.md	Documentation on design principles, creating generators, and versioning
mlflow/demo/generators/init.py	Empty module for future generator implementations
tests/demo/conftest.py	Test fixtures with stub generator implementations
tests/demo/test_base.py	Tests for base class validation, versioning, and data existence checks
tests/demo/test_registry.py	Tests for registry operations (register, get, list, contains)
tests/demo/test_generate.py	Tests for the main generation flow with version management
tests/demo/init.py	Empty test module initialization

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

mlflow/demo/base.py

tests/demo/conftest.py

tests/demo/test_base.py

github-actions · 2026-01-14T22:03:59Z

Documentation preview for 6ced72e is available at:

https://pr-19994--mlflow-docs-preview.netlify.app/docs/latest/

More info

Ignore this comment if this PR does not change the documentation.
The preview is updated when a new commit is pushed to this PR.
This comment was created by this workflow run.
The documentation was built by this workflow run.

harupy · 2026-01-15T04:08:55Z

mlflow/demo/README.md

+
+## Design Principles
+
+1. **Auto-generated on startup** - Demo data is created when `mlflow server` starts, requiring no user action.


I think users don't want this in prod. We could provide an option to disable demo data generation but that's inconvenient since they have to set it.

+1, I thought we would introduce a UI or cli hook like mlflow demo generate instead of generating a demo data by default

The worst scenario is the data generation step has a bug and block users.

Yep, agreed. I'm going to update that internal README with the 2 entry point paths (cli / UI-based ajax call) as well as providing information about why, for the existing users who generate demo data in their tracking server and then upgrade to a newer version of the tracking server why we'll want to have versioning available to prevent potentially broken demo experiences.

TomeHirata · 2026-01-15T07:48:32Z

mlflow/demo/base.py

+        navigation_url: URL path to navigate to view the demo data in the UI.
+    """
+
+    feature: str


Can we define an enum for the feature field?

good call on keeping this cleaner

TomeHirata · 2026-01-15T07:49:00Z

mlflow/demo/base.py

+    """
+
+    feature: str
+    entities_created: list[str]


Do we want to return an identifier instead of the actual entity? Then can we rename this field?

yep! Changed to entity_ids to be more accurate.

harupy · 2026-01-15T10:39:04Z

mlflow/demo/README.md

+from mlflow.demo.base import DEMO_EXPERIMENT_NAME, DEMO_PROMPT_PREFIX
+```
+
+## Versioning


Do we really need this? Can you explain a scenario where this is useful?

Added some explanation for why we will likely really want this functionality within the README.

harupy · 2026-01-16T04:25:28Z

mlflow/demo/README.md

+### When to Bump Version
+
+Bump the version when making changes to demo data that require regeneration:
+
+- Changing the structure of generated traces/spans
+- Adding new required fields to assessments or evaluations
+- Modifying prompt templates
+- Any change that makes old demo data incompatible with the current UI


Suppose we have a typo in the demo data. Do we need to bump the version to fix the typo?

It appears we always to need to bump the version to refresh the demo data in a user machine

I think we should to ensure that if the demo data is already on a running tracking server, we can hot reload only for version mismatches. The reason I think this is important is because of the latency involved with generating trace linkages (writing association table mappings for trace linking takes several seconds) and forcing reload of the contents violates the goal of idempotency in the data generation.
Hopefully we won't have typos though, since lint rules are pretty solid.

TomeHirata · 2026-01-19T06:45:20Z

mlflow/demo/base.py

+    def _data_exists(self) -> bool:
+        """Check if demo data exists (regardless of version)."""
+
+    def delete_demo(self) -> None:


should we @abstractmethod here too?

I didn't want to force demos that don't need direct cleanup of data (transitive demos that might do something with data that another demo generates to showcase functionality) to have to create a concrete implementation that is a no-op. It's purely to reduce boilerplate.

TomeHirata · 2026-01-19T06:46:43Z

mlflow/demo/README.md

+- Modifying prompt templates
+- Any change that makes old demo data incompatible with the current UI
+
+## Creating a New Generator


Do we expect users to implement a custom demo generator?

Not at all. The README is for maintainers / contributors for providing guidance on how to add new demos. Added statements to this doc to make that clear.

harupy · 2026-01-19T07:30:54Z

mlflow/demo/README.md

+- Creates a temporary, self-contained environment (SQLite in temp directory)
+- Generates demo data automatically on startup
+- Opens browser directly to the MLflow Demo experiment
+- Auto-cleanup on exit


is there a reason we don't cache the generated data? This would be painful if the demo data generation is slow (e.g., 10 seconds).

The full demo data takes less than 1s to generate. It is faster than most loading spinners for other pages that have even modest amounts of data.

harupy · 2026-01-19T07:34:52Z

mlflow/demo/README.md

+### 2. Launch Demo Button (Home Page)
+
+For users who start `mlflow server` normally:


Do we need another entrypoint?

For testing out the functionality in corporate environments where the server and associated commands are inaccessible to users, having the ability to generate this data silently from within the UI is critical.

mlflow/demo/README.md

harupy · 2026-01-19T08:51:20Z

mlflow/demo/base.py

+from dataclasses import dataclass
+from enum import Enum
+
+DEMO_EXPERIMENT_NAME = "MLflow Demo"


Is this experiment deletable? What if a user accidentally removes it and wants to restore, or a user deletes it after trying the demo, then another user attempts to do the demo?

Yes, updated the README with all pertinent details

harupy · 2026-01-23T03:02:43Z

mlflow/demo/README.md

+### Test Structure
+
+```
+tests/demo/
+├── conftest.py              # Fixtures (tracking_uri for isolated environments)
+├── test_base.py             # BaseDemoGenerator tests
+├── test_registry.py         # DemoRegistry tests


Let's remove this. Agents can figure it out without this.

harupy · 2026-01-23T03:04:13Z

mlflow/demo/README.md

I think this README.md is too detailed. That makes it easy for it to become outdated and hard to keep in sync with the codebase. Let's keep only the essentials.

harupy

LGTM

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

Copilot AI review requested due to automatic review settings January 14, 2026 21:55

BenWilson2 mentioned this pull request Jan 14, 2026

[MLflow Demo] Add trace data for demo #19995

Merged

29 tasks

Copilot started reviewing on behalf of BenWilson2 January 14, 2026 21:56 View session

Copilot AI reviewed Jan 14, 2026

View reviewed changes

mlflow/demo/base.py Outdated Show resolved Hide resolved

tests/demo/conftest.py Outdated Show resolved Hide resolved

tests/demo/test_base.py Show resolved Hide resolved

github-actions bot added area/tracking Tracking service, tracking client APIs, autologging rn/feature Mention under Features in Changelogs. labels Jan 14, 2026

BenWilson2 force-pushed the stack/demo/scaffold branch from 72f8820 to 9d64af4 Compare January 14, 2026 22:08

BenWilson2 added the team-review Trigger a team review request label Jan 14, 2026

github-actions bot requested review from TomeHirata and harupy January 14, 2026 22:35

BenWilson2 requested a review from B-Step62 January 15, 2026 01:51

harupy reviewed Jan 15, 2026

View reviewed changes

github-actions bot assigned harupy Jan 15, 2026

TomeHirata reviewed Jan 15, 2026

View reviewed changes

github-actions bot assigned TomeHirata Jan 15, 2026

harupy reviewed Jan 15, 2026

View reviewed changes

BenWilson2 force-pushed the stack/demo/scaffold branch from 9d64af4 to 5929f11 Compare January 16, 2026 00:33

This was referenced Jan 16, 2026

[MLflow Demo] Add Eval simulation data #20046

Merged

[MLflow Demo] Add Prompt demo data #20047

Merged

[MLflow Demo] Add mlflow demo cli command #20048

Merged

BenWilson2 changed the title ~~Base implementation for demo framework~~ [MLflow Demo] Base implementation for demo framework Jan 16, 2026

BenWilson2 force-pushed the stack/demo/scaffold branch from 5929f11 to f7b4b07 Compare January 16, 2026 04:12

harupy reviewed Jan 16, 2026

View reviewed changes

TomeHirata reviewed Jan 19, 2026

View reviewed changes

harupy reviewed Jan 19, 2026

View reviewed changes

mlflow/demo/README.md Outdated Show resolved Hide resolved

harupy reviewed Jan 19, 2026

View reviewed changes

BenWilson2 force-pushed the stack/demo/scaffold branch from f7b4b07 to c27098c Compare January 20, 2026 22:46

BenWilson2 mentioned this pull request Jan 20, 2026

[ MLflow Demo ] UI updates for MLflow Demo interfaces #20162

Merged

29 tasks

BenWilson2 force-pushed the stack/demo/scaffold branch 2 times, most recently from 1a722f7 to ba7f215 Compare January 22, 2026 17:19

BenWilson2 mentioned this pull request Jan 23, 2026

[MLflow Demo] Docs for GenAI Demo #20240

Merged

29 tasks

harupy reviewed Jan 23, 2026

View reviewed changes

harupy approved these changes Jan 23, 2026

View reviewed changes

BenWilson2 mentioned this pull request Jan 23, 2026

[MLflow Demo] Add scorers demo #20287

Merged

29 tasks

Base implementation for demo framework

6ced72e

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

BenWilson2 force-pushed the stack/demo/scaffold branch from ba7f215 to 6ced72e Compare January 23, 2026 21:40

BenWilson2 added this pull request to the merge queue Jan 26, 2026

Merged via the queue into mlflow:master with commit 71495e5 Jan 26, 2026
52 checks passed

BenWilson2 deleted the stack/demo/scaffold branch January 26, 2026 14:59


		## Design Principles

		1. Auto-generated on startup - Demo data is created when `mlflow server` starts, requiring no user action.

		### 2. Launch Demo Button (Home Page)

		For users who start `mlflow server` normally:

Conversation

BenWilson2 commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🥞 Stacked PR

Related Issues/PRs

What changes are proposed in this pull request?

How is this PR tested?

Does this PR require documentation update?

Release Notes

Is this a user-facing change?

What component(s), interfaces, languages, and integrations does this PR affect?

How should the PR be classified in the release notes? Choose one:

Should this PR be included in the next patch release?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

harupy Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harupy Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harupy Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TomeHirata Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harupy Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

harupy Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

BenWilson2 commented Jan 14, 2026 •

edited

Loading

github-actions bot commented Jan 14, 2026 •

edited

Loading

harupy Jan 15, 2026 •

edited

Loading

harupy Jan 15, 2026 •

edited

Loading

harupy Jan 16, 2026 •

edited

Loading

TomeHirata Jan 19, 2026 •

edited

Loading

harupy Jan 19, 2026 •

edited

Loading

harupy Jan 19, 2026 •

edited

Loading

harupy Jan 23, 2026 •

edited

Loading