[MLflow Demo] Add mlflow demo cli command by BenWilson2 · Pull Request #20048 · mlflow/mlflow

BenWilson2 · 2026-01-16T00:33:47Z

🥞 Stacked PR

Use this link to review incremental changes.

stack/demo/cli [Files changed]
- stack/demo/home [Files changed]
  - stack/demo/docs [Files changed]
    - stack/demo/scorers [Files changed]
      - stack/demo/handling [Files changed]

Related Issues/PRs

#xxx

What changes are proposed in this pull request?

Adds a new mlflow demo cli command that:

Starts up a sqlite DB in a temp directory with a local artifact path in another temp directory
Starts the tracking server pointing to the ephemeral tracking DB, finding a random safe port (or allowing override)
Generates the Demo experiment
Populates all demo data to the Demo experiment
Opens the MLflow UI in the system default browser

There is log suppression activated for the ephemeral demo server to reduce the noise in stdout. Logging can be turned on by setting the --debug flag when starting the demo server.

There is an option to start up the demo to point to an existing tracking server. A prompt is included when running the command (default "no") about whether an existing running tracking server is available. If 'y' is selected, an additional prompt for the tracking server URL is shown that will permit generating the demos and providing a link to see the demo experiment directly.

How is this PR tested?

Existing unit/integration tests
New unit/integration tests
Manual tests

Does this PR require documentation update?

Release Notes

Is this a user-facing change?

No. You can skip the rest of this section.
Yes. Give a description of this change to be included in the release notes for MLflow users.

What component(s), interfaces, languages, and integrations does this PR affect?

Components

How should the PR be classified in the release notes? Choose one:

rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
rn/feature - A new user-facing feature worth mentioning in the release notes
rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
rn/documentation - A user-facing documentation change worth mentioning in the release notes

Should this PR be included in the next patch release?

Yes should be selected for bug fixes, documentation updates, and other small changes. No should be selected for new features and larger changes. If you're unsure about the release classification of this PR, leave this unchecked to let the maintainers decide.

What is a minor/patch release?

Minor release: a release that increments the second part of the version number (e.g., 1.2.0 -> 1.3.0).
Bug fixes, doc updates and new features usually go into minor releases.
Patch release: a release that increments the third part of the version number (e.g., 1.2.0 -> 1.2.1).
Bug fixes and doc updates usually go into patch releases.

Yes (this PR will be cherry-picked and included in the next patch release)
No (this PR will be included in the next minor release)

github-actions · 2026-01-16T01:47:26Z

🛠 DevTools 🛠

Install mlflow from this PR

# mlflow
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/20048/merge
# mlflow-skinny
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/20048/merge#subdirectory=libs/skinny

For Databricks, use the following command:

%sh curl -LsSf https://raw.githubusercontent.com/mlflow/mlflow/HEAD/dev/install-skinny.sh | sh -s pull/20048/merge

Copilot

Pull request overview

This PR adds a new mlflow demo CLI command that provides a frictionless way to explore MLflow's GenAI features. The command creates a temporary environment with pre-populated demo data including traces, evaluations, and prompts, then starts a local MLflow server and opens it in a browser.

Changes:

Adds new mlflow demo CLI command with port selection and browser control options
Implements demo data generation framework with versioning support for traces, evaluations, and prompts
Adds utility functions for port availability checking
Creates comprehensive test suite and integration tests
Adds HTTP API endpoints for demo data generation and deletion

Reviewed changes

Copilot reviewed 25 out of 26 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
mlflow/cli/demo.py	New CLI command implementation that sets up ephemeral environment and launches server
mlflow/cli/init.py	Registers the new demo command
mlflow/utils/init.py	Adds `is_port_available()` helper function
mlflow/demo/	Complete demo framework with generators, registry, and base classes
mlflow/server/handlers.py	API endpoints for programmatic demo generation/deletion
tests/demo/	Comprehensive test suite including unit, integration, and CLI tests
.github/workflows/master.yml	CI configuration for running demo tests

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-16T01:49:49Z

tests/demo/test_cli.py

+        max_results=100,
+    )
+
+    assert len(traces) == 70  # 35 v1 + 35 v2


The expected trace count appears incorrect based on the test data. According to test_traces_generator.py line 98, the total should be 82 traces (35 v1 + 47 v2), not 70. The v2 set includes 12 additional prompt-linked traces that are not present in v1.

Suggested change

assert len(traces) == 70 # 35 v1 + 35 v2

assert len(traces) == 82 # 35 v1 + 47 v2

Copilot · 2026-01-16T01:49:49Z

mlflow/cli/demo.py

+            def open_browser():
+                time.sleep(1.5)
+                # Open just the base URL - the React app will handle routing
+                # Opening with fragment directly causes encoding issues
+                webbrowser.open(f"http://127.0.0.1:{port}/")


The hardcoded 1.5 second delay before opening the browser may not be sufficient on slower systems or may be unnecessarily long on fast systems. Consider using a more robust approach such as polling the server's health endpoint until it responds successfully, or making the delay configurable.

Copilot · 2026-01-16T01:49:50Z

mlflow/cli/demo.py

+
+        from urllib.parse import quote
+
+        encoded_experiment = quote(DEMO_EXPERIMENT_NAME, safe="")


The variable experiment_url is constructed but only used for display in a console message. Given the comment on line 88-89 about fragment encoding issues, consider clarifying why this URL is displayed but not used for the actual browser navigation.

Suggested change

encoded_experiment = quote(DEMO_EXPERIMENT_NAME, safe="")

encoded_experiment = quote(DEMO_EXPERIMENT_NAME, safe="")

# Construct a direct link to the demo experiment traces for display only.

# Note: We intentionally do not use this URL with `webbrowser.open()` because

# opening with a fragment causes encoding issues (see comment below).

Copilot · 2026-01-16T01:49:50Z

tests/utils/test_utils.py

+
+    port = find_free_port()
+    assert isinstance(port, int)
+    assert 1 <= port <= 65535


The port range validation is incorrect. Valid port numbers range from 1 to 65535, but line 210 checks 1 <= port <= 65535 which would allow port 0. Additionally, privileged ports (1-1023) typically cannot be bound without elevated permissions. The assertion should be 1024 <= port <= 65535 or the test should handle potential permission errors when testing with low port numbers.

Suggested change

assert 1 <= port <= 65535

assert 1024 <= port <= 65535

Copilot · 2026-01-16T01:49:51Z

mlflow/demo/generators/traces.py

+        # Also create the EntityAssociation record for proper filtering support
+        if trace_id and trace_def.linked_prompt:
+            try:
+                client = mlflow.MlflowClient()
+                prompt_version = client.get_prompt_version(
+                    name=trace_def.linked_prompt.prompt_name,
+                    version=str(trace_def.linked_prompt.version),
+                )
+                client.link_prompt_versions_to_trace(
+                    prompt_versions=[prompt_version],
+                    trace_id=trace_id,
+                )
+            except Exception:
+                _logger.debug("Failed to link prompt to trace", exc_info=True)


The broad except Exception clause silently swallows all errors. While this is intentional for demo data generation resilience, consider catching more specific exceptions (e.g., MlflowException) to avoid masking unexpected errors like AttributeError or KeyError that might indicate code bugs.

Copilot · 2026-01-16T01:49:51Z

mlflow/demo/generators/evaluation.py

+    def pass_fail_scorer(inputs, outputs) -> Feedback:
+        content = str(inputs) + str(outputs)
+        hash_input = f"{content}:{name}"
+        hash_val = int(hashlib.md5(hash_input.encode(), usedforsecurity=False).hexdigest()[:8], 16)


The usedforsecurity=False parameter is correct for this demo use case (deterministic scoring based on content hashes). However, this parameter was added in Python 3.9. While MLflow likely requires Python 3.9+, consider verifying the minimum supported Python version to ensure compatibility.

Copilot · 2026-01-16T01:49:51Z

mlflow/server/handlers.py

+    deleted_features = []
+    for name in demo_registry.list_generators():
+        generator = demo_registry.get(name)()
+        if generator._data_exists():


The API endpoint is calling a private method _data_exists() on the generator. This breaks encapsulation. Consider exposing a public method or property for this check, or using the existing is_generated() method if appropriate.

Suggested change

if generator._data_exists():

if generator.is_generated():

Copilot · 2026-01-16T01:49:52Z

.github/workflows/master.yml

            --quiet --requires-ssh --ignore-flavors \
            --ignore=tests/examples \
            --ignore=tests/evaluate \
            --ignore=tests/genai \


The demo tests are excluded from the main test job but run in a separate job. Consider adding a comment explaining why demo tests are separated (e.g., for parallelization, different dependencies, or to manage test execution time).

Suggested change

--ignore=tests/genai \

--ignore=tests/genai \

# Demo tests are excluded from the main test job and executed in a separate job to keep this

# job's runtime manageable and to isolate their additional dependencies.

github-actions · 2026-01-16T01:55:45Z

Documentation preview for f424136 is available at:

https://pr-20048--mlflow-docs-preview.netlify.app/docs/latest/

More info

Ignore this comment if this PR does not change the documentation.
The preview is updated when a new commit is pushed to this PR.
This comment was created by this workflow run.
The documentation was built by this workflow run.

harupy · 2026-01-19T07:53:03Z

.github/workflows/master.yml

            pytest tests/genai --ignore tests/genai/test_genai_import_without_agent_sdk.py \
            --ignore tests/genai/optimize --ignore tests/genai/prompts

+  demo:


is there a reason we need to separate this job?

Good call - there isn't a reason. I've updated the CI def to include it in the Python tests. Thanks for the call out!

B-Step62 · 2026-01-29T12:24:28Z

mlflow/cli/demo.py

+    if value is None:
+        return None
+    if not value.startswith(("http://", "https://")):
+        raise click.BadParameter("Tracking URI must start with 'http://' or 'https://'")


Why does this need to be http? Does demo generation logic break with sql tracking URI?

I actually wasn't even thinking of a local tracking server haha. Great point!

B-Step62 · 2026-01-29T12:26:08Z

mlflow/cli/demo.py

+    if tracking_uri is None:
+        tracking_uri = _get_tracking_uri_interactive(port)
+
+    if tracking_uri == _NEW_SERVER_SENTINEL:


Do we need _NEW_SERVER_SENTINEL? Does None work?

Nah None will work. Updated!

B-Step62 · 2026-01-29T12:29:38Z

mlflow/cli/demo.py

+    import mlflow
+    from mlflow.demo import generate_all_demos
+    from mlflow.demo.base import DEMO_EXPERIMENT_NAME


Do we need lazy import?

Yeah otherwise the eager load chain creates a huge import dependency that really messes up CI.

B-Step62 · 2026-01-29T12:37:05Z

mlflow/cli/demo.py

+    with tempfile.TemporaryDirectory() as tmpdir:
+        tmpdir_path = Path(tmpdir)
+        db_path = tmpdir_path / "mlflow.db"
+        artifact_path = tmpdir_path / "artifacts"
+        artifact_path.mkdir()


Is the intention of using temporary directory is not to persist the demo data? I wonder if it is useful to keep the demo data around, otherwise user need to regenerate it again if they stop the demo command once.

There wasn't a strong reason to do it other than to not pollute a user's machine if they just wanted to check it out. I wasn't strongly opinionated either way. Updated to use a reserved directory name for persistence.

B-Step62 · 2026-01-29T13:30:21Z

mlflow/server/handlers.py

+
+@catch_mlflow_exception
+@_disable_if_artifacts_only
+def _generate_demo():


Shall we add a parameter for list of features to generate demo, with default to all?

good idea! Will add

B-Step62 · 2026-02-02T11:09:37Z

mlflow/cli/demo.py

+    demo_dir = Path.home() / ".mlflow" / "demo"
+    demo_dir.mkdir(parents=True, exist_ok=True)


@BenWilson2 Is there any reason for not using the current directory? I think it is confusing the mlflow server and mlflow demo behaves differently in terms of where to create database.

Excellent point - there's really no need to have a different path here. Updated to use the standard path

…plates - Add timestamp distribution across 7 days (v1 in days 0-3.5, v2 in days 3.5-7) - Add token count estimation as span attributes (SpanAttributeKey.CHAT_USAGE) - Add prompt-based traces with template rendering and variables - Restructure sessions to have 2-4 turns each across 3 sessions - Fix trace metadata by using InMemoryTraceManager directly - Update test expectations for new trace counts (34 total: 4 RAG, 4 agent, 12 prompt, 14 session) Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

B-Step62

LGTM!

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com> Co-authored-by: Claude <noreply@anthropic.com>

This was referenced Jan 16, 2026

[MLflow Demo] Base implementation for demo framework #19994

Merged

[MLflow Demo] Add trace data for demo #19995

Merged

[MLflow Demo] Add Eval simulation data #20046

Merged

[MLflow Demo] Add Prompt demo data #20047

Merged

BenWilson2 changed the title ~~add cli for demo~~ [MLflow Demo] Add mlflow demo cli command Jan 16, 2026

BenWilson2 marked this pull request as ready for review January 16, 2026 01:47

Copilot AI review requested due to automatic review settings January 16, 2026 01:47

github-actions bot added area/tracking Tracking service, tracking client APIs, autologging rn/feature Mention under Features in Changelogs. labels Jan 16, 2026

Copilot started reviewing on behalf of BenWilson2 January 16, 2026 01:47 View session

Copilot AI reviewed Jan 16, 2026

View reviewed changes

BenWilson2 force-pushed the stack/demo/cli branch 3 times, most recently from 6f6e5cd to a6f6926 Compare January 17, 2026 04:24

BenWilson2 added the team-review Trigger a team review request label Jan 18, 2026

github-actions bot requested review from daniellok-db and harupy January 18, 2026 19:29

harupy reviewed Jan 19, 2026

View reviewed changes

github-actions bot assigned harupy Jan 19, 2026

BenWilson2 force-pushed the stack/demo/cli branch from a6f6926 to 8a97c4b Compare January 20, 2026 22:46

BenWilson2 mentioned this pull request Jan 20, 2026

[ MLflow Demo ] UI updates for MLflow Demo interfaces #20162

Merged

29 tasks

BenWilson2 force-pushed the stack/demo/cli branch 5 times, most recently from 70a0aa1 to 1495094 Compare January 22, 2026 18:56

This was referenced Jan 23, 2026

[MLflow Demo] Docs for GenAI Demo #20240

Merged

[MLflow Demo] Add scorers demo #20287

Merged

BenWilson2 force-pushed the stack/demo/cli branch 2 times, most recently from 6178743 to 5b91a6f Compare January 26, 2026 15:00

BenWilson2 mentioned this pull request Jan 26, 2026

[MLflow Demo] Add server availability handling checks #20349

Merged

29 tasks

BenWilson2 force-pushed the stack/demo/cli branch 3 times, most recently from 04d0776 to 6858f52 Compare January 28, 2026 19:16

B-Step62 reviewed Jan 29, 2026

View reviewed changes

github-actions bot assigned B-Step62 Jan 29, 2026

BenWilson2 force-pushed the stack/demo/cli branch 5 times, most recently from 30b2c83 to 095ef9a Compare January 30, 2026 03:26

BenWilson2 requested a review from B-Step62 January 30, 2026 03:29

B-Step62 reviewed Feb 2, 2026

View reviewed changes

BenWilson2 and others added 6 commits February 2, 2026 10:13

add eval generation

4c99c45

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

add prompt generation

07cc952

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

add eval generation

e4bc6f5

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

add cli for demo

f424136

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

BenWilson2 force-pushed the stack/demo/cli branch from 095ef9a to f424136 Compare February 2, 2026 16:06

B-Step62 approved these changes Feb 3, 2026

View reviewed changes

B-Step62 added this pull request to the merge queue Feb 3, 2026

Merged via the queue into mlflow:master with commit ef73690 Feb 3, 2026
51 of 53 checks passed

BenWilson2 added a commit to BenWilson2/mlflow that referenced this pull request Feb 5, 2026

[MLflow Demo] Add mlflow demo cli command (mlflow#20048)

8a82575

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com> Co-authored-by: Claude <noreply@anthropic.com>

BenWilson2 added a commit to BenWilson2/mlflow that referenced this pull request Feb 11, 2026

[MLflow Demo] Add mlflow demo cli command (mlflow#20048)

7e641ea

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com> Co-authored-by: Claude <noreply@anthropic.com>

BenWilson2 added a commit to BenWilson2/mlflow that referenced this pull request Feb 18, 2026

[MLflow Demo] Add mlflow demo cli command (mlflow#20048)

5d073a2

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com> Co-authored-by: Claude <noreply@anthropic.com>

	assert len(traces) == 70 # 35 v1 + 35 v2
	assert len(traces) == 82 # 35 v1 + 47 v2


		from urllib.parse import quote

		encoded_experiment = quote(DEMO_EXPERIMENT_NAME, safe="")

-        encoded_experiment = quote(DEMO_EXPERIMENT_NAME, safe="")
+        encoded_experiment = quote(DEMO_EXPERIMENT_NAME, safe="")
+        # Construct a direct link to the demo experiment traces for display only.
+        # Note: We intentionally do not use this URL with `webbrowser.open()` because
+        # opening with a fragment causes encoding issues (see comment below).

		demo_dir = Path.home() / ".mlflow" / "demo"
		demo_dir.mkdir(parents=True, exist_ok=True)

Conversation

BenWilson2 commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🥞 Stacked PR

Related Issues/PRs

What changes are proposed in this pull request?

How is this PR tested?

Does this PR require documentation update?

Release Notes

Is this a user-facing change?

What component(s), interfaces, languages, and integrations does this PR affect?

How should the PR be classified in the release notes? Choose one:

Should this PR be included in the next patch release?

Uh oh!

github-actions bot commented Jan 16, 2026

Install mlflow from this PR

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

B-Step62 left a comment

Choose a reason for hiding this comment

Uh oh!

BenWilson2 commented Jan 16, 2026 •

edited

Loading

github-actions bot commented Jan 16, 2026 •

edited

Loading