[MLflow Demo] Add Prompt demo data by BenWilson2 · Pull Request #20047 · mlflow/mlflow

BenWilson2 · 2026-01-16T00:33:44Z

🥞 Stacked PR

Use this link to review incremental changes.

stack/demo/prompts [Files changed]
- stack/demo/cli [Files changed]
  - stack/demo/home [Files changed]
    - stack/demo/docs [Files changed]
      - stack/demo/scorers [Files changed]
        
        stack/demo/handling [Files changed]

Related Issues/PRs

#xxx

What changes are proposed in this pull request?

Adds data for simulating prompt evolution, including example traces (with evaluation) on versions of prompts for 3 separate domains.

How is this PR tested?

Existing unit/integration tests
New unit/integration tests
Manual tests

Does this PR require documentation update?

Release Notes

Is this a user-facing change?

No. You can skip the rest of this section.
Yes. Give a description of this change to be included in the release notes for MLflow users.

What component(s), interfaces, languages, and integrations does this PR affect?

Components

How should the PR be classified in the release notes? Choose one:

rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
rn/feature - A new user-facing feature worth mentioning in the release notes
rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
rn/documentation - A user-facing documentation change worth mentioning in the release notes

Should this PR be included in the next patch release?

Yes should be selected for bug fixes, documentation updates, and other small changes. No should be selected for new features and larger changes. If you're unsure about the release classification of this PR, leave this unchecked to let the maintainers decide.

What is a minor/patch release?

Minor release: a release that increments the second part of the version number (e.g., 1.2.0 -> 1.3.0).
Bug fixes, doc updates and new features usually go into minor releases.
Patch release: a release that increments the third part of the version number (e.g., 1.2.0 -> 1.2.1).
Bug fixes and doc updates usually go into patch releases.

Yes (this PR will be cherry-picked and included in the next patch release)
No (this PR will be included in the next minor release)

github-actions · 2026-01-16T01:40:38Z

🛠 DevTools 🛠

Install mlflow from this PR

# mlflow
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/20047/merge
# mlflow-skinny
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/20047/merge#subdirectory=libs/skinny

For Databricks, use the following command:

%sh curl -LsSf https://raw.githubusercontent.com/mlflow/mlflow/HEAD/dev/install-skinny.sh | sh -s pull/20047/merge

Copilot

Pull request overview

This PR adds demo prompt data to the MLflow demo data framework, creating realistic examples of prompt evolution across three different domains (customer support, document summarization, and code review). The prompts demonstrate versioning, alias management, and linkage to traces.

Changes:

Adds 3 demo prompts with 4 versions each showing realistic prompt evolution
Creates 12 prompt-linked traces that demonstrate prompt-to-trace associations
Implements PromptsDemoGenerator with full lifecycle management (create, check, delete)
Adds comprehensive test coverage for prompt generation and validation
Updates CI workflow to run demo tests in a separate job

Reviewed changes

Copilot reviewed 18 out of 19 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
mlflow/demo/data.py	Adds 3 prompt definitions (CUSTOMER_SUPPORT_PROMPT, DOCUMENT_SUMMARIZER_PROMPT, CODE_REVIEWER_PROMPT) with version history and 12 prompt-linked trace examples
mlflow/demo/generators/prompts.py	Implements PromptsDemoGenerator for creating, checking, and deleting prompt demo data
mlflow/demo/generators/traces.py	Adds _create_prompt_linked_trace method to link traces to specific prompt versions
mlflow/demo/generators/init.py	Registers PromptsDemoGenerator with the demo registry
tests/demo/test_prompts_generator.py	Unit tests for PromptsDemoGenerator functionality
tests/demo/test_demo_integration.py	Integration tests validating prompt generation against a real server
.github/workflows/master.yml	Adds separate CI job for demo tests

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-16T01:43:09Z

tests/demo/test_traces_generator.py

+    TracesDemoGenerator.version = 2
+    assert generator.is_generated() is False
+
+    TracesDemoGenerator.version = 1


Missing cleanup after test modifies class variable. The version is reset to 1 at the end, but if the test fails before line 111, it will leave TracesDemoGenerator in an inconsistent state. Use a fixture with proper cleanup like in test_demo_integration.py (lines 48-52).

Copilot · 2026-01-16T01:43:09Z

tests/demo/test_evaluation_generator.py

+    EvaluationDemoGenerator.version = 99
+    fresh_generator = EvaluationDemoGenerator()
+    assert fresh_generator.is_generated() is False
+
+    EvaluationDemoGenerator.version = 1


Missing cleanup after test modifies class variable. The version is reset to 1 at the end, but if the test fails before line 122, it will leave EvaluationDemoGenerator in an inconsistent state. Use a fixture with proper cleanup like in test_demo_integration.py (lines 56-60).

Copilot · 2026-01-16T01:43:09Z

tests/demo/test_prompts_generator.py

+    PromptsDemoGenerator.version = 99
+    fresh_generator = PromptsDemoGenerator()
+    assert fresh_generator.is_generated() is False
+
+    PromptsDemoGenerator.version = 2


Missing cleanup after test modifies class variable. The version is reset to 2 at the end, but if the test fails before line 105, it will leave PromptsDemoGenerator in an inconsistent state. Use a fixture with proper cleanup like in test_demo_integration.py (lines 64-68).

Copilot · 2026-01-16T01:43:10Z

mlflow/demo/generators/prompts.py

+            navigation_url="#/prompts",
+        )
+
+    def _create_prompt_with_versions(self, prompt_def) -> int:


Missing type hint for prompt_def parameter. Should be annotated as DemoPromptDef for better code clarity and IDE support.

Copilot · 2026-01-16T01:43:10Z

mlflow/demo/data.py

+
+def get_expected_answers() -> dict[str, str]:
+    """Build a dict mapping queries to expected responses for evaluation."""
+    return {trace.query.lower(): trace.expected_response for trace in ALL_DEMO_TRACES}


Dictionary is rebuilt on every call. Since ALL_DEMO_TRACES is a constant list, consider caching this dictionary at module level to avoid repeated iteration and lowercasing operations.

Suggested change

def get_expected_answers() -> dict[str, str]:

"""Build a dict mapping queries to expected responses for evaluation."""

return {trace.query.lower(): trace.expected_response for trace in ALL_DEMO_TRACES}

_EXPECTED_ANSWERS_BY_QUERY: dict[str, str] = {

trace.query.lower(): trace.expected_response for trace in ALL_DEMO_TRACES

}

def get_expected_answers() -> dict[str, str]:

"""Build a dict mapping queries to expected responses for evaluation."""

return _EXPECTED_ANSWERS_BY_QUERY

github-actions · 2026-01-16T01:48:12Z

Documentation preview for 5c2543c is available at:

https://pr-20047--mlflow-docs-preview.netlify.app/docs/latest/

More info

Ignore this comment if this PR does not change the documentation.
The preview is updated when a new commit is pushed to this PR.
This comment was created by this workflow run.
The documentation was built by this workflow run.

B-Step62

LGTM w/ a minor suggestion

B-Step62 · 2026-01-29T12:11:45Z

mlflow/demo/data.py

+@dataclass
+class PromptVersionDef:
+    template: str | list[dict[str, str]]
+    commit_message: str
+    alias: str | None = None
+
+
+@dataclass
+class DemoPromptDef:
+    name: str
+    versions: list[PromptVersionDef]


Can we use the real PromptVersion object?

yep will try!

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

This was referenced Jan 16, 2026

[MLflow Demo] Base implementation for demo framework #19994

Merged

[MLflow Demo] Add trace data for demo #19995

Merged

[MLflow Demo] Add Eval simulation data #20046

Merged

[MLflow Demo] Add mlflow demo cli command #20048

Merged

BenWilson2 changed the title ~~add prompt generation~~ [MLflow Demo] Add Prompt demo data Jan 16, 2026

BenWilson2 marked this pull request as ready for review January 16, 2026 01:40

Copilot AI review requested due to automatic review settings January 16, 2026 01:40

github-actions bot added area/tracking Tracking service, tracking client APIs, autologging rn/feature Mention under Features in Changelogs. labels Jan 16, 2026

Copilot started reviewing on behalf of BenWilson2 January 16, 2026 01:40 View session

Copilot AI reviewed Jan 16, 2026

View reviewed changes

BenWilson2 force-pushed the stack/demo/prompts branch 2 times, most recently from 991dccd to 23cadcf Compare January 17, 2026 04:24

BenWilson2 added team-review Trigger a team review request and removed team-review Trigger a team review request labels Jan 18, 2026

github-actions bot requested review from B-Step62 and WeichenXu123 January 20, 2026 15:08

BenWilson2 force-pushed the stack/demo/prompts branch from 23cadcf to c599580 Compare January 20, 2026 22:46

BenWilson2 mentioned this pull request Jan 20, 2026

[ MLflow Demo ] UI updates for MLflow Demo interfaces #20162

Merged

29 tasks

BenWilson2 force-pushed the stack/demo/prompts branch 4 times, most recently from 39a2634 to 0ad9742 Compare January 22, 2026 17:20

This was referenced Jan 23, 2026

[MLflow Demo] Docs for GenAI Demo #20240

Merged

[MLflow Demo] Add scorers demo #20287

Merged

BenWilson2 force-pushed the stack/demo/prompts branch 2 times, most recently from ae40158 to b89db10 Compare January 26, 2026 15:00

BenWilson2 mentioned this pull request Jan 26, 2026

[MLflow Demo] Add server availability handling checks #20349

Merged

29 tasks

BenWilson2 force-pushed the stack/demo/prompts branch from b89db10 to b4bcdc7 Compare January 27, 2026 19:45

B-Step62 approved these changes Jan 29, 2026

View reviewed changes

github-actions bot assigned B-Step62 Jan 29, 2026

BenWilson2 force-pushed the stack/demo/prompts branch 3 times, most recently from 856dc7b to e1f4aac Compare January 29, 2026 22:11

add prompt generation

5c2543c

Signed-off-by: Ben Wilson <benjamin.wilson@databricks.com>

BenWilson2 force-pushed the stack/demo/prompts branch from e1f4aac to 5c2543c Compare January 29, 2026 23:01

BenWilson2 added this pull request to the merge queue Jan 30, 2026

Merged via the queue into mlflow:master with commit 63bb97a Jan 30, 2026
49 checks passed

BenWilson2 deleted the stack/demo/prompts branch January 30, 2026 02:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MLflow Demo] Add Prompt demo data#20047

[MLflow Demo] Add Prompt demo data#20047
BenWilson2 merged 1 commit intomlflow:masterfrom
BenWilson2:stack/demo/prompts

BenWilson2 commented Jan 16, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Jan 16, 2026

Install mlflow from this PR

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jan 16, 2026

Uh oh!

Copilot AI Jan 16, 2026

Uh oh!

Copilot AI Jan 16, 2026

Uh oh!

Copilot AI Jan 16, 2026

Uh oh!

Copilot AI Jan 16, 2026

Uh oh!

github-actions bot commented Jan 16, 2026 •

edited

Loading

Uh oh!

B-Step62 left a comment

Uh oh!

B-Step62 Jan 29, 2026

Uh oh!

BenWilson2 Jan 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

BenWilson2 commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🥞 Stacked PR

Related Issues/PRs

What changes are proposed in this pull request?

How is this PR tested?

Does this PR require documentation update?

Release Notes

Is this a user-facing change?

What component(s), interfaces, languages, and integrations does this PR affect?

How should the PR be classified in the release notes? Choose one:

Should this PR be included in the next patch release?

Uh oh!

github-actions bot commented Jan 16, 2026

Install mlflow from this PR

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

B-Step62 left a comment

Choose a reason for hiding this comment

Uh oh!

B-Step62 Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

BenWilson2 Jan 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

BenWilson2 commented Jan 16, 2026 •

edited

Loading

github-actions bot commented Jan 16, 2026 •

edited

Loading