mlflow.genai.evaluate(): handle case where root span is unavailable by dbczumar · Pull Request #19220 · mlflow/mlflow

dbczumar · 2025-12-04T18:55:06Z

🛠 DevTools 🛠

Install mlflow from this PR

# mlflow
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/19220/merge
# mlflow-skinny
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/19220/merge#subdirectory=libs/skinny

For Databricks, use the following command:

%sh curl -LsSf https://raw.githubusercontent.com/mlflow/mlflow/HEAD/dev/install-skinny.sh | sh -s pull/19220/merge

Related Issues/PRs

#xxx

What changes are proposed in this pull request?

mlflow.genai.evaluate(): handle case where root span is unavailable. This can happen for partial third-party OTel traces or if the user calls search_traces(include_spans=False) and then tries to pass the resulting traces to eval.

How is this PR tested?

Existing unit/integration tests
New unit/integration tests
Manual tests

Does this PR require documentation update?

Release Notes

Is this a user-facing change?

No. You can skip the rest of this section.
Yes. Give a description of this change to be included in the release notes for MLflow users.

What component(s), interfaces, languages, and integrations does this PR affect?

Components

How should the PR be classified in the release notes? Choose one:

rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
rn/feature - A new user-facing feature worth mentioning in the release notes
rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
rn/documentation - A user-facing documentation change worth mentioning in the release notes

Should this PR be included in the next patch release?

Yes should be selected for bug fixes, documentation updates, and other small changes. No should be selected for new features and larger changes. If you're unsure about the release classification of this PR, leave this unchecked to let the maintainers decide.

What is a minor/patch release?

Minor release: a release that increments the second part of the version number (e.g., 1.2.0 -> 1.3.0).
Bug fixes, doc updates and new features usually go into minor releases.
Patch release: a release that increments the third part of the version number (e.g., 1.2.0 -> 1.2.1).
Bug fixes and doc updates usually go into patch releases.

Yes (this PR will be cherry-picked and included in the next patch release)
No (this PR will be included in the next minor release)

Add None check before accessing root span inputs/outputs to prevent AttributeError. Warn when traces are missing root spans but keep them in the dataset for downstream processing. Signed-off-by: dbczumar <corey.zumar@databricks.com>

dbczumar · 2025-12-04T18:56:17Z

tests/genai/evaluate/test_utils.py

    assert transformed_data["inputs"].isna().all()


+def test_convert_to_eval_set_with_missing_root_span():


On master, this fails with:

Traceback (most recent call last): File "/Users/corey.zumar/mlflowrepos/mlflow4/find_jarvis_issues.py", line 112, in <module> main() File "/Users/corey.zumar/mlflowrepos/mlflow4/find_jarvis_issues.py", line 86, in main issues = find_issues( File "/Users/corey.zumar/mlflowrepos/mlflow4/insights.py", line 431, in find_issues mlflow.genai.evaluate( File "/Users/corey.zumar/mlflowrepos/mlflow4/mlflow/genai/evaluation/base.py", line 248, in evaluate df = _convert_to_eval_set(data) File "/Users/corey.zumar/mlflowrepos/mlflow4/mlflow/genai/evaluation/utils.py", line 121, in _convert_to_eval_set .pipe(_extract_request_response_from_trace) File "/Users/corey.zumar/miniconda3/envs/mlflow/lib/python3.10/site-packages/pandas/core/generic.py", line 6253, in pipe return common.pipe(self, func, *args, **kwargs) File "/Users/corey.zumar/miniconda3/envs/mlflow/lib/python3.10/site-packages/pandas/core/common.py", line 502, in pipe return func(obj, *args, **kwargs) File "/Users/corey.zumar/mlflowrepos/mlflow4/mlflow/genai/evaluation/utils.py", line 185, in _extract_request_response_from_trace df["inputs"] = df["trace"].apply(lambda trace: trace.data._get_root_span().inputs) File "/Users/corey.zumar/miniconda3/envs/mlflow/lib/python3.10/site-packages/pandas/core/series.py", line 4943, in apply ).apply() File "/Users/corey.zumar/miniconda3/envs/mlflow/lib/python3.10/site-packages/pandas/core/apply.py", line 1422, in apply return self.apply_standard() File "/Users/corey.zumar/miniconda3/envs/mlflow/lib/python3.10/site-packages/pandas/core/apply.py", line 1502, in apply_standard mapped = obj._map_values( File "/Users/corey.zumar/miniconda3/envs/mlflow/lib/python3.10/site-packages/pandas/core/base.py", line 925, in _map_values return algorithms.map_array(arr, mapper, na_action=na_action, convert=convert) File "/Users/corey.zumar/miniconda3/envs/mlflow/lib/python3.10/site-packages/pandas/core/algorithms.py", line 1743, in map_array return lib.map_infer(values, mapper, convert=convert) File "pandas/_libs/lib.pyx", line 2999, in pandas._libs.lib.map_infer File "/Users/corey.zumar/mlflowrepos/mlflow4/mlflow/genai/evaluation/utils.py", line 185, in <lambda> df["inputs"] = df["trace"].apply(lambda trace: trace.data._get_root_span().inputs) AttributeError: 'NoneType' object has no attribute 'inputs'

Copilot

Pull request overview

This PR adds defensive handling for cases where the root span is unavailable in mlflow.genai.evaluate(), preventing potential AttributeErrors when traces are fetched without spans (e.g., using search_traces(..., include_spans=False)).

Introduces a _safe_extract_from_root_span helper function that safely checks for None root spans before accessing attributes
Adds a warning when traces without root spans are detected to inform users about the issue
Includes comprehensive test coverage for the missing root span scenario

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
mlflow/genai/evaluation/utils.py	Adds `_safe_extract_from_root_span` helper function to safely handle None root spans and logs warnings when traces lack root spans
tests/genai/evaluate/test_utils.py	Adds test case `test_convert_to_eval_set_with_missing_root_span` to verify correct handling of traces without root spans

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-04T18:58:16Z

mlflow/genai/evaluation/utils.py

+        )
+
+    # Warn about traces that don't have a root span (where inputs/outputs are None)
+    missing_count = df[["inputs", "outputs"]].isna().any(axis=1).sum()


The warning logic counts rows where either inputs or outputs is None, but the warning message suggests it's specifically about missing root spans. When a root span is missing, both inputs and outputs will be None. Consider using .all(axis=1) instead of .any(axis=1) to only count rows where both are None, which would more accurately identify traces without root spans:

missing_count = df[["inputs", "outputs"]].isna().all(axis=1).sum()

This would avoid false positives where a trace has a root span but only one of inputs or outputs is None.

Suggested change

missing_count = df[["inputs", "outputs"]].isna().any(axis=1).sum()

missing_count = df[["inputs", "outputs"]].isna().all(axis=1).sum()

github-actions · 2025-12-04T19:02:09Z

Documentation preview for bc7ea8a is available at:

https://pr-19220--mlflow-docs-preview.netlify.app/docs/latest/

More info

Ignore this comment if this PR does not change the documentation.
The preview is updated when a new commit is pushed to this PR.
This comment was created by this workflow run.
The documentation was built by this workflow run.

Add None check before accessing root span inputs/outputs to prevent AttributeError. Warn when traces are missing root spans but keep them in the dataset for downstream processing. Signed-off-by: dbczumar <corey.zumar@databricks.com>

Signed-off-by: dbczumar <corey.zumar@databricks.com>

BenWilson2

Looks like a good guard.

…lflow#19220) Signed-off-by: dbczumar <corey.zumar@databricks.com>

…19220) Signed-off-by: dbczumar <corey.zumar@databricks.com>

dbczumar added 2 commits December 4, 2025 10:51

Copilot AI review requested due to automatic review settings December 4, 2025 18:55

github-actions bot added the v3.7.0 label Dec 4, 2025

dbczumar requested a review from AveshCSingh December 4, 2025 18:55

github-actions bot added area/evaluation MLflow Evaluation rn/none List under Small Changes in Changelogs. labels Dec 4, 2025

Copilot started reviewing on behalf of dbczumar December 4, 2025 18:55 View session

dbczumar commented Dec 4, 2025

View reviewed changes

Copilot finished reviewing on behalf of dbczumar December 4, 2025 18:57

Copilot AI reviewed Dec 4, 2025

View reviewed changes

dbczumar added 3 commits December 4, 2025 11:25

fix

18766da

Signed-off-by: dbczumar <corey.zumar@databricks.com>

fix

bc7ea8a

Signed-off-by: dbczumar <corey.zumar@databricks.com>

BenWilson2 approved these changes Dec 4, 2025

View reviewed changes

AveshCSingh approved these changes Dec 4, 2025

View reviewed changes

BenWilson2 added this pull request to the merge queue Dec 4, 2025

Merged via the queue into mlflow:master with commit 35d0705 Dec 4, 2025
48 of 50 checks passed

BenWilson2 pushed a commit to BenWilson2/mlflow that referenced this pull request Dec 4, 2025

mlflow.genai.evaluate(): handle case where root span is unavailable (m…

7fce792

…lflow#19220) Signed-off-by: dbczumar <corey.zumar@databricks.com>

BenWilson2 pushed a commit that referenced this pull request Dec 4, 2025

mlflow.genai.evaluate(): handle case where root span is unavailable (#…

e6857c5

…19220) Signed-off-by: dbczumar <corey.zumar@databricks.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mlflow.genai.evaluate(): handle case where root span is unavailable#19220

mlflow.genai.evaluate(): handle case where root span is unavailable#19220
BenWilson2 merged 5 commits intomlflow:masterfrom
dbczumar:root_span_miss

dbczumar commented Dec 4, 2025 •

edited

Loading

Uh oh!

dbczumar Dec 4, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Dec 4, 2025

Uh oh!

github-actions bot commented Dec 4, 2025 •

edited

Loading

Uh oh!

BenWilson2 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		assert transformed_data["inputs"].isna().all()


		def test_convert_to_eval_set_with_missing_root_span():

	missing_count = df[["inputs", "outputs"]].isna().any(axis=1).sum()
	missing_count = df[["inputs", "outputs"]].isna().all(axis=1).sum()

Conversation

dbczumar commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Install mlflow from this PR

Related Issues/PRs

What changes are proposed in this pull request?

How is this PR tested?

Does this PR require documentation update?

Release Notes

Is this a user-facing change?

What component(s), interfaces, languages, and integrations does this PR affect?

How should the PR be classified in the release notes? Choose one:

Should this PR be included in the next patch release?

Uh oh!

dbczumar Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BenWilson2 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dbczumar commented Dec 4, 2025 •

edited

Loading

github-actions bot commented Dec 4, 2025 •

edited

Loading