Update RAGAS documentation for wrapped scorer integration by smoorjani · Pull Request #19451 · mlflow/mlflow

smoorjani · 2025-12-17T03:43:26Z

🛠 DevTools 🛠

Install mlflow from this PR

# mlflow
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/19451/merge
# mlflow-skinny
pip install git+https://github.com/mlflow/mlflow.git@refs/pull/19451/merge#subdirectory=libs/skinny

For Databricks, use the following command:

%sh curl -LsSf https://raw.githubusercontent.com/mlflow/mlflow/HEAD/dev/install-skinny.sh | sh -s pull/19451/merge

Related Issues/PRs

#xxx

What changes are proposed in this pull request?

As titled - updates MLflow docs with RAGAS scorer integration.

How is this PR tested?

Existing unit/integration tests
New unit/integration tests
Manual tests

Does this PR require documentation update?

Release Notes

Is this a user-facing change?

No. You can skip the rest of this section.
Yes. Give a description of this change to be included in the release notes for MLflow users.

What component(s), interfaces, languages, and integrations does this PR affect?

Components

How should the PR be classified in the release notes? Choose one:

rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
rn/feature - A new user-facing feature worth mentioning in the release notes
rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
rn/documentation - A user-facing documentation change worth mentioning in the release notes

Should this PR be included in the next patch release?

Yes should be selected for bug fixes, documentation updates, and other small changes. No should be selected for new features and larger changes. If you're unsure about the release classification of this PR, leave this unchecked to let the maintainers decide.

What is a minor/patch release?

Minor release: a release that increments the second part of the version number (e.g., 1.2.0 -> 1.3.0).
Bug fixes, doc updates and new features usually go into minor releases.
Patch release: a release that increments the third part of the version number (e.g., 1.2.0 -> 1.2.1).
Bug fixes and doc updates usually go into patch releases.

Yes (this PR will be cherry-picked and included in the next patch release)
No (this PR will be included in the next minor release)

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

github-actions · 2025-12-17T18:34:22Z

Documentation preview for b8f1b73 is available at:

https://pr-19451--mlflow-docs-preview.netlify.app/docs/latest/

Changed Pages (3)

genai/eval-monitor/scorers/third-party (modified)
genai/eval-monitor/scorers/third-party/deepeval (modified)
genai/eval-monitor/scorers/third-party/ragas (added)

More info

Ignore this comment if this PR does not change the documentation.
The preview is updated when a new commit is pushed to this PR.
This comment was created by this workflow run.
The documentation was built by this workflow run.

SomtochiUmeh · 2025-12-17T22:50:30Z

docs/docs/genai/eval-monitor/scorers/third-party.mdx

+    inputs="What is MLflow?",
+    outputs="MLflow is an open-source platform for managing machine learning workflows.",


Suggested change

inputs="What is MLflow?",

outputs="MLflow is an open-source platform for managing machine learning workflows.",

Can we make this either inputs/outputs or pass in a trace and not both? Using trace here since Faithfulness needs retrieval context

SomtochiUmeh · 2025-12-17T22:51:20Z

docs/docs/genai/eval-monitor/scorers/third-party.mdx

+        "inputs": {"query": "What is MLflow?"},
+        "outputs": "MLflow is an open-source platform for managing machine learning workflows.",


Suggested change

"inputs": {"query": "What is MLflow?"},

"outputs": "MLflow is an open-source platform for managing machine learning workflows.",

Same comment as above

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

AveshCSingh

Left a couple requests inline

AveshCSingh · 2025-12-18T20:53:37Z

docs/docs/genai/eval-monitor/scorers/third-party/index.mdx

    title="DeepEval"
    href="/genai/eval-monitor/scorers/third-party/deepeval"
  />
+  <TileCard


I love these tiles linking to 3p integrations!

AveshCSingh · 2025-12-18T20:54:35Z

docs/docs/genai/eval-monitor/scorers/third-party/ragas.mdx

+```python
+from mlflow.genai.scorers.ragas import Faithfulness
+
+scorer = Faithfulness(model="openai:/gpt-4")
+feedback = scorer(trace=trace)
+
+print(feedback.value)  # Score between 0.0 and 1.0
+print(feedback.rationale)  # Explanation of the score
+```


These blocks are showing up as single lines in the PR Preview

it shows up as full for me - can you try hard-refreshing? cmd + shift + r

Good suggestion. That fixed it

docs/docs/genai/eval-monitor/scorers/third-party/ragas.mdx

AveshCSingh · 2025-12-18T20:57:48Z

docs/docs/genai/eval-monitor/scorers/third-party/ragas.mdx

+
+## Creating Scorers by Name
+
+If a particular RAGAS metric is not listed above, you can create it dynamically using <APILink fn="mlflow.genai.scorers.ragas.get_scorer">get_scorer</APILink>:


Suggested change

If a particular RAGAS metric is not listed above, you can create it dynamically using <APILink fn="mlflow.genai.scorers.ragas.get_scorer">get_scorer</APILink>:

If a particular RAGAS metric is not listed above, you can create it dynamically using <APILink fn="mlflow.genai.scorers.ragas.get_scorer">mlflow.genai.scorers.ragas.get_scorer</APILink>:

Unless they click the link, users may think they can call mlflow.genai.scorers.get_scorer.

hmm I'm probably just being dumb but why is that/why is this an issue? there's a code example of how to use it right after

oh no you're right --the code example makes this clear. Ignore me :)

AveshCSingh · 2025-12-18T20:58:36Z

docs/docs/genai/eval-monitor/scorers/third-party/ragas.mdx

+
+## Configuration
+
+RAGAS scorers accept metric-specific parameters. Any additional keyword arguments are passed directly to the RAGAS metric constructor:


Could we add an example of instantiating a RAGAS scorer that requires additional kwargs?

I actually tried finding a metric for this and didn't see any - did you have one in mind?

Maybe the strictness parameter on ResponseRelevancy? This suggestion doesn't block merge

Ah yeah, that one we haven't added yet. Let's do this as follow-up

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

smoorjani added 30 commits November 24, 2025 12:17

Add basic deepeval judge wrapping

6b02eed

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

minor fixes

9e558b3

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

move from adapter to scorer

a4e08bd

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

minor fixes

44da108

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

add unit tests

f71c5cb

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

.

2a05028

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

minor fixes

906ffe5

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

add support for dbx model serving

00baea8

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

comment out conversational metrics for now

046e117

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

fix tests and lint

a63baac

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

address pr comments

213a5c2

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

address pr feedback

523761d

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

fix ci

5c5d366

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

.

20ea827

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

,

4d2cf5f

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

remove indicators

ef77749

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

Namespace deepeval scorers in mlflow.genai.scorers.deepeval

918309a

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

fix docs example

2dcfbd5

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

.cleanup

d302edc

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

merge with master

a9768b7

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

[3/4] Add support for multi-turn deepeval scorers

5a548bf

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

fix bug

c46cbd3

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

fix bug

96b4b11

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

initial commit

1662ea5

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

update docs

b47f78c

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

merge with master

7e1e345

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

.

a737950

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

minor comments

6d03c91

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

merge with master

df027cf

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

.

f9bf969

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

smoorjani added 3 commits December 17, 2025 09:54

.

fb57922

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

Merge branch 'master' into ragas-docs

8d0a3ce

.

af73496

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

SomtochiUmeh reviewed Dec 17, 2025

View reviewed changes

github-actions bot assigned SomtochiUmeh Dec 18, 2025

smoorjani added 6 commits December 17, 2025 22:45

merged with master

afe3f0b

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

update to fit deepeval format

ae629b8

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

.

d505b9e

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

prettier

c04818d

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

.

eb2bc5a

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

merge with master

0a9d248

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

smoorjani requested review from AveshCSingh and SomtochiUmeh December 18, 2025 18:47

smoorjani added 2 commits December 18, 2025 11:10

small updates

36ef159

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

fix check

14daf28

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

AveshCSingh reviewed Dec 18, 2025

View reviewed changes

github-actions bot assigned AveshCSingh Dec 18, 2025

.

b8f1b73

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

smoorjani requested a review from AveshCSingh December 18, 2025 22:04

AveshCSingh approved these changes Dec 18, 2025

View reviewed changes

smoorjani added this pull request to the merge queue Dec 18, 2025

Merged via the queue into mlflow:master with commit d3a056d Dec 18, 2025
51 of 53 checks passed

smoorjani deleted the ragas-docs branch December 18, 2025 23:19

github-actions bot added v3.8.1 and removed v3.8.0 labels Dec 22, 2025

WeichenXu123 pushed a commit to WeichenXu123/mlflow that referenced this pull request Dec 22, 2025

Update RAGAS documentation for wrapped scorer integration (mlflow#19451)

ebe180c

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

WeichenXu123 pushed a commit that referenced this pull request Dec 22, 2025

Update RAGAS documentation for wrapped scorer integration (#19451)

73c46ad

Signed-off-by: Samraj Moorjani <samraj.moorjani@databricks.com>

		inputs="What is MLflow?",
		outputs="MLflow is an open-source platform for managing machine learning workflows.",

		"inputs": {"query": "What is MLflow?"},
		"outputs": "MLflow is an open-source platform for managing machine learning workflows.",


		## Creating Scorers by Name

		If a particular RAGAS metric is not listed above, you can create it dynamically using <APILink fn="mlflow.genai.scorers.ragas.get_scorer">get_scorer</APILink>:


		## Configuration

		RAGAS scorers accept metric-specific parameters. Any additional keyword arguments are passed directly to the RAGAS metric constructor:

Conversation

smoorjani commented Dec 17, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Install mlflow from this PR

Related Issues/PRs

What changes are proposed in this pull request?

How is this PR tested?

Does this PR require documentation update?

Release Notes

Is this a user-facing change?

What component(s), interfaces, languages, and integrations does this PR affect?

How should the PR be classified in the release notes? Choose one:

Should this PR be included in the next patch release?

Uh oh!

github-actions bot commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AveshCSingh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

smoorjani commented Dec 17, 2025 •

edited by github-actions bot

Loading

github-actions bot commented Dec 17, 2025 •

edited

Loading