feat: add `logfire.url_from_eval(report)` method by Kludex · Pull Request #1694 · pydantic/logfire

Kludex · 2026-02-09T09:21:47Z

Summary

Adds logfire.url_from_eval(report) method that generates a Logfire dashboard link to view a pydantic_evals.EvaluationReport
Stores project_url on LogfireConfig from credentials (file or token validation)
Constructs URL as {project_url}/evals/compare?experiment={trace_id}-{span_id}, returning None if any piece is missing
Adds pydantic-evals as a dev dependency for type checking

Test plan

make typecheck — 0 errors
make lint — all checks passed
uv run pytest tests/test_url_from_eval.py — 5 tests covering happy path and all None cases
uv run pytest tests/test_logfire_api.py — logfire-api shim tests pass
Pre-commit hooks pass

🤖 Generated with Claude Code

cloudflare-workers-and-pages · 2026-02-09T09:22:52Z

Deploying logfire-docs with Cloudflare Pages

Latest commit:	`c3b864c`
Status:	✅ Deploy successful!
Preview URL:	https://b7e8b84d.logfire-docs.pages.dev
Branch Preview URL:	https://feat-url-from-eval.logfire-docs.pages.dev

View logs

…rd links for eval reports Users running pydantic-evals evaluations can now easily get a Logfire dashboard link to view their evaluation report by calling `logfire.url_from_eval(report)`. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Strip trailing slash from project_url before building URL - Load project_url from credentials file regardless of send_to_logfire - Add test for trailing slash handling Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

# Conflicts: # uv.lock

…_eval When the token is provided via env var without a credentials file, project_url is only populated in a background thread. url_from_eval now joins that thread first to ensure the URL is available. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Uses requests_mock to exercise the full path: configure with token (no creds file) -> background thread validates token -> url_from_eval waits for thread -> returns correct URL. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

dmontagu · 2026-02-13T06:42:14Z

I feel like we need a place to put functions that are specific to our web platform, ideally grouped a bit more by function. There are going to be variables soon, then datasets, and more to come. I feel like we should be a bit thoughtful about where the APIs go rather than just haphazardly adding methods to the logfire instance. But 🤷 if it unblocks stuff it's probably worth biasing toward action

Use a generation counter so that check_tokens threads from a previous configure() call do not write a stale project_url. Also fix the test_url_from_eval_waits_for_token_validation test to use tmp_path so it does not read from a local credentials file. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Covers the generation-check branches in check_tokens to satisfy 100% coverage. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Use two tokens so the orphaned thread's second loop iteration hits the generation-mismatch early return. Also assert that Event.wait and Thread.join actually succeed within the timeout. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Save _check_tokens_thread to a local variable before checking and joining, so a concurrent configure() cannot set it to None between the two reads. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

alexmojaki · 2026-03-25T13:26:39Z

logfire/_internal/config.py


+            # Try loading credentials from a file.
+            # We do this before checking send_to_logfire so that project_url
+            # is available for url_from_eval even when not sending data.


but should url_from_eval return something when not sending data?

No - addressed in bca78cf. Credential loading moved back inside the send_to_logfire block, so url_from_eval returns None when not sending data.

alexmojaki · 2026-03-25T13:29:21Z

logfire/_internal/config.py

                        with suppress_instrumentation():
                            for token in token_list:
+                                if self._config_generation != generation:
+                                    return


needs a comment

No longer applicable - the generation check was removed entirely in bca78cf.

alexmojaki · 2026-03-25T13:30:12Z

logfire/_internal/config.py

+                    raise
+                credentials = None
+            if credentials is not None:
+                self.project_url = self.project_url or credentials.project_url


Suggested change

self.project_url = self.project_url or credentials.project_url

self.project_url = credentials.project_url

when would this differ?

If project_url is not available (i.e. not sending data to Logfire), url_from_eval returns None since the URL wouldn't work anyway. This removes the early credential loading, background thread waiting, and generation tracking that were added to support url_from_eval when not sending data. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

When the token comes from an env var without a creds file, project_url was never set because credentials was None at that point. Now check_tokens also sets project_url from the validated credentials returned by the /v1/info endpoint. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

logfire-api/logfire_api/__init__.py

tests/test_url_from_eval.py

- Use ellipsis body style in logfire-api shim for url_from_eval - Use ImportError instead of Exception in test import guard

The import chain can fail with AttributeError (not ImportError) when pydantic is too old, since pydantic_evals transitively imports pydantic_ai which uses pydantic.Tag.

Kludex · 2026-03-26T09:22:29Z

Re: "P3: After printing a validated token summary, add the token to printed_tokens" - not an issue. Each token appears at most once in token_list, and printed_tokens already guards against double-printing: a token is added at line 1101 (eager print from creds file), and checked at line 1115 (background thread). There's no path where a token prints twice.

Kludex · 2026-03-26T09:25:08Z

I think we are good now @alexmojaki

Kludex · 2026-03-26T17:18:25Z

logfire/_internal/config.py

        self.advanced = advanced

        self.additional_span_processors = additional_span_processors
+        self.project_url: str | None = None


Add _ prefix project_url.

Kludex · 2026-03-27T08:58:34Z

omg, I need to this manually

Kludex · 2026-03-27T09:04:03Z

@alexmojaki applied manually the leading underscore.

devin-ai-integration

Devin Review found 1 new potential issue.

View 5 additional findings in Devin Review.

devin-ai-integration · 2026-03-27T09:09:26Z

logfire/_internal/main.py

+    def url_from_eval(self, report: EvaluationReport[Any, Any, Any]) -> str | None:
+        """Generate a Logfire URL to view an evaluation report.
+
+        Args:
+            report: An evaluation report from `pydantic_evals`.
+
+        Returns:
+            The URL string, or `None` if the project URL or trace/span IDs are not available.
+        """
+        project_url = self._config._project_url  # type: ignore[reportPrivateUsage]
+        trace_id = report.trace_id
+        span_id = report.span_id
+        if not project_url or not trace_id or not span_id:
+            return None
+        return f'{project_url.rstrip("/")}/evals/compare?experiment={trace_id}-{span_id}'


🚩 _project_url availability depends on async token validation completing

When Logfire is configured with a token from an environment variable (no credentials file), _project_url will remain None until the background thread at logfire/_internal/config.py:1121 completes the HTTP request to /v1/info and sets it at line 1114. If url_from_eval is called before this thread completes, it will return None even though the project URL will eventually be available. This is documented behavior ('or None if the project URL ... are not available'), but users may find it surprising if they call url_from_eval immediately after configure(). A potential improvement would be to provide a way to wait for the background thread to complete, or to document the timing dependency more prominently.

Was this helpful? React with 👍 or 👎 to provide feedback.

Kludex requested a review from Copilot February 9, 2026 09:22

Copilot started reviewing on behalf of Kludex February 9, 2026 09:23 View session

Kludex force-pushed the feat/url-from-eval branch 2 times, most recently from b86eb5c to 8cf0b55 Compare February 9, 2026 09:28

Kludex changed the title ~~feat: Add logfire.url_from_eval(report) method~~ feat: add logfire.url_from_eval(report) method Feb 9, 2026

Kludex requested a review from dmontagu February 9, 2026 09:29

This comment was marked as resolved.

Sign in to view

Kludex force-pushed the feat/url-from-eval branch 2 times, most recently from edb283a to dd8d80b Compare February 9, 2026 09:34

Kludex force-pushed the feat/url-from-eval branch from dd8d80b to d654bed Compare February 9, 2026 09:38

Kludex requested a review from alexmojaki February 9, 2026 09:46

This comment was marked as resolved.

Sign in to view

fix: Address PR review comments

17de3ed

- Strip trailing slash from project_url before building URL - Load project_url from credentials file regardless of send_to_logfire - Add test for trailing slash handling Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

This comment was marked as resolved.

Sign in to view

Kludex and others added 3 commits February 9, 2026 16:14

Merge remote-tracking branch 'origin/main' into feat/url-from-eval

7d6cc6e

# Conflicts: # uv.lock

This comment was marked as resolved.

Sign in to view

Kludex and others added 3 commits March 25, 2026 11:57

Merge remote-tracking branch 'origin/main' into feat/url-from-eval

46e77da

Add test for stale project_url race on reconfigure

2bfc8d4

Covers the generation-check branches in check_tokens to satisfy 100% coverage. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

This comment was marked as resolved.

Sign in to view

Kludex and others added 2 commits March 25, 2026 14:00

Avoid TOCTOU race in wait_for_token_validation

5d2a63a

Save _check_tokens_thread to a local variable before checking and joining, so a concurrent configure() cannot set it to None between the two reads. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into feat/url-from-eval

065ea81

alexmojaki reviewed Mar 25, 2026

View reviewed changes

This comment was marked as resolved.

Sign in to view

Kludex and others added 2 commits March 25, 2026 16:39

Match url_from_eval return type in logfire-api shim

afb32ee

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

This comment was marked as resolved.

Sign in to view

Kludex commented Mar 26, 2026

View reviewed changes

logfire-api/logfire_api/__init__.py Outdated Show resolved Hide resolved

tests/test_url_from_eval.py Show resolved Hide resolved

Kludex added 3 commits March 26, 2026 10:16

Address pending review comments

b5ea178

- Use ellipsis body style in logfire-api shim for url_from_eval - Use ImportError instead of Exception in test import guard

Merge remote-tracking branch 'origin/main' into feat/url-from-eval

f9d835d

Revert to except Exception for pydantic_evals import guard

d463c11

The import chain can fail with AttributeError (not ImportError) when pydantic is too old, since pydantic_evals transitively imports pydantic_ai which uses pydantic.Tag.

This comment was marked as resolved.

Sign in to view

Kludex requested a review from alexmojaki March 26, 2026 09:24

Kludex commented Mar 26, 2026

View reviewed changes

This comment was marked as resolved.

Sign in to view

Kludex force-pushed the feat/url-from-eval branch from 2f51fbc to d463c11 Compare March 27, 2026 08:58

Kludex added 2 commits March 27, 2026 10:02

fix lint

d742de8

Merge branch 'main' into feat/url-from-eval

c3b864c

devin-ai-integration bot reviewed Mar 27, 2026

View reviewed changes

alexmojaki merged commit 7ee31f6 into main Mar 27, 2026
22 checks passed

alexmojaki deleted the feat/url-from-eval branch March 27, 2026 10:14

	self.project_url = self.project_url or credentials.project_url
	self.project_url = credentials.project_url

Conversation

Kludex commented Feb 9, 2026

Summary

Test plan

Uh oh!

cloudflare-workers-and-pages bot commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying logfire-docs with Cloudflare Pages

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

dmontagu commented Feb 13, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

alexmojaki Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Kludex Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

alexmojaki Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Kludex Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

alexmojaki Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

Kludex commented Mar 26, 2026

Uh oh!

Kludex commented Mar 26, 2026

Uh oh!

Kludex Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

Kludex commented Mar 27, 2026

Uh oh!

Kludex commented Mar 27, 2026

Uh oh!

devin-ai-integration bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration bot Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cloudflare-workers-and-pages bot commented Feb 9, 2026 •

edited

Loading