Add mdtests for Ruff by ntBre · Pull Request #24617 · astral-sh/ruff

ntBre · 2026-04-13T21:29:41Z

Summary

This PR adds the ruff_test crate, a parallel crate to ty_test for Ruff, to
enable the new mdtests in ruff_linter. I opted to follow the Db-based
structure of ty_test to simplify the integration, but we end up basically just
unpacking the files from the Db to call the test_contents function from the
linter.

I also updated the section of the contributing guidelines on lint rule testing to suggest using mdtests.

Test Plan

I copied over the S704 tests into a new mdtest. I selected this rule because it had several separate fixture files that all used different settings and each of the fixtures was relatively short. I had initially reached for a more complicated rule (UP046), but this seemed less likely to clutter the diff while still exercising most mdtest features.

astral-sh-bot · 2026-04-13T21:31:46Z

Typing conformance results

No changes detected ✅

Current numbers

The percentage of diagnostics emitted that were expected errors held steady at 89.36%. The percentage of expected errors that received a diagnostic held steady at 85.49%. The number of fully passing files held steady at 88/134.

astral-sh-bot · 2026-04-13T21:32:04Z

Memory usage report

Memory usage unchanged ✅

astral-sh-bot · 2026-04-13T21:35:13Z

`ecosystem-analyzer` results

No diagnostic changes detected ✅

Full report with detailed diff (timing results)

astral-sh-bot · 2026-04-13T21:42:07Z

`ruff-ecosystem` results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

Summary -- This is a first step toward adding mdtests for Ruff. I actually wrote the code in the opposite order, first copy-pasting `ty_test` to a `ruff_test` crate, and then factoring out the shared code, but I figured it would be easier to review in this order. I also opened a stacked PR with the `ruff_test` changes (#24617) to show that the API works well for that too. The main change here is moving several of the modules from `ty_test` to a new `mdtest` crate: - `assertion` - `diagnostic` - `matcher` - `parser` Beyond moving these files to the new crate, I made `Matcher` functions take a `&dyn Db` to support passing a different concrete type from `ruff_test`, and I also made the parser generic over an `MdtestConfig` trait to allow Ruff to use a separate config struct. I also introduced new `TestConfig` and `TestDb` types to allow testing the `matcher` and `parser` within the `mdtest` crate without depending on either the real ty `Db` or `ty_test` config type. The lib.rs file from `ty_test` was essentially split in half, with the shared code moved to the `mdtest` crate and the ty-specific parts kept in `ty_test`. Test Plan -- All existing mdtests and the unit tests from `ty_test` should still pass, and the stacked branch with the `ruff_test` crate tests the split API

## Summary This is a follow-up to #24616 to share a bit more infrastructure between `ty_test` and the `ruff_test` crate introduced in #24617. In particular, the primary `run` function has moved into `mdtest`, along with the `attempt_test`, `check_panic` and `snapshot_diagnostics` helpers. You should be able to review commit-by-commit, but the first commit is probably the most important to look at since it moves the `tracing` setup and inconsistency checks from `ty_test::run` into `ty_test::run_test`. I think that narrows the span of the `tracing` logs a bit, but the important section should still be covered. The other commits should be pretty straightforward and reviewable as one diff. ## Test Plan Existing tests, as well as rebasing #24617 onto this branch.

ntBre · 2026-05-14T18:14:10Z

+        Self::default()
+    }
+
+    pub(crate) fn use_in_memory_system(&mut self) {


Codex made a rather subtle observation when I had it review this today. I think the use of a Db, or at least of the InMemorySystem, means that Ruff rules that depend on actual file system checks won't operate correctly if we try to write them as mdtests. Codex linked to this is_package function as just one example:

ruff/crates/ruff_linter/src/packaging.rs

Lines 28 to 33 in b3e4466

pub fn is_package(path: &Path, namespace_packages: &[PathBuf]) -> bool {

namespace_packages

.iter()

.any(|namespace_package| path.starts_with(namespace_package))

|| path.join("__init__.py").is_file()

}

I think this is a reasonable limitation for now since this will affect a small number of Ruff's rules, and I don't think we should prioritize migrating existing tests to mdtests anyway, but I at least wanted to call it out to get other opinions.

I liked how clean this Db version ended up looking compared to making the mdtest crate generic enough to handle Ruff's native file types, but it does introduce potential mismatches like this.

ty_mdtest supports using the os system for tests like these

ruff/crates/ty_test/src/config.rs

Lines 25 to 32 in ffa347b

pub(crate) log: Option<Log>,

pub(crate) analysis: Option<Analysis>,

/// The [`ruff_db::system::System`] to use for tests.

///

/// Defaults to the case-sensitive [`ruff_db::system::InMemorySystem`].

pub(crate) system: Option<SystemKind>,

It also supports enabling logging, which I often found very useful to debug test errors.

I don't think it's necessary to land this as part of this PR but it should also not be too hard, I think

Oh right. I realized last night that we may want a layer of indirection in the config to support this kind of test-specific setting instead of using the real Options directly. I also wouldn't mind deferring it for now, though, since it shouldn't be too hard to add later if/when we need it.

Codex threw together this patch when I was exploring the idea, which looks pretty reasonable (at least after moving ty_test::MdtestSystem to mdtest to reuse it here instead of duplicating it):

Details

diff --git a/Cargo.lock b/Cargo.lock index 7c7beb3961..3a370e38a9 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3282,6 +3282,9 @@ dependencies = [ "ruff_python_ast", "ruff_workspace", "salsa", + "serde", + "tempfile", + "toml 1.1.2+spec-1.1.0", ] [[package]] diff --git a/crates/ruff_linter/resources/mdtest/test-rules.md b/crates/ruff_linter/resources/mdtest/test-rules.md index 2dccda83d6..bb0ee67530 100644 --- a/crates/ruff_linter/resources/mdtest/test-rules.md +++ b/crates/ruff_linter/resources/mdtest/test-rules.md @@ -13,3 +13,17 @@ select = ["RUF990"] ```py 1 ``` + +# Testing Ruff mdtest config + +```toml +lint.select = ["F401"] + +[mdtest] +log = false +system = "os" +``` + +```py +import os # error: [unused-import] +``` diff --git a/crates/ruff_mdtest/Cargo.toml b/crates/ruff_mdtest/Cargo.toml index 0c3e2d1b24..0526b12fb3 100644 --- a/crates/ruff_mdtest/Cargo.toml +++ b/crates/ruff_mdtest/Cargo.toml @@ -25,6 +25,9 @@ ruff_workspace = { workspace = true } anyhow = { workspace = true } camino = { workspace = true } salsa = { workspace = true } +serde = { workspace = true, features = ["derive"] } +tempfile = { workspace = true } +toml = { workspace = true } [dev-dependencies] datatest-stable = { workspace = true } diff --git a/crates/ruff_mdtest/src/db.rs b/crates/ruff_mdtest/src/db.rs index 4770ee591f..9d687be986 100644 --- a/crates/ruff_mdtest/src/db.rs +++ b/crates/ruff_mdtest/src/db.rs @@ -1,24 +1,42 @@ +use camino::{Utf8Component, Utf8PathBuf}; use ruff_db::Db as SourceDb; use ruff_db::files::Files; -use ruff_db::system::{DbWithWritableSystem, InMemorySystem, System, SystemPath, WritableSystem}; +use ruff_db::system::{ + CaseSensitivity, DbWithWritableSystem, InMemorySystem, OsSystem, System, SystemPath, + SystemPathBuf, WhichResult, WritableSystem, +}; use ruff_db::vendored::VendoredFileSystem; +use ruff_notebook::{Notebook, NotebookError}; +use std::borrow::Cow; +use std::sync::Arc; +use tempfile::TempDir; #[salsa::db] -#[derive(Clone, Default)] +#[derive(Clone)] pub(crate) struct Db { storage: salsa::Storage<Self>, files: Files, - system: InMemorySystem, + system: MdtestSystem, vendored: VendoredFileSystem, } impl Db { pub(crate) fn setup() -> Self { - Self::default() + Self { + storage: salsa::Storage::default(), + files: Files::default(), + system: MdtestSystem::in_memory(), + vendored: VendoredFileSystem::default(), + } + } + + pub(crate) fn use_os_system_with_temp_dir(&mut self, cwd: SystemPathBuf, temp_dir: TempDir) { + self.system.with_os(cwd, temp_dir); + Files::sync_all(self); } pub(crate) fn use_in_memory_system(&mut self) { - self.system.fs().remove_all(); + self.system.with_in_memory(); Files::sync_all(self); } @@ -50,8 +68,193 @@ impl SourceDb for Db { impl salsa::Database for Db {} impl DbWithWritableSystem for Db { - type System = InMemorySystem; + type System = MdtestSystem; fn writable_system(&self) -> &Self::System { &self.system } } + +#[derive(Debug, Clone)] +pub(crate) struct MdtestSystem(Arc<MdtestSystemInner>); + +#[derive(Debug)] +enum MdtestSystemInner { + InMemory(InMemorySystem), + Os { + os_system: OsSystem, + _temp_dir: TempDir, + }, +} + +impl MdtestSystem { + fn in_memory() -> Self { + Self(Arc::new(MdtestSystemInner::InMemory( + InMemorySystem::default(), + ))) + } + + fn as_system(&self) -> &dyn WritableSystem { + match &*self.0 { + MdtestSystemInner::InMemory(system) => system, + MdtestSystemInner::Os { os_system, .. } => os_system, + } + } + + fn with_os(&mut self, cwd: SystemPathBuf, temp_dir: TempDir) { + self.0 = Arc::new(MdtestSystemInner::Os { + os_system: OsSystem::new(cwd), + _temp_dir: temp_dir, + }); + } + + fn with_in_memory(&mut self) { + if let MdtestSystemInner::InMemory(in_memory) = &*self.0 { + in_memory.fs().remove_all(); + } else { + self.0 = Arc::new(MdtestSystemInner::InMemory(InMemorySystem::default())); + } + } + + fn normalize_path<'a>(&self, path: &'a SystemPath) -> Cow<'a, SystemPath> { + match &*self.0 { + MdtestSystemInner::InMemory(_) => Cow::Borrowed(path), + MdtestSystemInner::Os { os_system, .. } => { + // Keep all mdtest paths confined to the temporary OS-backed root. + let without_root: Utf8PathBuf = path + .components() + .skip_while(|component| { + matches!( + component, + Utf8Component::RootDir | Utf8Component::Prefix(..) + ) + }) + .collect(); + Cow::Owned(os_system.current_directory().join(&without_root)) + } + } + } +} + +impl System for MdtestSystem { + fn path_metadata( + &self, + path: &SystemPath, + ) -> ruff_db::system::Result<ruff_db::system::Metadata> { + self.as_system().path_metadata(&self.normalize_path(path)) + } + + fn canonicalize_path(&self, path: &SystemPath) -> ruff_db::system::Result<SystemPathBuf> { + let canonicalized = self + .as_system() + .canonicalize_path(&self.normalize_path(path))?; + + if let MdtestSystemInner::Os { os_system, .. } = &*self.0 { + Ok(canonicalized + .strip_prefix(os_system.current_directory()) + .unwrap() + .to_owned()) + } else { + Ok(canonicalized) + } + } + + fn read_to_string(&self, path: &SystemPath) -> ruff_db::system::Result<String> { + self.as_system().read_to_string(&self.normalize_path(path)) + } + + fn read_to_notebook(&self, path: &SystemPath) -> Result<Notebook, NotebookError> { + self.as_system() + .read_to_notebook(&self.normalize_path(path)) + } + + fn read_virtual_path_to_string( + &self, + path: &ruff_db::system::SystemVirtualPath, + ) -> ruff_db::system::Result<String> { + self.as_system().read_virtual_path_to_string(path) + } + + fn read_virtual_path_to_notebook( + &self, + path: &ruff_db::system::SystemVirtualPath, + ) -> Result<Notebook, NotebookError> { + self.as_system().read_virtual_path_to_notebook(path) + } + + fn path_exists_case_sensitive(&self, path: &SystemPath, prefix: &SystemPath) -> bool { + self.as_system() + .path_exists_case_sensitive(&self.normalize_path(path), &self.normalize_path(prefix)) + } + + fn case_sensitivity(&self) -> CaseSensitivity { + self.as_system().case_sensitivity() + } + + fn which(&self, name: &str) -> WhichResult { + self.as_system().which(name) + } + + fn current_directory(&self) -> &SystemPath { + self.as_system().current_directory() + } + + fn user_config_directory(&self) -> Option<SystemPathBuf> { + self.as_system().user_config_directory() + } + + fn cache_dir(&self) -> Option<SystemPathBuf> { + self.as_system().cache_dir() + } + + fn read_directory<'a>( + &'a self, + path: &SystemPath, + ) -> ruff_db::system::Result< + Box<dyn Iterator<Item = ruff_db::system::Result<ruff_db::system::DirectoryEntry>> + 'a>, + > { + self.as_system().read_directory(&self.normalize_path(path)) + } + + fn walk_directory( + &self, + path: &SystemPath, + ) -> ruff_db::system::walk_directory::WalkDirectoryBuilder { + self.as_system().walk_directory(&self.normalize_path(path)) + } + + fn as_writable(&self) -> Option<&dyn WritableSystem> { + Some(self) + } + + fn as_any(&self) -> &dyn std::any::Any { + self + } + + fn as_any_mut(&mut self) -> &mut dyn std::any::Any { + self + } + + fn dyn_clone(&self) -> Box<dyn System> { + Box::new(self.clone()) + } +} + +impl WritableSystem for MdtestSystem { + fn create_new_file(&self, path: &SystemPath) -> ruff_db::system::Result<()> { + self.as_system().create_new_file(&self.normalize_path(path)) + } + + fn write_file_bytes(&self, path: &SystemPath, content: &[u8]) -> ruff_db::system::Result<()> { + self.as_system() + .write_file_bytes(&self.normalize_path(path), content) + } + + fn create_directory_all(&self, path: &SystemPath) -> ruff_db::system::Result<()> { + self.as_system() + .create_directory_all(&self.normalize_path(path)) + } + + fn dyn_clone(&self) -> Box<dyn WritableSystem> { + Box::new(self.clone()) + } +} diff --git a/crates/ruff_mdtest/src/lib.rs b/crates/ruff_mdtest/src/lib.rs index e48762942e..ee7a825e97 100644 --- a/crates/ruff_mdtest/src/lib.rs +++ b/crates/ruff_mdtest/src/lib.rs @@ -8,14 +8,16 @@ use ruff_db::diagnostic::{FileResolver, Input, UnifiedFile}; use ruff_db::files::{File, system_path_to_file}; use ruff_db::source::source_text; use ruff_db::system::{DbWithWritableSystem as _, SystemPathBuf}; +use ruff_db::testing::{setup_logging, setup_logging_with_filter}; use ruff_linter::source_kind::SourceKind; use ruff_linter::test::test_contents; use ruff_notebook::NotebookIndex; use ruff_workspace::configuration::Configuration; -use ruff_workspace::options::Options; +use crate::config::{Log, MarkdownTestConfig, SystemKind}; use crate::db::Db; +mod config; mod db; pub fn run( @@ -49,10 +51,30 @@ fn run_test( db: &mut Db, relative_fixture_path: &Utf8Path, snapshot_path: &Utf8Path, - test: &parser::MarkdownTest<Options>, + test: &parser::MarkdownTest<MarkdownTestConfig>, ) -> Result<(TestOutcome, Vec<MarkdownEdit>), Failures> { + let _tracing = test.configuration().log.as_ref().and_then(|log| match log { + Log::Bool(enabled) => enabled.then(setup_logging), + Log::Filter(filter) => setup_logging_with_filter(filter), + }); + // Initialize the system and remove all files and directories to reset the system to a clean state. - db.use_in_memory_system(); + match test.configuration().system.unwrap_or_default() { + SystemKind::InMemory => db.use_in_memory_system(), + SystemKind::Os => { + let dir = tempfile::TempDir::new().expect("Creating a temporary directory to succeed"); + let root_path = dir + .path() + .canonicalize() + .expect("Canonicalizing to succeed"); + let root_path = SystemPathBuf::from_path_buf(root_path) + .expect("Temp directory to be a valid UTF8 path") + .simplified() + .to_path_buf(); + + db.use_os_system_with_temp_dir(root_path, dir); + } + } let project_root = SystemPathBuf::from("/src"); db.create_directory_all(&project_root) @@ -84,7 +106,7 @@ fn run_test( .collect(); let settings = Configuration::from_options( - test.configuration().clone(), + test.configuration().options.clone(), None, project_root.as_std_path(), ) @@ -203,6 +225,6 @@ impl FileResolver for RuffResolver<'_> { fn parse<'s>( short_title: &'s str, source: &'s str, -) -> anyhow::Result<parser::MarkdownTestSuite<'s, Options>> { - parser::parse::<Options>(short_title, source, |_| Ok(())) +) -> anyhow::Result<parser::MarkdownTestSuite<'s, MarkdownTestConfig>> { + parser::parse::<MarkdownTestConfig>(short_title, source, |_| Ok(())) }

codspeed-hq · 2026-05-14T18:14:18Z

Merging this PR will not alter performance

✅ 117 untouched benchmarks

_{Comparing brent/ruff-mdtests (a7ea610) with main (ec93ddf)}

ntBre · 2026-05-14T21:05:55Z

I'll keep re-running the macOS CI because I don't think I actually broke anything, but I think this is otherwise ready for a look.

MichaReiser

Nice! I've mainly one comment about using Db for Ruff tests. We're now in this weird state where the mdtest crate has partial support for systems under tests without a Db. I suggest in my inline comment to remove the FileResolver support from the mdtest crate, which then justifies the use of a Db within ruff_test. I also propose a solution for rules that access the file system directly (like isort and such)

MichaReiser · 2026-05-15T07:02:30Z

+#[salsa::db]
+#[derive(Clone, Default)]
+pub(crate) struct Db {
+    storage: salsa::Storage<Self>,


Can you say more why we use a salsa DB here over e.g. a type that simply wraps a TempDir guard? It's not clear to me what the advantages of using Salsa are, given that Ruff doesn't use Salsa (today).

I'm asking because it feels somewhat inconsistent, given that some parts of the mdtest framework support the FileResolver abstractions when others don't. But maybe that's fine. I also haven't looked into what it would take to use the FileResolver abstraction consistently in the mdtest crate.

I don't have a strong opinion on this one. It has just been very unexpected.

Edit: After mulling over this a little longer, I sort of like using a db, but we should frame it differently. It's not required that the code under test uses a db, but the mdtest framework requires the use of a db, and it should not support FileResolver.

We can accomplish this very easily by rewriting the Diagnostics from Ruff and mapping every Span with a RuffFile to a span with a File, then pass these diagnostics to the mdtest framework. I think I would like this the most because it simplifies the mdtest framework. We don't need to think about supporting both RuffFile and File, instead it can assume that it always operates with a Db.

Patch for dropping `FileResolver` from `mdtest` (Codex generated)

diff --git a/Cargo.lock b/Cargo.lock index 8759886130..5b35cc1899 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3555,7 +3555,6 @@ dependencies = [ "mdtest", "ruff_db", "ruff_linter", - "ruff_notebook", "ruff_python_ast", "ruff_workspace", "salsa", diff --git a/crates/mdtest/src/lib.rs b/crates/mdtest/src/lib.rs index 2389076123..9bc8f1a945 100644 --- a/crates/mdtest/src/lib.rs +++ b/crates/mdtest/src/lib.rs @@ -6,9 +6,10 @@ use colored::Colorize; use similar::{ChangeTag, TextDiff}; use ruff_db::Db; -use ruff_db::diagnostic::{Diagnostic, DisplayDiagnosticConfig, FileResolver}; +use ruff_db::diagnostic::{Diagnostic, DisplayDiagnosticConfig}; use ruff_db::files::File; use ruff_db::panic::{PanicError, catch_unwind}; +use ruff_db::source::line_index; use ruff_diagnostics::Applicability; use ruff_source_file::{LineIndex, OneIndexed}; use ruff_text_size::{Ranged, TextRange}; @@ -319,24 +320,20 @@ pub(crate) fn diagnostic_display_config(tool_name: &'static str) -> DisplayDiagn .context(0) } -pub fn render_diagnostic( - resolver: &dyn FileResolver, - tool_name: &'static str, - diagnostic: &Diagnostic, -) -> String { +pub fn render_diagnostic(db: &dyn Db, tool_name: &'static str, diagnostic: &Diagnostic) -> String { diagnostic - .display(resolver, &diagnostic_display_config(tool_name)) + .display(&db, &diagnostic_display_config(tool_name)) .to_string() } pub(crate) fn render_diagnostics( - resolver: &dyn FileResolver, + db: &dyn Db, tool_name: &'static str, diagnostics: &[Diagnostic], ) -> String { let mut rendered = String::new(); for diag in diagnostics { - writeln!(rendered, "{}", render_diagnostic(resolver, tool_name, diag)).unwrap(); + writeln!(rendered, "{}", render_diagnostic(db, tool_name, diag)).unwrap(); } rendered.trim_end_matches('\n').to_string() @@ -357,15 +354,14 @@ pub(crate) fn apply_snapshot_filters(rendered: &str) -> std::borrow::Cow<'_, str } pub fn validate_inline_snapshot( - resolver: &dyn FileResolver, + db: &dyn Db, tool_name: &'static str, test_file: &TestFile<'_>, inline_diagnostics: &[Diagnostic], markdown_edits: &mut Vec<MarkdownEdit>, ) -> Result<(), matcher::FailuresByLine> { let update_snapshots = is_update_inline_snapshots_enabled(); - let input = resolver.input(test_file.file); - let line_index = input.line_index(); + let line_index = line_index(db, test_file.file); let mut failures = matcher::FailuresByLine::default(); let mut inline_diagnostics = inline_diagnostics; @@ -419,9 +415,8 @@ pub fn validate_inline_snapshot( continue; }; - let actual = - apply_snapshot_filters(&render_diagnostics(resolver, tool_name, block_diagnostics)) - .into_owned(); + let actual = apply_snapshot_filters(&render_diagnostics(db, tool_name, block_diagnostics)) + .into_owned(); let Some(snapshot_code_block) = code_block.inline_snapshot_block() else { if update_snapshots { @@ -522,7 +517,7 @@ fn try_apply_markdown_edits( } pub fn create_diagnostic_snapshot<'d, C>( - resolver: &dyn FileResolver, + db: &dyn Db, tool_name: &'static str, relative_fixture_path: &Utf8Path, test: &parser::MarkdownTest<'_, '_, C>, @@ -564,12 +559,7 @@ pub fn create_diagnostic_snapshot<'d, C>( writeln!(snapshot).unwrap(); } writeln!(snapshot, "```").unwrap(); - write!( - snapshot, - "{}", - render_diagnostic(resolver, tool_name, diagnostic) - ) - .unwrap(); + write!(snapshot, "{}", render_diagnostic(db, tool_name, diagnostic)).unwrap(); writeln!(snapshot, "```").unwrap(); } snapshot @@ -695,7 +685,7 @@ pub fn check_panic<C>(test: &MarkdownTest<'_, '_, C>, panic_info: Option<PanicEr pub fn snapshot_diagnostics<C>( test: &MarkdownTest<'_, '_, C>, - resolver: &dyn FileResolver, + db: &dyn Db, tool_name: &'static str, relative_fixture_path: &Utf8Path, snapshot_path: &Utf8Path, @@ -710,7 +700,7 @@ pub fn snapshot_diagnostics<C>( ); let snapshot = crate::create_diagnostic_snapshot( - resolver, + db, tool_name, relative_fixture_path, test, diff --git a/crates/ruff_linter/resources/mdtest/flake8-bandit.md b/crates/ruff_linter/resources/mdtest/flake8-bandit.md index 9b8e5e6c18..04f64e0797 100644 --- a/crates/ruff_linter/resources/mdtest/flake8-bandit.md +++ b/crates/ruff_linter/resources/mdtest/flake8-bandit.md @@ -23,7 +23,7 @@ Markup(object="safe") ```snapshot error[S704]: Unsafe use of `markupsafe.Markup` detected - --> mdtest_snippet.py:5:1 + --> src/mdtest_snippet.py:5:1 | 5 | Markup(f"unsafe {content}") # snapshot: unsafe-markup-use | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -31,7 +31,7 @@ error[S704]: Unsafe use of `markupsafe.Markup` detected error[S704]: Unsafe use of `flask.Markup` detected - --> mdtest_snippet.py:6:1 + --> src/mdtest_snippet.py:6:1 | 6 | flask.Markup("unsafe {}".format(content)) # snapshot: unsafe-markup-use | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -39,7 +39,7 @@ error[S704]: Unsafe use of `flask.Markup` detected error[S704]: Unsafe use of `markupsafe.Markup` detected - --> mdtest_snippet.py:10:1 + --> src/mdtest_snippet.py:10:1 | 10 | Markup(content) # snapshot: unsafe-markup-use | ^^^^^^^^^^^^^^^ @@ -47,7 +47,7 @@ error[S704]: Unsafe use of `markupsafe.Markup` detected error[S704]: Unsafe use of `flask.Markup` detected - --> mdtest_snippet.py:11:1 + --> src/mdtest_snippet.py:11:1 | 11 | flask.Markup("unsafe %s" % content) # snapshot: unsafe-markup-use | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -71,7 +71,7 @@ flask.Markup("hello {}".format("world")) # snapshot: unsafe-markup-use ```snapshot error[S704]: Unsafe use of `markupsafe.Markup` detected - --> mdtest_snippet.py:14:1 + --> src/mdtest_snippet.py:14:1 | 14 | Markup("*" * 8) # snapshot: unsafe-markup-use | ^^^^^^^^^^^^^^^ @@ -79,7 +79,7 @@ error[S704]: Unsafe use of `markupsafe.Markup` detected error[S704]: Unsafe use of `flask.Markup` detected - --> mdtest_snippet.py:15:1 + --> src/mdtest_snippet.py:15:1 | 15 | flask.Markup("hello {}".format("world")) # snapshot: unsafe-markup-use | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -104,7 +104,7 @@ literal(f"unsafe {content}") # snapshot: unsafe-markup-use ```snapshot error[S704]: Unsafe use of `markupsafe.Markup` detected - --> mdtest_snippet.py:5:1 + --> src/mdtest_snippet.py:5:1 | 5 | Markup(f"unsafe {content}") # snapshot: unsafe-markup-use | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -112,7 +112,7 @@ error[S704]: Unsafe use of `markupsafe.Markup` detected error[S704]: Unsafe use of `webhelpers.html.literal` detected - --> mdtest_snippet.py:6:1 + --> src/mdtest_snippet.py:6:1 | 6 | literal(f"unsafe {content}") # snapshot: unsafe-markup-use | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -132,7 +132,7 @@ literal(f"unsafe {content}") # snapshot: unsafe-markup-use ```snapshot error[S704]: Unsafe use of `webhelpers.html.literal` detected - --> mdtest_snippet.py:10:1 + --> src/mdtest_snippet.py:10:1 | 10 | literal(f"unsafe {content}") # snapshot: unsafe-markup-use | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -160,7 +160,7 @@ Markup(cleaned) # snapshot: unsafe-markup-use ```snapshot error[S704]: Unsafe use of `markupsafe.Markup` detected - --> mdtest_snippet.py:9:1 + --> src/mdtest_snippet.py:9:1 | 9 | Markup(cleaned) # snapshot: unsafe-markup-use | ^^^^^^^^^^^^^^^ diff --git a/crates/ruff_test/Cargo.toml b/crates/ruff_test/Cargo.toml index 2074709c01..04a11b6ea7 100644 --- a/crates/ruff_test/Cargo.toml +++ b/crates/ruff_test/Cargo.toml @@ -18,7 +18,6 @@ test = false mdtest = { workspace = true } ruff_db = { workspace = true, features = ["os", "testing"] } ruff_linter = { workspace = true, features = ["testing"] } -ruff_notebook = { workspace = true } ruff_python_ast = { workspace = true } ruff_workspace = { workspace = true } diff --git a/crates/ruff_test/src/lib.rs b/crates/ruff_test/src/lib.rs index 8298c3ed9f..bc1ca14363 100644 --- a/crates/ruff_test/src/lib.rs +++ b/crates/ruff_test/src/lib.rs @@ -5,13 +5,12 @@ use mdtest::{ Failures, FileFailures, MarkdownEdit, TestFile, TestOutcome, attempt_test, matcher, parser, }; use ruff_db::Db as _; -use ruff_db::diagnostic::{FileResolver, Input, UnifiedFile}; +use ruff_db::diagnostic::{Annotation, Diagnostic, Span}; use ruff_db::files::{File, FileRootKind, system_path_to_file}; use ruff_db::source::source_text; use ruff_db::system::{DbWithWritableSystem as _, SystemPathBuf}; use ruff_linter::source_kind::SourceKind; use ruff_linter::test::test_contents; -use ruff_notebook::NotebookIndex; use ruff_workspace::configuration::Configuration; use ruff_workspace::options::Options; @@ -100,8 +99,6 @@ fn run_test( // Edits for updating changed inline snapshots. let mut markdown_edits = vec![]; - let resolver = RuffResolver(db); - let mut panic_info = None; let failures: Failures = test_files @@ -123,7 +120,7 @@ fn run_test( test_file, ); - let diagnostics = match mdtest_result { + let mut diagnostics = match mdtest_result { Ok(diagnostics) => diagnostics, Err(failures) => { if test.should_expect_panic().is_ok() { @@ -134,11 +131,12 @@ fn run_test( return Some(failures.into_file_failures(db, "run mdtest", None)); } }; + normalize_diagnostics(test_file.file, &mut diagnostics); let failure = match matcher::match_file(db, test_file.file, &diagnostics).and_then( |inline_diagnostics| { mdtest::validate_inline_snapshot( - &resolver, + db, "ruff", test_file, &inline_diagnostics, @@ -177,30 +175,22 @@ fn run_test( } } -/// Wrap the db to avoid panicking when provided a Ruff file like the blanket `FileResolver` -/// implementation. -struct RuffResolver<'a>(&'a Db); - -impl FileResolver for RuffResolver<'_> { - fn path(&self, file: File) -> &str { - self.0.path(file) - } - - fn input(&self, file: File) -> Input { - self.0.input(file) - } - - fn current_directory(&self) -> &std::path::Path { - self.0.current_directory() - } +fn normalize_diagnostics(file: File, diagnostics: &mut [Diagnostic]) { + for diagnostic in diagnostics { + for annotation in diagnostic.annotations_mut() { + normalize_annotation(file, annotation); + } - fn notebook_index(&self, _file: &UnifiedFile) -> Option<NotebookIndex> { - None + for sub_diagnostic in diagnostic.sub_diagnostics_mut() { + for annotation in sub_diagnostic.annotations_mut() { + normalize_annotation(file, annotation); + } + } } +} - fn is_notebook(&self, _file: &UnifiedFile) -> bool { - false - } +fn normalize_annotation(file: File, annotation: &mut Annotation) { + annotation.set_span(Span::from(file).with_optional_range(annotation.get_span().range())); } fn parse<'s>(

Applied the patch, this makes a lot of sense to me. Thanks!

MichaReiser · 2026-05-15T07:19:42Z

+        Self::default()
+    }
+
+    pub(crate) fn use_in_memory_system(&mut self) {


ty_mdtest supports using the os system for tests like these

ruff/crates/ty_test/src/config.rs

Lines 25 to 32 in ffa347b

pub(crate) log: Option<Log>,

pub(crate) analysis: Option<Analysis>,

/// The [`ruff_db::system::System`] to use for tests.

///

/// Defaults to the case-sensitive [`ruff_db::system::InMemorySystem`].

pub(crate) system: Option<SystemKind>,

It also supports enabling logging, which I often found very useful to debug test errors.

I don't think it's necessary to land this as part of this PR but it should also not be too hard, I think

MichaReiser · 2026-05-15T07:22:25Z

+/// Wrap the db to avoid panicking when provided a Ruff file like the blanket `FileResolver`
+/// implementation.
+struct RuffResolver<'a>(&'a Db);


What are cases where we provide a Ruff file? Does that mean, that the mdtest harness behaves slightly different than the Ruff CLI would?

As you noted above, the Ruff files come from the diagnostics, which trips these unimplemented! blocks:

ruff/crates/ruff_db/src/diagnostic/render.rs

Lines 875 to 892 in f495ff2

fn notebook_index(&self, file: &UnifiedFile) -> Option<NotebookIndex> {

match file {

UnifiedFile::Ty(file) => self

.input(*file)

.text

.as_notebook()

.map(Notebook::index)

.cloned(),

UnifiedFile::Ruff(_) => unimplemented!("Expected an interned ty file"),

}

}

fn is_notebook(&self, file: &UnifiedFile) -> bool {

match file {

UnifiedFile::Ty(file) => self.input(*file).text.as_notebook().is_some(),

UnifiedFile::Ruff(_) => unimplemented!("Expected an interned ty file"),

}

}

We don't support notebooks in mdtests anyway, so I was just trying to avoid calling into these.

I think your other suggestion about remapping the diagnostics will avoid this anyway, though!

Co-authored-by: Micha Reiser <micha@reiser.io>

MichaReiser

This is so cool! And thanks for taking the time to share the mdtest infra over just copy pasting it.

ntBre · 2026-05-18T13:28:05Z

Thank you! And thanks for all of the helpful reviews, as always 😄

## Summary This is a follow-up to astral-sh#24616 to share a bit more infrastructure between `ty_test` and the `ruff_test` crate introduced in astral-sh#24617. In particular, the primary `run` function has moved into `mdtest`, along with the `attempt_test`, `check_panic` and `snapshot_diagnostics` helpers. You should be able to review commit-by-commit, but the first commit is probably the most important to look at since it moves the `tracing` setup and inconsistency checks from `ty_test::run` into `ty_test::run_test`. I think that narrows the span of the `tracing` logs a bit, but the important section should still be covered. The other commits should be pretty straightforward and reviewable as one diff. ## Test Plan Existing tests, as well as rebasing astral-sh#24617 onto this branch.

Summary -- This PR adds the `ruff_test` crate, a parallel crate to `ty_test` for Ruff, to enable the new mdtests in `ruff_linter`. I opted to follow the `Db`-based structure of `ty_test` to simplify the integration, but we end up basically just unpacking the files from the `Db` to call the `test_contents` function from the linter. I also updated the section of the contributing guidelines on lint rule testing to suggest using mdtests. Test Plan -- I copied over the S704 tests into a new mdtest. I selected this rule because it had several separate fixture files that all used different settings and each of the fixtures was relatively short. I had initially reached for a more complicated rule (UP046), but this seemed less likely to clutter the diff while still exercising most mdtest features. --------- Co-authored-by: Micha Reiser <micha@reiser.io>

ntBre added the testing Related to testing Ruff itself label Apr 13, 2026

ntBre commented Apr 13, 2026

View reviewed changes

Comment thread crates/ruff_linter/tests/mdtest.rs Outdated

ntBre mentioned this pull request Apr 13, 2026

Factor out the mdtest crate #24616

Merged

MichaReiser reviewed Apr 14, 2026

View reviewed changes

Comment thread crates/ruff_linter/tests/mdtest.rs Outdated

ntBre force-pushed the brent/ruff-mdtests branch 2 times, most recently from 39890d5 to 7f6a3ca Compare April 15, 2026 13:07

Base automatically changed from brent/mdtest-crate to main April 16, 2026 14:32

ntBre force-pushed the brent/ruff-mdtests branch 3 times, most recently from 57c1643 to 83f7f97 Compare April 20, 2026 17:11

ntBre mentioned this pull request Apr 20, 2026

Factor out more mdtest helpers #24754

Merged

ntBre changed the base branch from main to brent/mdtest-run April 21, 2026 14:24

ntBre force-pushed the brent/ruff-mdtests branch 2 times, most recently from 41d0d57 to 0ad4abe Compare April 21, 2026 14:28

ntBre force-pushed the brent/mdtest-run branch from 9287424 to e3fb85f Compare April 22, 2026 15:10

ntBre force-pushed the brent/mdtest-run branch from 61c2a49 to ca5cf14 Compare May 11, 2026 19:05

Base automatically changed from brent/mdtest-run to main May 12, 2026 14:24

ntBre force-pushed the brent/ruff-mdtests branch 3 times, most recently from 9013549 to fb7599d Compare May 14, 2026 18:03

ntBre commented May 14, 2026

View reviewed changes

make ruff_linter::test::test_contents public

5befd93

ntBre force-pushed the brent/ruff-mdtests branch from fb7599d to 7d747b1 Compare May 14, 2026 19:31

ntBre added 3 commits May 14, 2026 16:14

add ruff_test and mdtest runner

a4c284b

move S704 tests to mdtest

095bf4e

update contributing docs to focus on mdtests

ffa347b

ntBre force-pushed the brent/ruff-mdtests branch from 7d747b1 to ffa347b Compare May 14, 2026 20:15

ntBre marked this pull request as ready for review May 14, 2026 21:05

MichaReiser reviewed May 15, 2026

View reviewed changes

ntBre and others added 11 commits May 15, 2026 13:21

use error annotations instead of all snapshots

a4df98f

prefer # error over # snapshot in docs too

385afb2

ruff_test -> ruff_mdtest

6b4a405

move mdtest runner from ruff_linter to ruff_mdtest

17280b2

drop unnecessary try_add_root call

4fb5771

test expect-panic

fa4afb4

normalize diagnostics instead of wrapping with RuffResolver

4d6e02e

Co-authored-by: Micha Reiser <micha@reiser.io>

extra whitespace

2bbce6a

include crates/ruff_linter/resources/mdtest in pre-commit checks

6873663

fix remaining whitespace issues

eb26145

only enable test-rules when testing

a7ea610

MichaReiser approved these changes May 16, 2026

View reviewed changes

ntBre merged commit 505a51c into main May 18, 2026
59 checks passed

ntBre deleted the brent/ruff-mdtests branch May 18, 2026 13:28

	pub fn is_package(path: &Path, namespace_packages: &[PathBuf]) -> bool {
	namespace_packages
	.iter()
	.any(\|namespace_package\| path.starts_with(namespace_package))
	\|\| path.join("__init__.py").is_file()
	}

	pub(crate) log: Option<Log>,

	pub(crate) analysis: Option<Analysis>,

	/// The [`ruff_db::system::System`] to use for tests.
	///
	/// Defaults to the case-sensitive [`ruff_db::system::InMemorySystem`].
	pub(crate) system: Option<SystemKind>,

	fn notebook_index(&self, file: &UnifiedFile) -> Option<NotebookIndex> {
	match file {
	UnifiedFile::Ty(file) => self
	.input(*file)
	.text
	.as_notebook()
	.map(Notebook::index)
	.cloned(),
	UnifiedFile::Ruff(_) => unimplemented!("Expected an interned ty file"),
	}
	}

	fn is_notebook(&self, file: &UnifiedFile) -> bool {
	match file {
	UnifiedFile::Ty(file) => self.input(*file).text.as_notebook().is_some(),
	UnifiedFile::Ruff(_) => unimplemented!("Expected an interned ty file"),
	}
	}

Uh oh!

Conversation

ntBre commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Plan

Uh oh!

astral-sh-bot Bot commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Typing conformance results

No changes detected ✅

Uh oh!

astral-sh-bot Bot commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Memory usage report

Uh oh!

astral-sh-bot Bot commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ecosystem-analyzer results

Uh oh!

astral-sh-bot Bot commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ruff-ecosystem results

Linter (stable)

Linter (preview)

Formatter (stable)

Formatter (preview)

Uh oh!

Uh oh!

Uh oh!

ntBre May 14, 2026

Choose a reason for hiding this comment

Uh oh!

MichaReiser May 15, 2026

Choose a reason for hiding this comment

Uh oh!

ntBre May 15, 2026

Choose a reason for hiding this comment

Uh oh!

codspeed-hq Bot commented May 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

ntBre commented May 14, 2026

Uh oh!

MichaReiser left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

MichaReiser May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ntBre May 15, 2026

Choose a reason for hiding this comment

Uh oh!

MichaReiser May 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MichaReiser May 15, 2026

Choose a reason for hiding this comment

Uh oh!

ntBre May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MichaReiser left a comment

Choose a reason for hiding this comment

Uh oh!

ntBre commented May 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

ntBre commented Apr 13, 2026 •

edited

Loading

astral-sh-bot Bot commented Apr 13, 2026 •

edited

Loading

astral-sh-bot Bot commented Apr 13, 2026 •

edited

Loading

astral-sh-bot Bot commented Apr 13, 2026 •

edited

Loading

`ecosystem-analyzer` results

astral-sh-bot Bot commented Apr 13, 2026 •

edited

Loading

`ruff-ecosystem` results

codspeed-hq Bot commented May 14, 2026 •

edited

Loading

MichaReiser May 15, 2026 •

edited

Loading

ntBre May 15, 2026 •

edited

Loading