I note that there appears to be an alternative design for testing full diagnostic output in the mdtest doc:

Lines 346 to 381 in 15dd3b5

    
           ### Asserting on full diagnostic output 
        
           The inline comment diagnostic assertions are useful for making quick, readable assertions about 
        
           diagnostics in a particular location. But sometimes we will want to assert on the full diagnostic 
        
           output of checking an embedded Python file. Or sometimes (see “incremental tests” below) we will 
        
           want to assert on diagnostics in a file, without impacting the contents of that file by changing a 
        
           comment in it. In these cases, a Python code block in a test could be followed by a fenced code 
        
           block with language `output`; this would contain the full diagnostic output for the preceding test 
        
           file: 
        
           ````markdown 
        
           # full output 
        
           ```py 
        
           x = 1 
        
           reveal_type(x) 
        
           ``` 
        
           This is just an example, not a proposal that red-knot would ever actually output diagnostics in 
        
           precisely this format: 
        
           ```output 
        
           mdtest_snippet__1.py, line 1, col 1: revealed type is 'Literal[1]' 
        
           ``` 
        
           ```` 
        
           We will want to build tooling to automatically capture and update these “full diagnostic output” 
        
           blocks, when tests are run in an update-output mode (probably specified by an environment variable.) 
        
           By default, an `output` block will specify diagnostic output for the file 
        
           `<workspace-root>/mdtest_snippet__1.py`. An `output` block can be prefixed by a 
        
           <code>`&lt;path>`:</code> label as usual, to explicitly specify the Python file for which it asserts 
        
           diagnostic output. 
        
           It is an error for an `output` block to exist, if there is no `py` or `python` block in the same 
        
           test for the same file path.

I think the output blocks suggested in that doc appeal to me a bit more. But I confess that I think I don't fully understand @MichaReiser's idea here. Could you say a bit more about how you envision this working?

	### Asserting on full diagnostic output

	The inline comment diagnostic assertions are useful for making quick, readable assertions about
	diagnostics in a particular location. But sometimes we will want to assert on the full diagnostic
	output of checking an embedded Python file. Or sometimes (see “incremental tests” below) we will
	want to assert on diagnostics in a file, without impacting the contents of that file by changing a
	comment in it. In these cases, a Python code block in a test could be followed by a fenced code
	block with language `output`; this would contain the full diagnostic output for the preceding test
	file:

	````markdown
	# full output

	```py
	x = 1
	reveal_type(x)
	```

	This is just an example, not a proposal that red-knot would ever actually output diagnostics in
	precisely this format:

	```output
	mdtest_snippet__1.py, line 1, col 1: revealed type is 'Literal[1]'
	```
	````

	We will want to build tooling to automatically capture and update these “full diagnostic output”
	blocks, when tests are run in an update-output mode (probably specified by an environment variable.)

	By default, an `output` block will specify diagnostic output for the file
	`<workspace-root>/mdtest_snippet__1.py`. An `output` block can be prefixed by a
	<code>`<path>`:</code> label as usual, to explicitly specify the Python file for which it asserts
	diagnostic output.

	It is an error for an `output` block to exist, if there is no `py` or `python` block in the same
	test for the same file path.

[red-knot] Assert full diagnostic output in mdtests #15867

Description

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions