Skip to content

Group warnings by message instead of by test id#4665

Merged
nicoddemus merged 1 commit into
pytest-dev:masterfrom
nicoddemus:group-warnings-by-message
Jan 24, 2019
Merged

Group warnings by message instead of by test id#4665
nicoddemus merged 1 commit into
pytest-dev:masterfrom
nicoddemus:group-warnings-by-message

Conversation

@nicoddemus

@nicoddemus nicoddemus commented Jan 22, 2019

Copy link
Copy Markdown
Member

Currently pytest shows warnings grouped by node id. What usually happens is that in large codebases the offending code ends up called from multiple tests, generating a lot of output which usually points to the same code over and over again.

For example:

import warnings
import pytest

def func():
    warnings.warn(UserWarning("foo"))

@pytest.mark.parametrize("i", range(5))
def test_foo(i):
    func()

def test_bar():
    func()

Currently generates:

========================= warnings summary ==========================
test-warning-grouping.py::test_foo[0]
  c:\pytest\.tmp\test-warning-grouping.py:6: UserWarning: foo
    warnings.warn(UserWarning("foo"))

test-warning-grouping.py::test_foo[1]
  c:\pytest\.tmp\test-warning-grouping.py:6: UserWarning: foo
    warnings.warn(UserWarning("foo"))

test-warning-grouping.py::test_foo[2]
  c:\pytest\.tmp\test-warning-grouping.py:6: UserWarning: foo
    warnings.warn(UserWarning("foo"))

test-warning-grouping.py::test_foo[3]
  c:\pytest\.tmp\test-warning-grouping.py:6: UserWarning: foo
    warnings.warn(UserWarning("foo"))

test-warning-grouping.py::test_foo[4]
  c:\pytest\.tmp\test-warning-grouping.py:6: UserWarning: foo
    warnings.warn(UserWarning("foo"))

test-warning-grouping.py::test_bar
  c:\pytest\.tmp\test-warning-grouping.py:6: UserWarning: foo
    warnings.warn(UserWarning("foo"))

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=============== 6 passed, 6 warnings in 0.03 seconds ================

This gives the impression that the user has a ton of warnings to fix, even though they all point to the same location. This is even worse when there are multiple warnings involved, because it is hard to find the actual offending code in the flood of messages generated by pytest.

This is the output with this PR:

========================= warnings summary ==========================
test-warning-grouping.py::test_foo[0]
test-warning-grouping.py::test_foo[1]
test-warning-grouping.py::test_foo[2]
test-warning-grouping.py::test_foo[3]
test-warning-grouping.py::test_foo[4]
test-warning-grouping.py::test_bar
  c:\pytest\.tmp\test-warning-grouping.py:6: UserWarning: foo
    warnings.warn(UserWarning("foo"))

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=============== 6 passed, 6 warnings in 0.04 seconds ================

This makes it much easier to see that we only have to fix the warning in one place, instead of several like the previous output, while still showing all the affected tests.

This also fixes #4402, although not for the reasons discussed there.

@codecov

codecov Bot commented Jan 22, 2019

Copy link
Copy Markdown

Codecov Report

Merging #4665 into master will increase coverage by <.01%.
The diff coverage is 92.59%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #4665      +/-   ##
==========================================
+ Coverage   95.71%   95.71%   +<.01%     
==========================================
  Files         113      113              
  Lines       24882    24893      +11     
  Branches     2458     2458              
==========================================
+ Hits        23815    23826      +11     
  Misses        754      754              
  Partials      313      313
Flag Coverage Δ
#docs 29.56% <3.7%> (+0.06%) ⬆️
#doctesting 29.56% <3.7%> (+0.06%) ⬆️
#linting 29.56% <3.7%> (+0.06%) ⬆️
#linux 95.54% <92.59%> (+0.01%) ⬆️
#nobyte 92.32% <92.59%> (ø) ⬆️
#numpy 93.12% <92.59%> (-0.02%) ⬇️
#pexpect 42.05% <66.66%> (-0.03%) ⬇️
#py27 93.71% <92.59%> (+0.02%) ⬆️
#py34 91.83% <92.59%> (+0.07%) ⬆️
#py35 91.85% <92.59%> (+0.05%) ⬆️
#py36 91.87% <92.59%> (+0.07%) ⬆️
#py37 93.87% <92.59%> (+0.01%) ⬆️
#trial 93.12% <92.59%> (-0.02%) ⬇️
#windows 93.91% <92.59%> (ø) ⬆️
#xdist 93.76% <92.59%> (+0.01%) ⬆️
Impacted Files Coverage Δ
testing/test_warnings.py 98.9% <100%> (+0.04%) ⬆️
src/_pytest/terminal.py 91.81% <90%> (+0.05%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ba452db...9543d19. Read the comment docs.

1 similar comment
@codecov

codecov Bot commented Jan 22, 2019

Copy link
Copy Markdown

Codecov Report

Merging #4665 into master will increase coverage by <.01%.
The diff coverage is 92.59%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #4665      +/-   ##
==========================================
+ Coverage   95.71%   95.71%   +<.01%     
==========================================
  Files         113      113              
  Lines       24882    24893      +11     
  Branches     2458     2458              
==========================================
+ Hits        23815    23826      +11     
  Misses        754      754              
  Partials      313      313
Flag Coverage Δ
#docs 29.56% <3.7%> (+0.06%) ⬆️
#doctesting 29.56% <3.7%> (+0.06%) ⬆️
#linting 29.56% <3.7%> (+0.06%) ⬆️
#linux 95.54% <92.59%> (+0.01%) ⬆️
#nobyte 92.32% <92.59%> (ø) ⬆️
#numpy 93.12% <92.59%> (-0.02%) ⬇️
#pexpect 42.05% <66.66%> (-0.03%) ⬇️
#py27 93.71% <92.59%> (+0.02%) ⬆️
#py34 91.83% <92.59%> (+0.07%) ⬆️
#py35 91.85% <92.59%> (+0.05%) ⬆️
#py36 91.87% <92.59%> (+0.07%) ⬆️
#py37 93.87% <92.59%> (+0.01%) ⬆️
#trial 93.12% <92.59%> (-0.02%) ⬇️
#windows 93.91% <92.59%> (ø) ⬆️
#xdist 93.76% <92.59%> (+0.01%) ⬆️
Impacted Files Coverage Δ
testing/test_warnings.py 98.9% <100%> (+0.04%) ⬆️
src/_pytest/terminal.py 91.81% <90%> (+0.05%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ba452db...9543d19. Read the comment docs.

RonnyPfannschmidt
RonnyPfannschmidt approved these changes Jan 23, 2019
View reviewed changes

@RonnyPfannschmidt RonnyPfannschmidt left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lovely, just awesome, now i know what i felt was missing but couldnt quite figure

@nicoddemus nicoddemus merged commit 7ddfc04 into pytest-dev:master Jan 24, 2019
@nicoddemus nicoddemus deleted the group-warnings-by-message branch January 24, 2019 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@RonnyPfannschmidt RonnyPfannschmidt RonnyPfannschmidt approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Warnings via parametrized tests are not displayed only once

2 participants

@nicoddemus @RonnyPfannschmidt