The problem:
JUnit breaks when it reads an XML generated by pytest if plugins make use of record-property. This behavior happens with newer versions of hypothesis (HypothesisWorks/hypothesis#1935).
[xUnit] [ERROR] - The result file '/somewhere/tests/pytests.xml' for the metric 'JUnit' is not valid. The result file has been skipped.
In fact, as already mentioned in #1126 (comment), record-property is adding <properties> inside <testcase> which seems to be wrong (it should be inside <testsuite>). See: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd .
It happens with all junit families.
Reproducing:
$ pip list
Package Version
-------------- --------
apipkg 1.5
atomicwrites 1.3.0
attrs 19.1.0
certifi 2019.3.9
execnet 1.6.0
hypothesis 4.18.3
more-itertools 4.3.0
pip 19.1
pluggy 0.9.0
py 1.8.0
pytest 4.4.1
pytest-forked 1.0.2
pytest-xdist 1.28.0
setuptools 41.0.1
six 1.12.0
wheel 0.33.1
test_xml_generation.py
from hypothesis import given, strategies
@given(x=strategies.integers(1, 10,))
def test_xml_generation(x):
assert 1 <= x <= 10
$ pytest --junitxml=report.xml
report.xml
<?xml version="1.0" encoding="utf-8"?>
<testsuite errors="0" failures="0" name="pytest" skipped="0" tests="1" time="0.211">
<testcase classname="test_xml_generation" file="test_xml_generation.py" line="3" name="test_xml_generation"
time="0.074">
<properties>
<property name="hypothesis-stats"
value="['test_xml_generation.py::test_xml_generation:', '', ' - 100 passing examples, 0 failing examples, 0 invalid examples', ' - Typical runtimes: < 1ms', ' - Fraction of time spent in data generation: ~ 49%', ' - Stopped because settings.max_examples=100', '']"/>
</properties>
</testcase>
</testsuite>
I was trying to create a PR to fix this, but when I saw https://github.com/pytest-dev/pytest/blob/7dcd9bf5add337686ec6f2ee81b24e8424319dba/src/_pytest/junitxml.py code I realized that what is needed to do could have more implications that I though. I think that nobody uses this feature with JUnit (as it breaks) and removing that is something to think about.
The problem:
JUnit breaks when it reads an XML generated by pytest if plugins make use of
record-property. This behavior happens with newer versions of hypothesis (HypothesisWorks/hypothesis#1935).In fact, as already mentioned in #1126 (comment),
record-propertyis adding<properties>inside<testcase>which seems to be wrong (it should be inside<testsuite>). See: https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd .It happens with all junit families.
Reproducing:
test_xml_generation.pyreport.xmlI was trying to create a PR to fix this, but when I saw https://github.com/pytest-dev/pytest/blob/7dcd9bf5add337686ec6f2ee81b24e8424319dba/src/_pytest/junitxml.py code I realized that what is needed to do could have more implications that I though. I think that nobody uses this feature with JUnit (as it breaks) and removing that is something to think about.