Investigation of test results in XML vs Console#313
Investigation of test results in XML vs Console#313im-konge wants to merge 1 commit intoapache:masterfrom
Conversation
Signed-off-by: Lukas Kral <lukywill16@gmail.com>
|
Hi @qingzhouluo |
|
We include the "number of run" information to give users some ideas about how many total runs there are, so they can better understand the flakiness of their test suite. For example, if I have one single test run 100 times, and it failed once, "test run: 100, failure: 1" will help me to understand how serve the issue is. Adding some extra information about the total number of test count also sounds good to me, thanks. |
|
Hello! Is there any progress on this issue please? I have opened https://issues.apache.org/jira/browse/SUREFIRE-1903 |
Signed-off-by: Lukas Kral lukywill16@gmail.com
This is more discussion or issue description than real PR with some changes.
How you can see in this code, I changed a little bit asserts in rerun suite. The main problem that I found is (from my POV) wrong counting of tests count. Or not displaying the real tests run count.
When we take a suite (class) where we have 3 tests, that 2 of them will run twice (one rerun), the count will be like this (how is it in current tests) :
Tests run: 5, errors: 1, failures: 1, skipped: 0 ...-> but this is not really the truth. It can be problem in translation, but I would like to know how many tests (not with reruns) I ran -> 3. Then if failures > 0 the test should be flaky etc.So I assume to add something like
tests count: 3, tests run: 5 ...to the console reporter andtests: 3, runs: 5to the XML reporter.Second problem that I found is that results from XML file != results from console reporter. I have some examples when I was talking to @Tibor17.
WDYT?