Implement support for todo tests and revamp reporting logic#137
Merged
vladikoff merged 2 commits intogruntjs:masterfrom Feb 7, 2017
Merged
Implement support for todo tests and revamp reporting logic#137vladikoff merged 2 commits intogruntjs:masterfrom
vladikoff merged 2 commits intogruntjs:masterfrom
Conversation
gibson042
reviewed
Feb 5, 2017
| phantomjs.on('qunit.testDone', function(name, failed, passed, total, runtime, skipped, todo) { | ||
| if (skipped) { | ||
| currentStatus.skipped++; | ||
| } else if (failed && !todo) { |
In order to properly support todo tests, the status of the test run should be determined by status of complete tests and not individual assertions.
Contributor
Author
|
Updated. |
Krinkle
added a commit
to qunitjs/qunit
that referenced
this pull request
Apr 18, 2022
== Integrate js-reporters spec == Keep the link to the js-reporters project for now, but no longer rely on it for discovery. The spec was written for implementors, which makes it a suboptimal reading experience for end-users. It also suffered from numerous nullable properties that in practice with QUnit are either never null or always null. In our docs, I've left these off or explicitly described them in their reliable non-null form to avoid any doubt or confusion over how or why they behave a certain way. Note that as of js-reporters 2.0, much of the spec has been removed which QUnit currently still implements for compatibility (such as upfront recursively declared child suites). These have not been documented and will be removed in QUnit 3.0. Ref https://github.com/js-reporters/js-reporters/releases/tag/v2.0.0. The integration of the spec, rather than improving or creating a page in the js-reporters repo for end-users, is motivated by qunitjs/js-reporters#133, in which I conclude that for now it is a better use of our effort for any re-usable cross-framework reporters to consume TAP rather than tight runtime coupling. This also has the benefit of not pushing down the "serialize actual value" and "format a diff" problems down to end-consumers, which in practice are often done poorly or not at all. == Document unofficial `done` details == For `done()`, the `details.modules` property was added by commit 168b048 in QUnit 1.16 (2014) for internal use by HTML Reporter. It was never documented, however. It originally came with a `test` property as well, but that hasn't been in use since commit 43a3d87 in QUnit 2.7 (2018). Keep this for now since it's not adding delay or complexity, but I've left a note to remove this in QUnit 3.0. Document the rest as now-officially supported, with retroactive changelog. Also document the oft-overlooked caveat with the legacy assertion counts exposed from `QUnit.done()`. I believe the community has large moved away from using the data provided by this callback. For example, gruntjs/grunt-contrib-qunit#137. Recommend `on('runEnd')` instead.
Krinkle
added a commit
to qunitjs/qunit
that referenced
this pull request
Apr 18, 2022
== Integrate js-reporters spec == Keep the link to the js-reporters project for now, but no longer rely on it for discovery. The spec was written for implementors, which makes it a suboptimal reading experience for end-users. It also suffered from numerous nullable properties that in practice with QUnit are either never null or always null. In our docs, I've left these off or explicitly described them in their reliable non-null form to avoid any doubt or confusion over how or why they behave a certain way. Note that as of js-reporters 2.0, much of the spec has been removed which QUnit currently still implements for compatibility (such as upfront recursively declared child suites). These have not been documented and will be removed in QUnit 3.0. Ref https://github.com/js-reporters/js-reporters/releases/tag/v2.0.0. The integration of the spec, rather than improving or creating a page in the js-reporters repo for end-users, is motivated by qunitjs/js-reporters#133, in which I conclude that for now it is a better use of our effort for any re-usable cross-framework reporters to consume TAP rather than tight runtime coupling. This also has the benefit of not pushing down the "serialize actual value" and "format a diff" problems down to end-consumers, which in practice are often done poorly or not at all. == Document unofficial `done` details == For `done()`, the `details.modules` property was added by commit 168b048 in QUnit 1.16 (2014) for internal use by HTML Reporter. It was never documented, however. It originally came with a `test` property as well, but that hasn't been in use since commit 43a3d87 in QUnit 2.7 (2018). Keep this for now since it's not adding delay or complexity, but I've left a note to remove this in QUnit 3.0. Document the rest as now-officially supported, with retroactive changelog. Also document the oft-overlooked caveat with the legacy assertion counts exposed from `QUnit.done()`. I believe the community has large moved away from using the data provided by this callback. For example, gruntjs/grunt-contrib-qunit#137. Recommend `on('runEnd')` instead.
Krinkle
added a commit
that referenced
this pull request
Apr 24, 2022
== Why == * Higher level API, exists for this purpose, thus resulting in simpler code that has to know less about QUnit internals. * Solves a number of problems, including two issues that would make CI on this repo fail if we updated its devDependencies to the latest qunit as-is. == Change == Remove all manual tracking of `QUnit.log()` and custom interpreting of results. Simply listen to `QUnit.on()` for "testStart", "testEnd" and "runEnd", and then report their counts, details, and boolean status as-is. The main observable change is that the printed success line is now one line instead of two: Before: > 2 tests completed with 0 failed, 0 skipped, and 0 todo. > 2 assertions (in 9ms), passed: 2, failed: 0 After: > 2 tests completed in 9ms, with 0 failed, 0 skipped, and 0 todo. This is the same text as displayed in the browser. == Details == The task previously tracked individually logged assertions, decided if it's a failure, and filtered out expected failures in a "todo" test from actually unexpected failures. This resulted in the odd situation where `QUnit.done()` is given a count but the task did not use it to decide whether the grunt task is a success, because this would not account for "todo" tests, and thus in #137 the logic grew even further. In QUnit 2.17 the reporting of global failures from window.onerror changed and the logic here was now stale and no longer reporting it correctly. Also, the qunit_noglobals.html case was wrongly counted as a failing test because the logic checked `failed === 0 && passed === total` based on raw assertion count, which again, does not account for "todo" tests. The `qunit_noglobals.html` is actually a passing test when opened in the browser, but the local Gruntfile logic treated it as a failure. Both of these two problems are naturally fixed by this patch, and so the next commit that updates QUnit in this repo's CI to the latest will continue to pass instead of having these two failures.
Krinkle
added a commit
that referenced
this pull request
Apr 25, 2022
== Why == * Higher level API, exists for this purpose, thus resulting in simpler code that has to know less about QUnit internals. * Solves a number of problems, including two issues that would make CI on this repo fail if we updated its devDependencies to the latest qunit as-is. == Change == Remove all manual tracking of `QUnit.log()` and custom interpreting of results. Simply listen to `QUnit.on()` for "testStart", "testEnd" and "runEnd", and then report their counts, details, and boolean status as-is. The main observable change is that the printed success line is now one line instead of two: Before: > 2 tests completed with 0 failed, 0 skipped, and 0 todo. > 2 assertions (in 9ms), passed: 2, failed: 0 After: > 2 tests completed in 9ms, with 0 failed, 0 skipped, and 0 todo. This is the same text as displayed in the browser. == Details == The task previously tracked individually logged assertions, decided if it's a failure, and filtered out expected failures in a "todo" test from actually unexpected failures. This resulted in the odd situation where `QUnit.done()` is given a count but the task did not use it to decide whether the grunt task is a success, because this would not account for "todo" tests, and thus in #137 the logic grew even further. In QUnit 2.17 the reporting of global failures from window.onerror changed and the logic here was now stale and no longer reporting it correctly. Also, the qunit_noglobals.html case was wrongly counted as a failing test because the logic checked `failed === 0 && passed === total` based on raw assertion count, which again, does not account for "todo" tests. The `qunit_noglobals.html` is actually a passing test when opened in the browser, but the local Gruntfile logic treated it as a failure. Both of these two problems are naturally fixed by this patch, and so the next commit that updates QUnit in this repo's CI to the latest will continue to pass instead of having these two failures.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In order to support
QUnit.todo(), the reporting logic for this package needed to be revamped.Previously, pass/fail was determined by the presence of individual failing assertions. With
todo, it is possible to have failing assertions without having a failing test suite. Thus, the logic for deciding whether or not a suite has failed needs to deal with the status of complete tests rather than individual assertions. That is what the bulk of this PR does.It also changes the summary text to be of a standard format like so:
Previously, you could get 1 of 3 different variation depending on the combination of passed/failed assertions your suite had.
cc @leobalter