Skip to content

Implement support for todo tests and revamp reporting logic#137

Merged
vladikoff merged 2 commits intogruntjs:masterfrom
trentmwillis:todo
Feb 7, 2017
Merged

Implement support for todo tests and revamp reporting logic#137
vladikoff merged 2 commits intogruntjs:masterfrom
trentmwillis:todo

Conversation

@trentmwillis
Copy link
Copy Markdown
Contributor

@trentmwillis trentmwillis commented Feb 5, 2017

In order to support QUnit.todo(), the reporting logic for this package needed to be revamped.

Previously, pass/fail was determined by the presence of individual failing assertions. With todo, it is possible to have failing assertions without having a failing test suite. Thus, the logic for deciding whether or not a suite has failed needs to deal with the status of complete tests rather than individual assertions. That is what the bulk of this PR does.

It also changes the summary text to be of a standard format like so:

>> 210 tests completed with 0 failed, 6 skipped, and 1 todo. 
>> 682 assertions (in 2638ms), passed: 681, failed: 1

Previously, you could get 1 of 3 different variation depending on the combination of passed/failed assertions your suite had.

cc @leobalter

@jsf-clabot
Copy link
Copy Markdown

jsf-clabot commented Feb 5, 2017

CLA assistant check
All committers have signed the CLA.

Comment thread tasks/qunit.js Outdated
phantomjs.on('qunit.testDone', function(name, failed, passed, total, runtime, skipped, todo) {
if (skipped) {
currentStatus.skipped++;
} else if (failed && !todo) {
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In order to properly support todo tests, the status of the test run
should be determined by status of complete tests and not individual
assertions.
@trentmwillis
Copy link
Copy Markdown
Contributor Author

Updated.

Copy link
Copy Markdown

@gibson042 gibson042 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vladikoff vladikoff merged commit 347c126 into gruntjs:master Feb 7, 2017
Krinkle added a commit to qunitjs/qunit that referenced this pull request Apr 18, 2022
== Integrate js-reporters spec ==

Keep the link to the js-reporters project for now, but no longer rely on it for
discovery. The spec was written for implementors, which makes it a suboptimal
reading experience for end-users. It also suffered from numerous nullable
properties that in practice with QUnit are either never null or always null.

In our docs, I've left these off or explicitly described them in their
reliable non-null form to avoid any doubt or confusion over how or why
they behave a certain way.

Note that as of js-reporters 2.0, much of the spec has been removed which
QUnit currently still implements for compatibility (such as upfront recursively
declared child suites). These have not been documented and will be removed
in QUnit 3.0.

Ref https://github.com/js-reporters/js-reporters/releases/tag/v2.0.0.

The integration of the spec, rather than improving or creating a page in the
js-reporters repo for end-users, is motivated by
qunitjs/js-reporters#133, in which I conclude
that for now it is a better use of our effort for any re-usable cross-framework
reporters to consume TAP rather than tight runtime coupling. This also has the
benefit of not pushing down the "serialize actual value" and "format a diff"
problems down to end-consumers, which in practice are often done poorly or not
at all.

== Document unofficial `done` details ==

For `done()`, the `details.modules` property was added by commit 168b048 in
QUnit 1.16 (2014) for internal use by HTML Reporter.

It was never documented, however. It originally came with a `test` property as
well, but that hasn't been in use since commit 43a3d87 in QUnit 2.7 (2018).
Keep this for now since it's not adding delay or complexity, but I've left a
note to remove this in QUnit 3.0.

Document the rest as now-officially supported, with retroactive changelog.

Also document the oft-overlooked caveat with the legacy assertion counts exposed
from `QUnit.done()`. I believe the community has large moved away from using
the data provided by this callback.
For example, gruntjs/grunt-contrib-qunit#137.

Recommend `on('runEnd')` instead.
Krinkle added a commit to qunitjs/qunit that referenced this pull request Apr 18, 2022
== Integrate js-reporters spec ==

Keep the link to the js-reporters project for now, but no longer rely on it for
discovery. The spec was written for implementors, which makes it a suboptimal
reading experience for end-users. It also suffered from numerous nullable
properties that in practice with QUnit are either never null or always null.

In our docs, I've left these off or explicitly described them in their
reliable non-null form to avoid any doubt or confusion over how or why
they behave a certain way.

Note that as of js-reporters 2.0, much of the spec has been removed which
QUnit currently still implements for compatibility (such as upfront recursively
declared child suites). These have not been documented and will be removed
in QUnit 3.0.

Ref https://github.com/js-reporters/js-reporters/releases/tag/v2.0.0.

The integration of the spec, rather than improving or creating a page in the
js-reporters repo for end-users, is motivated by
qunitjs/js-reporters#133, in which I conclude
that for now it is a better use of our effort for any re-usable cross-framework
reporters to consume TAP rather than tight runtime coupling. This also has the
benefit of not pushing down the "serialize actual value" and "format a diff"
problems down to end-consumers, which in practice are often done poorly or not
at all.

== Document unofficial `done` details ==

For `done()`, the `details.modules` property was added by commit 168b048 in
QUnit 1.16 (2014) for internal use by HTML Reporter.

It was never documented, however. It originally came with a `test` property as
well, but that hasn't been in use since commit 43a3d87 in QUnit 2.7 (2018).
Keep this for now since it's not adding delay or complexity, but I've left a
note to remove this in QUnit 3.0.

Document the rest as now-officially supported, with retroactive changelog.

Also document the oft-overlooked caveat with the legacy assertion counts exposed
from `QUnit.done()`. I believe the community has large moved away from using
the data provided by this callback.
For example, gruntjs/grunt-contrib-qunit#137.

Recommend `on('runEnd')` instead.
Krinkle added a commit that referenced this pull request Apr 24, 2022
== Why ==

* Higher level API, exists for this purpose, thus resulting in simpler code
  that has to know less about QUnit internals.

* Solves a number of problems, including two issues that would make CI on this
  repo fail if we updated its devDependencies to the latest qunit as-is.

== Change ==

Remove all manual tracking of `QUnit.log()` and custom interpreting of results.
Simply listen to `QUnit.on()` for "testStart", "testEnd" and "runEnd", and
then report their counts, details, and boolean status as-is.

The main observable change is that the printed success line is now one line
instead of two:

Before:
> 2 tests completed with 0 failed, 0 skipped, and 0 todo.
> 2 assertions (in 9ms), passed: 2, failed: 0

After:
> 2 tests completed in 9ms, with 0 failed, 0 skipped, and 0 todo.

This is the same text as displayed in the browser.

== Details ==

The task previously tracked individually logged assertions, decided if it's a
failure, and filtered out expected failures in a "todo" test from actually
unexpected failures.

This resulted in the odd situation where `QUnit.done()` is given a count but
the task did not use it to decide whether the grunt task is a success, because
this would not account for "todo" tests, and thus in
#137 the logic grew even
further. In QUnit 2.17 the reporting of global failures from window.onerror
changed and the logic here was now stale and no longer reporting it correctly.

Also, the qunit_noglobals.html case was wrongly counted as a failing test
because the logic checked `failed === 0 && passed === total` based on raw
assertion count, which again, does not account for "todo" tests.
The `qunit_noglobals.html` is actually a passing test when opened in the
browser, but the local Gruntfile logic treated it as a failure.

Both of these two problems are naturally fixed by this patch, and so the next
commit that updates QUnit in this repo's CI to the latest will continue to pass
instead of having these two failures.
Krinkle added a commit that referenced this pull request Apr 25, 2022
== Why ==

* Higher level API, exists for this purpose, thus resulting in simpler code
  that has to know less about QUnit internals.

* Solves a number of problems, including two issues that would make CI on this
  repo fail if we updated its devDependencies to the latest qunit as-is.

== Change ==

Remove all manual tracking of `QUnit.log()` and custom interpreting of results.
Simply listen to `QUnit.on()` for "testStart", "testEnd" and "runEnd", and
then report their counts, details, and boolean status as-is.

The main observable change is that the printed success line is now one line
instead of two:

Before:
> 2 tests completed with 0 failed, 0 skipped, and 0 todo.
> 2 assertions (in 9ms), passed: 2, failed: 0

After:
> 2 tests completed in 9ms, with 0 failed, 0 skipped, and 0 todo.

This is the same text as displayed in the browser.

== Details ==

The task previously tracked individually logged assertions, decided if it's a
failure, and filtered out expected failures in a "todo" test from actually
unexpected failures.

This resulted in the odd situation where `QUnit.done()` is given a count but
the task did not use it to decide whether the grunt task is a success, because
this would not account for "todo" tests, and thus in
#137 the logic grew even
further. In QUnit 2.17 the reporting of global failures from window.onerror
changed and the logic here was now stale and no longer reporting it correctly.

Also, the qunit_noglobals.html case was wrongly counted as a failing test
because the logic checked `failed === 0 && passed === total` based on raw
assertion count, which again, does not account for "todo" tests.
The `qunit_noglobals.html` is actually a passing test when opened in the
browser, but the local Gruntfile logic treated it as a failure.

Both of these two problems are naturally fixed by this patch, and so the next
commit that updates QUnit in this repo's CI to the latest will continue to pass
instead of having these two failures.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants