Tell us about your runtime:

• QUnit version: 2.4.0
• What environment are you running QUnit in? (e.g., browser, Node): Node 8.6.0, Chrome 61.
• How are you running QUnit? (e.g., script, testem, Grunt): custom npm script

What are you trying to do?

I'm trying to use the step(..) / verifySteps(..) API, but I had a failure related to the number of expected assertions.

The documentation for this feature doesn't mention the impact on expected assertion count at all. So at a minimum, the docs need to be updated. But I also think the current behavior is counter-intuitive.

Code that reproduces the problem:

QUnit.test( "verify steps", function test(assert){
    assert.step( "do stuff 1" );
    assert.step( "do stuff 2" );

    assert.expect( 1 );  // or 2
    assert.verifySteps( ["do stuff 1","do stuff 2"] );
} );

// test fails:
//   Expected 1 assertions, but 3 were run

What did you expect to happen?

I assumed the number of assertions to expect would either correlate to the number of step(..) calls (2), OR to the number of verifySteps(..) calls (1).

What actually happened?

The failure error message says 3, not 1 or 2. So clearly the counter is incrementing with both the step(..) calls and the verifySteps(..) calls.

This feels very strange and surprising to me. Even if it had been documented that way, I think it leads to more confusion with test authoring. It should consider only one or the other, not both.

1. The argument for using only the step(..) calls in the counter:

   step(..) is kinda like an ok( true, .. ) call, so each time step(..) happens, make sure it's counted. If you know there are 5 steps to some algorithm, it makes intuitive sense to increase your expected count by 5.

   Moreover, it doesn't make sense to include verifySteps(..) in this count in the same way that the call to assert.expect(..) doesn't itself get included in the count.

2. The argument for using only the verifySteps(..) calls in the counter:

   step(..) is conceptually like just pushing an entry into an array. We haven't verified anything yet. There's no true or false passing that's happening at that point. The assertion doesn't happen until you call the verifySteps(..) call, which is conceptually like calling deepEqual(..) on the array that step(..) is pushing into.

   Usually you make lots of step(..) calls, but only one verifySteps(..) call. So the counter should only increment once with that call, regardless of how many step(..)s there are.

   Moreover, you're already implicitly counting the number of step(..) calls that happened, because the only way entires get into the internal array you're comparing to is by calling step(..), so the verifySteps(..) is already checking that the number -- not just the order! -- of step(..) calls is correct. No need for that to be included in the count.

I think either of these lines of argument is compelling. Personally, I think (2) is how my brain works. The style of how I lay out my tests, my assertions are all collected together at the end, so I expect to be able to see the same number of assertions listed, line-by-line, as what I pass to expect(..). If step(..) is included in that count, I have to look all over other parts of the test code to verify that my number matches. This is clunky.