Skip to content

[🐛 Bug]: wdio-runner sends wrong exitCode when cucumber Feature fails in Before hook #10619

New issue
New issue
Closed
#10693
Closed
[🐛 Bug]: wdio-runner sends wrong exitCode when cucumber Feature fails in Before hook#10619
#10693
Assignees
christian-bromann
Labels
Bug 🐛
@Ernst-D

Description

@Ernst-D
Ernst-D
opened on Jun 22, 2023

Have you read the Contributing Guidelines on issues?

WebdriverIO Version

latest

Node.js Version

18.12.1

Mode

WDIO Testrunner

Which capabilities are you using?

{
 'goog:chromeOptions': {
            args: ['headless', 'disable-gpu']
        },
        browserName: 'chrome' 
}

What happened?

When I use webdriverio with cucumber adapter and Feature file fails in Before hook - wdio runner sends wrong result which leads to wrong exitCode when process completes.

I've created a branch in repo which help to reproduce this behaviour. Basically, after running npm run wdio:hook-error - output should have 2 failed specs, but it has 1.

If you try to debug wdio.conf, especially hooks afterScenario and after - you'll see that afterScenario sends correct result when test fails in Before hook, but after sends wrong one.

What is your expected behavior?

When Feature file fails in before hook - wdio runner should send exitCode 1 and consider file to be failed.

How to reproduce the bug.

Please, check out this branch in next repo and run npm run wdio:hook-error

Relevant log output

ernsts-MacBook-Pro:wdio-cucumber-framework-bug-reproduce ernst$ npm run wdio:hook-error

> my-new-project@1.0.0 wdio:hook-error
> npm run wdio -- --spec features/before-hook-fail/**/*.feature


> my-new-project@1.0.0 wdio
> wdio run ./wdio.conf.ts --spec features/before-hook-fail/**/*.feature


Execution of 2 workers started at 2023-06-22T19:05:24.756Z

2023-06-22T19:05:24.767Z DEBUG @wdio/utils:initialiseServices: initialise service "chromedriver" as NPM package
2023-06-22T19:05:24.788Z INFO chromedriver: Initiate Chromedriver Launcher (v8.1.1)
2023-06-22T19:05:24.788Z INFO @wdio/cli:launcher: Run onPrepare hook
2023-06-22T19:05:24.790Z INFO chromedriver: Start Chromedriver (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/chromedriver/lib/chromedriver/chromedriver) with args --port=9515 --url-base=/
2023-06-22T19:05:24.810Z INFO chromedriver: Starting ChromeDriver 113.0.5672.63 (0e1a4471d5ae5bf128b1bd8f4d627c8cbd55f70c-refs/branch-heads/5672@{#912}) on port 9515
2023-06-22T19:05:24.810Z INFO chromedriver: Only local connections are allowed.
2023-06-22T19:05:24.810Z INFO chromedriver: Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
2023-06-22T19:05:24.827Z INFO chromedriver: ChromeDriver was started successfully.
2023-06-22T19:05:24.901Z DEBUG @wdio/cli:utils: Finished to run "onPrepare" hook in 113ms
2023-06-22T19:05:24.901Z INFO @wdio/cli:launcher: Run onWorkerStart hook
2023-06-22T19:05:24.902Z INFO @wdio/cli:launcher: Run onWorkerStart hook
2023-06-22T19:05:24.902Z DEBUG @wdio/cli:utils: Finished to run "onWorkerStart" hook in 0ms
2023-06-22T19:05:24.902Z DEBUG @wdio/cli:utils: Finished to run "onWorkerStart" hook in 0ms
2023-06-22T19:05:24.902Z INFO @wdio/local-runner: Start worker 0-0 with arg: run,./wdio.conf.ts,--spec,features/before-hook-fail/**/*.feature
2023-06-22T19:05:24.907Z DEBUG @wdio/local-runner: Send command run to worker with cid "0-0"
2023-06-22T19:05:24.907Z INFO @wdio/local-runner: Start worker 0-1 with arg: run,./wdio.conf.ts,--spec,features/before-hook-fail/**/*.feature
2023-06-22T19:05:24.909Z DEBUG @wdio/local-runner: Send command run to worker with cid "0-1"
[0-0] 2023-06-22T19:05:25.401Z INFO @wdio/local-runner: Run worker command: run
[0-1] 2023-06-22T19:05:25.401Z INFO @wdio/local-runner: Run worker command: run
[0-1] 2023-06-22T19:05:25.459Z DEBUG @wdio/runner: init remote session
[0-0] 2023-06-22T19:05:25.460Z DEBUG @wdio/runner: init remote session
[0-0] 2023-06-22T19:05:25.462Z DEBUG @wdio/utils:initialiseServices: initialise service "chromedriver" as NPM package
[0-1] 2023-06-22T19:05:25.463Z DEBUG @wdio/utils:initialiseServices: initialise service "chromedriver" as NPM package
[0-1] RUNNING in chrome - file:///features/before-hook-fail/should-fail-in-given.feature
[0-1] 2023-06-22T19:05:25.680Z DEBUG @wdio/runner: init remote session
[0-0] RUNNING in chrome - file:///features/before-hook-fail/should-fail-in-before.feature
[0-0] 2023-06-22T19:05:25.680Z DEBUG @wdio/runner: init remote session
[0-1] 2023-06-22T19:05:25.681Z INFO webdriver: Initiate new session using the WebDriver protocol
[0-0] 2023-06-22T19:05:25.681Z INFO webdriver: Initiate new session using the WebDriver protocol
[0-1] 2023-06-22T19:05:25.741Z INFO webdriver: [POST] http://localhost:9515/session
[0-0] 2023-06-22T19:05:25.741Z INFO webdriver: [POST] http://localhost:9515/session
[0-1] 2023-06-22T19:05:25.742Z INFO webdriver: DATA {
[0-1]   capabilities: {
[0-1]     alwaysMatch: { 'goog:chromeOptions': [Object], browserName: 'chrome' },
[0-1]     firstMatch: [ {} ]
[0-1]   },
[0-1]   desiredCapabilities: { 'goog:chromeOptions': { args: [Array] }, browserName: 'chrome' }
[0-1] }
[0-0] 2023-06-22T19:05:25.742Z INFO webdriver: DATA {
[0-0]   capabilities: {
[0-0]     alwaysMatch: { 'goog:chromeOptions': [Object], browserName: 'chrome' },
[0-0]     firstMatch: [ {} ]
[0-0]   },
[0-0]   desiredCapabilities: { 'goog:chromeOptions': { args: [Array] }, browserName: 'chrome' }
[0-0] }
2023-06-22T19:05:26.000Z WARN chromedriver: [1687460726.000][WARNING]: This version of ChromeDriver has not been tested with Chrome version 114.
2023-06-22T19:05:26.002Z WARN chromedriver: [1687460726.002][WARNING]: This version of ChromeDriver has not been tested with Chrome version 114.
[0-1] Error in "0: Given step fails"
Error: Given throw
    at World.<anonymous> (file:///Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/features/step-definitions/before-hook-fail.steps.ts:8:11)
    at async wrapPromiseWithTimeout (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/time.ts:55:10)
    at async Object.run (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/user_code_runner.ts:86:16)
    at async Object.run (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/runtime/step_runner.ts:50:20)
    at async TestCaseRunner.invokeStep (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/runtime/test_case_runner.ts:134:12)
    at async TestCaseRunner.runStep (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/runtime/test_case_runner.ts:331:20)
    at async /Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/runtime/test_case_runner.ts:237:34
    at async TestCaseRunner.aroundTestStep (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/runtime/test_case_runner.ts:167:28)
[0-0] {
[0-0]   passed: false,
[0-0]   error: 'Error: Before throw\n' +
[0-0]     '    at World.<anonymous> (file:///Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/features/step-definitions/before-hook-fail.steps.ts:4:11)\n' +
[0-0]     '    at async wrapPromiseWithTimeout (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/time.ts:55:10)\n' +
[0-0]     '    at async Object.run (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/user_code_runner.ts:86:16)\n' +
[0-0]     '    at async Object.run (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/runtime/step_runner.ts:50:20)\n' +
[0-0]     '    at async TestCaseRunner.invokeStep (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/runtime/test_case_runner.ts:134:12)\n' +
[0-0]     '    at async TestCaseRunner.runHook (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/runtime/test_case_runner.ts:270:12)\n' +
[0-0]     '    at async /Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/runtime/test_case_runner.ts:228:18',
[0-0]   duration: 2.422582
[0-0] }
[0-0] 2023-06-22T19:05:26.117Z DEBUG @wdio/utils:shim: Finished to run "afterScenario" hook in 0ms
[0-0] 0
[0-0] 2023-06-22T19:05:26.118Z DEBUG @wdio/utils:shim: Finished to run "after" hook in 0ms
[0-0] 2023-06-22T19:05:26.118Z INFO webdriver: COMMAND deleteSession()
[0-0] 2023-06-22T19:05:26.118Z INFO webdriver: [DELETE] http://localhost:9515/session/5e764496e98a6b77a9977aa7b37fbe63
[0-1] {
[0-1]   passed: false,
[0-1]   error: 'Error: Given throw\n' +
[0-1]     '    at World.<anonymous> (file:///Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/features/step-definitions/before-hook-fail.steps.ts:8:11)\n' +
[0-1]     '    at async wrapPromiseWithTimeout (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/time.ts:55:10)\n' +
[0-1]     '    at async Object.run (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/user_code_runner.ts:86:16)\n' +
[0-1]     '    at async Object.run (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/runtime/step_runner.ts:50:20)\n' +
[0-1]     '    at async TestCaseRunner.invokeStep (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/runtime/test_case_runner.ts:134:12)\n' +
[0-1]     '    at async TestCaseRunner.runStep (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/runtime/test_case_runner.ts:331:20)\n' +
[0-1]     '    at async /Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/runtime/test_case_runner.ts:237:34\n' +
[0-1]     '    at async TestCaseRunner.aroundTestStep (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/runtime/test_case_runner.ts:167:28)',
[0-1]   duration: 3.212122
[0-1] }
[0-1] 2023-06-22T19:05:26.118Z DEBUG @wdio/utils:shim: Finished to run "afterScenario" hook in 0ms
[0-1] 1
[0-1] 2023-06-22T19:05:26.118Z DEBUG @wdio/utils:shim: Finished to run "after" hook in 0ms
[0-1] 2023-06-22T19:05:26.119Z INFO webdriver: COMMAND deleteSession()
[0-1] 2023-06-22T19:05:26.119Z INFO webdriver: [DELETE] http://localhost:9515/session/c707ea3ea7bdeb14c6bd114cee5780a9
[0-1] 2023-06-22T19:05:26.177Z INFO webdriver: RESULT null
[0-0] 2023-06-22T19:05:26.177Z INFO webdriver: RESULT null
2023-06-22T19:05:26.290Z DEBUG @wdio/local-runner: Runner 0-0 finished with exit code 0
[0-0] PASSED in chrome - file:///features/before-hook-fail/should-fail-in-before.feature
2023-06-22T19:05:26.290Z INFO @wdio/cli:launcher: Run onWorkerEnd hook
2023-06-22T19:05:26.290Z DEBUG @wdio/cli:utils: Finished to run "onWorkerEnd" hook in 0ms
2023-06-22T19:05:26.290Z DEBUG @wdio/local-runner: Runner 0-1 finished with exit code 1
[0-1] FAILED in chrome - file:///features/before-hook-fail/should-fail-in-given.feature
2023-06-22T19:05:26.290Z INFO @wdio/cli:launcher: Run onWorkerEnd hook
2023-06-22T19:05:26.290Z DEBUG @wdio/cli:utils: Finished to run "onWorkerEnd" hook in 0ms
2023-06-22T19:05:26.290Z INFO @wdio/cli:launcher: Run onComplete hook
1
{ finished: 2, passed: 1, retries: 0, failed: 1 }
2023-06-22T19:05:26.291Z DEBUG @wdio/cli:utils: Finished to run "onComplete" hook in 0ms

 "spec" Reporter:
------------------------------------------------------------------
[chrome 114.0.5735.133 mac os x #0-1] Running: chrome (v114.0.5735.133) on mac os x
[chrome 114.0.5735.133 mac os x #0-1] Session ID: c707ea3ea7bdeb14c6bd114cee5780a9
[chrome 114.0.5735.133 mac os x #0-1]
[chrome 114.0.5735.133 mac os x #0-1] » /features/before-hook-fail/should-fail-in-given.feature
[chrome 114.0.5735.133 mac os x #0-1] Should fail in Given step
[chrome 114.0.5735.133 mac os x #0-1] Fails in Given step and considered failed
[chrome 114.0.5735.133 mac os x #0-1]    ✖ Given step fails
[chrome 114.0.5735.133 mac os x #0-1]    - When step passes
[chrome 114.0.5735.133 mac os x #0-1]    - Then step passes
[chrome 114.0.5735.133 mac os x #0-1]
[chrome 114.0.5735.133 mac os x #0-1] 1 failing (137ms)
[chrome 114.0.5735.133 mac os x #0-1] 2 skipped
[chrome 114.0.5735.133 mac os x #0-1]
[chrome 114.0.5735.133 mac os x #0-1] 1) Fails in Given step and considered failed Given step fails
[chrome 114.0.5735.133 mac os x #0-1] Error: Given throw
[chrome 114.0.5735.133 mac os x #0-1] Error: Given throw
[chrome 114.0.5735.133 mac os x #0-1]     at World.<anonymous> (file:///Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/features/step-definitions/before-hook-fail.steps.ts:8:11)
[chrome 114.0.5735.133 mac os x #0-1]     at async wrapPromiseWithTimeout (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/time.ts:55:10)
[chrome 114.0.5735.133 mac os x #0-1]     at async Object.run (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/user_code_runner.ts:86:16)
[chrome 114.0.5735.133 mac os x #0-1]     at async Object.run (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/runtime/step_runner.ts:50:20)
[chrome 114.0.5735.133 mac os x #0-1]     at async TestCaseRunner.invokeStep (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/runtime/test_case_runner.ts:134:12)
[chrome 114.0.5735.133 mac os x #0-1]     at async TestCaseRunner.runStep (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/runtime/test_case_runner.ts:331:20)
[chrome 114.0.5735.133 mac os x #0-1]     at async /Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/runtime/test_case_runner.ts:237:34
[chrome 114.0.5735.133 mac os x #0-1]     at async TestCaseRunner.aroundTestStep (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/runtime/test_case_runner.ts:167:28)
------------------------------------------------------------------
[chrome 114.0.5735.133 mac os x #0-0] Running: chrome (v114.0.5735.133) on mac os x
[chrome 114.0.5735.133 mac os x #0-0] Session ID: 5e764496e98a6b77a9977aa7b37fbe63
[chrome 114.0.5735.133 mac os x #0-0]
[chrome 114.0.5735.133 mac os x #0-0] » /features/before-hook-fail/should-fail-in-before.feature
[chrome 114.0.5735.133 mac os x #0-0] Should fail in Before hook
[chrome 114.0.5735.133 mac os x #0-0] Fails in Hook step and considered failed
[chrome 114.0.5735.133 mac os x #0-0]    ✖ 
[chrome 114.0.5735.133 mac os x #0-0]    - Given step fails
[chrome 114.0.5735.133 mac os x #0-0]    - When step passes
[chrome 114.0.5735.133 mac os x #0-0]    - Then step passes
[chrome 114.0.5735.133 mac os x #0-0]
[chrome 114.0.5735.133 mac os x #0-0] 1 failing (137ms)
[chrome 114.0.5735.133 mac os x #0-0] 3 skipped
[chrome 114.0.5735.133 mac os x #0-0]
[chrome 114.0.5735.133 mac os x #0-0] 1) Fails in Hook step and considered failed 
[chrome 114.0.5735.133 mac os x #0-0] Error: Before throw
[chrome 114.0.5735.133 mac os x #0-0] Error: Before throw
[chrome 114.0.5735.133 mac os x #0-0]     at World.<anonymous> (file:///Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/features/step-definitions/before-hook-fail.steps.ts:4:11)
[chrome 114.0.5735.133 mac os x #0-0]     at async wrapPromiseWithTimeout (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/time.ts:55:10)
[chrome 114.0.5735.133 mac os x #0-0]     at async Object.run (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/user_code_runner.ts:86:16)
[chrome 114.0.5735.133 mac os x #0-0]     at async Object.run (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/runtime/step_runner.ts:50:20)
[chrome 114.0.5735.133 mac os x #0-0]     at async TestCaseRunner.invokeStep (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/runtime/test_case_runner.ts:134:12)
[chrome 114.0.5735.133 mac os x #0-0]     at async TestCaseRunner.runHook (/Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/runtime/test_case_runner.ts:270:12)
[chrome 114.0.5735.133 mac os x #0-0]     at async /Users/ernst/Documents/prog/github/wdio-cucumber-framework-bug-reproduce/node_modules/@cucumber/cucumber/src/runtime/test_case_runner.ts:228:18


Spec Files:      1 passed, 1 failed, 2 total (100% completed) in 00:00:01 

2023-06-22T19:05:26.297Z INFO @wdio/local-runner: Shutting down spawned worker
2023-06-22T19:05:26.549Z INFO @wdio/local-runner: Waiting for 0 to shut down gracefully
2023-06-22T19:05:26.550Z INFO @wdio/local-runner: shutting down
ernsts-MacBook-Pro:wdio-cucumber-framework-bug-reproduce ernst$

Code of Conduct

  • I agree to follow this project's Code of Conduct

Is there an existing issue for this?

  • I have searched the existing issues

Metadata

Metadata

Assignees

Labels

Bug 🐛

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions