Skip to content

e2e: when CircleCI hits a Chrome console error, do not cause cascading failures#22660

Merged
legobeat merged 2 commits intodevelopfrom
e2e/ganache-and-errors
Jan 30, 2024
Merged

e2e: when CircleCI hits a Chrome console error, do not cause cascading failures#22660
legobeat merged 2 commits intodevelopfrom
e2e/ganache-and-errors

Conversation

@HowardBraham
Copy link
Copy Markdown
Contributor

@HowardBraham HowardBraham commented Jan 25, 2024

Description

This PR fixes two related bug reports:

  1. @hjetpoluru showed me that add-custom-network.spec.js can have one test crash from a Chrome error, and then the rest of the tests in that file automatically fail with listen EADDRINUSE: address already in use 127.0.0.1:8545
  2. @Gudahtt pointed out that if a test gets a Chrome error, it can often retry and succeed, but never close, and then report as a failure to CircleCI

Method to fix:

  1. Ignore console errors that start with Warning: as these shouldn't be causing test failures anyway
  2. The throw new Error() used to be inside driver.logBrowserError(), but that function is running inside the callback this.driver.onLogEvent(), and not in the main test thread. It now checks for and throws the Error in the main test thread, after await testSuite() but before console.log(`\nSuccess on testcase: '${title}'\n`);.
  3. It would previously get the Chrome console errors in the test logs correctly with yarn start:test, but now it supports yarn build:test too (this was an oversight of mine in the past)

Related issues

Manual testing steps

Screenshots/Recordings

Pre-merge author checklist

  • I’ve followed MetaMask Coding Standards.
  • I've clearly explained what problem this PR is solving and how it is solved.
  • I've linked related issues
  • I've included manual testing steps
  • I've included screenshots/recordings if applicable
  • I’ve included tests if applicable
  • I’ve documented my code using JSDoc format if applicable
  • I’ve applied the right labels on the PR (see labeling guidelines). Not required for external contributors.
  • I’ve properly set the pull request status:
    • In case it's not yet "ready for review", I've set it to "draft".
    • In case it's "ready for review", I've changed it from "draft" to "non-draft".

Pre-merge reviewer checklist

  • I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
  • I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

@HowardBraham HowardBraham self-assigned this Jan 25, 2024
@HowardBraham HowardBraham requested a review from a team as a code owner January 25, 2024 04:04
@github-actions
Copy link
Copy Markdown
Contributor

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

@HowardBraham HowardBraham requested a review from dan437 January 25, 2024 04:04
Copy link
Copy Markdown
Contributor

@hjetpoluru hjetpoluru left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! I've tested it locally and its working.

@metamaskbot
Copy link
Copy Markdown
Collaborator

Builds ready [8633291]
Page Load Metrics (831 ± 31 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint101159126178
domContentLoaded15372363
load7069768316431
domInteractive15372263
Bundle size diffs
  • background: 0 Bytes (0.00%)
  • ui: 0 Bytes (0.00%)
  • common: 0 Bytes (0.00%)

@HowardBraham HowardBraham force-pushed the e2e/ganache-and-errors branch from 8633291 to ac3d8b7 Compare January 26, 2024 19:47
@metamaskbot
Copy link
Copy Markdown
Collaborator

Builds ready [ac3d8b7]
Page Load Metrics (751 ± 13 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint9011110363
domContentLoaded9211431
load7008047512713
domInteractive9211431
Bundle size diffs
  • background: 0 Bytes (0.00%)
  • ui: 0 Bytes (0.00%)
  • common: 0 Bytes (0.00%)

@HowardBraham HowardBraham force-pushed the e2e/ganache-and-errors branch from ac3d8b7 to 929fd58 Compare January 29, 2024 23:45
@metamaskbot
Copy link
Copy Markdown
Collaborator

Builds ready [929fd58]
Page Load Metrics (765 ± 24 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint95135106115
domContentLoaded9211232
load7049197655024
domInteractive9211232
Bundle size diffs
  • background: 0 Bytes (0.00%)
  • ui: 0 Bytes (0.00%)
  • common: 0 Bytes (0.00%)

@codecov
Copy link
Copy Markdown

codecov bot commented Jan 30, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (3ff646b) 68.08% compared to head (3bdd875) 68.08%.
Report is 1 commits behind head on develop.

Additional details and impacted files
@@           Coverage Diff            @@
##           develop   #22660   +/-   ##
========================================
  Coverage    68.08%   68.08%           
========================================
  Files         1091     1091           
  Lines        42923    42923           
  Branches     11415    11415           
========================================
  Hits         29223    29223           
  Misses       13700    13700           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@metamaskbot
Copy link
Copy Markdown
Collaborator

Builds ready [3bdd875]
Page Load Metrics (804 ± 27 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint84144112168
domContentLoaded8281563
load7119698045727
domInteractive8281563
Bundle size diffs
  • background: 0 Bytes (0.00%)
  • ui: 0 Bytes (0.00%)
  • common: 0 Bytes (0.00%)

@legobeat legobeat merged commit 80451e3 into develop Jan 30, 2024
@legobeat legobeat deleted the e2e/ganache-and-errors branch January 30, 2024 19:13
@github-actions github-actions bot locked and limited conversation to collaborators Jan 30, 2024
@legobeat legobeat restored the e2e/ganache-and-errors branch January 30, 2024 19:13
@metamaskbot metamaskbot added the release-11.11.0 Issue or pull request that will be included in release 11.11.0 label Jan 30, 2024
@HowardBraham HowardBraham deleted the e2e/ganache-and-errors branch January 31, 2024 17:54
@HowardBraham HowardBraham added the contributor experience An issue that impacts, or planned improvement to, the contributor experience. label Oct 29, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-testSuite contributor experience An issue that impacts, or planned improvement to, the contributor experience. flaky tests release-11.11.0 Issue or pull request that will be included in release 11.11.0 team-extension-platform Extension Platform team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants