Skip to content

investigate uptick in failing integration tests on Travis/Appveyor #6793

@shiftkey

Description

@shiftkey

Opening this up to brain dump so it's not just in my head

Travis CI:

AppVeyor:

Other infrastructure: unclear

Problem behaviour:

  • jest runs to execute launch test
  • spectron initializes using build application bundle
  • test fails because of a timeout to launch the app
$ cross-env TEST_ENV=1 ELECTRON_NO_ATTACH_CONSOLE=1 xvfb-maybe --auto-servernum -- jest --config ./app/jest.integration.config.js
FAIL app/test/integration/launch-test.ts (12.399s)
  App
    ✕ opens a window on launch (10008ms)
  ● App › opens a window on launch
    Timeout - Async callback was not invoked within the 10000ms timeout specified by jest.setTimeout.
      at mapper (../node_modules/jest-jasmine2/build/queue_runner.js:41:52)
  ● App › opens a window on launch
    TypeError: Cannot read property 'isVisible' of undefined
      40 | 
      41 |   it('opens a window on launch', async () => {
    > 42 |     await app.client.waitUntil(() => app.browserWindow.isVisible(), 5000)
         |                                                        ^
      43 | 
      44 |     const count = await app.client.getWindowCount()
      45 |     expect(count).toBe(1)
      at Object.app.client.waitUntil (test/integration/launch-test.ts:42:56)
      at Timer.tick (../node_modules/webdriverio/build/lib/utils/Timer.js:111:31)
      at Timer.start (../node_modules/webdriverio/build/lib/utils/Timer.js:75:22)
      at new Timer (../node_modules/webdriverio/build/lib/utils/Timer.js:62:14)
      at Object.<anonymous>.exports.default (../node_modules/webdriverio/build/lib/commands/waitUntil.js:25:17)
      at Object.exec (../node_modules/webdriverio/build/lib/helpers/safeExecute.js:28:24)
      at Object.resolve (../node_modules/webdriverio/build/lib/webdriverio.js:190:29)
      at lastPromise.then.resolve.call.depth (../node_modules/webdriverio/build/lib/webdriverio.js:485:32)
      at _fulfilled (../node_modules/q/q.js:854:54)
      at self.promiseDispatch.done (../node_modules/q/q.js:883:30)
      at Promise.promise.promiseDispatch (../node_modules/q/q.js:816:13)
      at ../node_modules/q/q.js:570:49
      at runSingle (../node_modules/q/q.js:137:13)
      at flush (../node_modules/q/q.js:125:13)

Test Suites: 1 failed, 1 total
Tests:       1 failed, 1 total
Snapshots:   0 total
Time:        12.976s
Ran all test suites.
  • jest does not terminate
Jest did not exit one second after the test run has completed.
This usually means that there are asynchronous operations that weren't stopped in your tests. Consider running Jest with `--detectOpenHandles` to troubleshoot this issue.
  • CI is aborted because of inactivity
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
The build has been terminated

Root cause ideas:

  • Electron 2/3 upgrades? Downgrades between major versions are not recommended, but we've been testing Electron 3 to get ready for next release. Maybe build agent is trying to launch a v2 app against a v3 data directory?
  • Build agent unexpected caching files between builds
  • Spectron upgrade too early and uncovered some sort of v2-specific issue? downgrade does not seem to fix this issue
  • using jest to drive end-to-end tests and in this case it seems to deadlock ignoring for now
  • ???

TODO:

  • verify enabling --detectOpenHandles does not change this behaviour no change, and no helpful information
  • run the test manually from ts-node to try and yield logs about the failure done, see below for context
  • try and get data from Spectron/ChromeDriver/WebDriver logs when build fails https://gist.github.com/shiftkey/1181223faec6aa8b961ac0d84bd23b34
  • try and reproduce on a VM we can control
  • ???

Metadata

Metadata

Assignees

Labels

infrastructureIssues and pull requests related to scripts and tooling for GitHub Desktop

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions