Skip to content
This repository was archived by the owner on Dec 18, 2018. It is now read-only.

Make it possible to run the Browser Functional Tests from the command line#1448

Merged
analogrelay merged 6 commits into
devfrom
anurse/automate-browser-tests
Feb 16, 2018
Merged

Make it possible to run the Browser Functional Tests from the command line#1448
analogrelay merged 6 commits into
devfrom
anurse/automate-browser-tests

Conversation

@analogrelay

@analogrelay analogrelay commented Feb 15, 2018

Copy link
Copy Markdown
Contributor

This PR configures the npm test command in the client-ts\FunctionalTests to automatically launch a browser (headless chrome, by default) and run the functional tests, collecting the results and writing them out.

There are a few elements to this:

  • We register a Jasmine Reporter in the Functional Tests that generates Test-Anywhere Protocol (TAP) output describing the test run as <li> elements in a hidden <ul> element (optionally made visible by the displayTap query string value)
  • The run-tests.js script, uses the selenium-webdriver package. It:
    1. Launches Selenium WebDriver
    2. Launches the server
    3. Instructs WebDriver to open the page on the server that runs the tests in a browser (any browser supported by Selenium WebDriver)
    4. Scrapes the DOM, reading the hidden <li> values and write them to the console, until the tests complete
    5. Terminates the browser, server and WebDriver.
  • Now, the run-tests.js script basically acts as a script that writes TAP output for our functional tests. We can pipe that through any TAP-compatible reporter (we use faucet here) to report output.

IMPORTANT: This is not yet integrated into our build, the first pass is just to make it runnable and provide the infrastructure.

Also, this mechanism should be able to automate BrowserStack/Sauce Labs/some other browser lab. We'll need to work out the mechanisms for deploying the server code somewhere, but we should be able to use this to move forward with that.

Examples:

Successful test run (click to embiggen):

https://thumbs.gfycat.com/WatchfulHarmoniousAmazontreeboa-size_restricted.gif

Failed test run (click to embiggen):

https://thumbs.gfycat.com/VillainousDeficientDuiker-size_restricted.gif

TODO

  • Rebase
  • Fix tslint errors (thanks @BrennanConroy :(... but seriously it's good)
  • Don't build the entire client in FunctionalTests.csproj, we can make sure it builds before we run the tests in the CI but doing it in FunctionalTests.csproj is bogging down VS builds.

@analogrelay analogrelay force-pushed the anurse/automate-browser-tests branch from fd39de5 to 9306956 Compare February 15, 2018 00:37
@analogrelay

Copy link
Copy Markdown
Contributor Author

Had to leave before tslint errors were all resolved but they're minimal and I will fix them before check-in (After all the build will be red ;))


<Target Name="ClientBuild" BeforeTargets="AfterBuild">
<!-- This will result in a double build of the JavaScript modules. Not sure the best way to deal with that right now. -->
<Exec Command="npm run build" WorkingDirectory="$(MSBuildThisFileDirectory)/.." />

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

YAAS

@davidfowl davidfowl left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

File bugs for the TODOs

@analogrelay

Copy link
Copy Markdown
Contributor Author

The TODOs are to be done in the PR before merge. In fact, they are done now.

"tree-kill": "^1.2.0",
"ts-node": "^4.1.0",
"webdriver-manager": "^12.0.6",
"yargs": "^11.0.0"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI @Eilon, some additional dev-time-only dependencies.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. I assume you did/will do the needful?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed I shall, just raising it FYI

@analogrelay

Copy link
Copy Markdown
Contributor Author

Still got some flaky tests going on in AppVeyor... @BrennanConroy / @davidfowl are you guys aware of these?

@davidfowl

Copy link
Copy Markdown
Member

I'm trying to fix the flakyness, I can't get my last PR to pass a single run.

@analogrelay analogrelay merged commit 1bd37ca into dev Feb 16, 2018
@analogrelay analogrelay deleted the anurse/automate-browser-tests branch March 13, 2018 21:30
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants