[functionalTestRunner] replace intern by spalger · Pull Request #10910 · elastic/kibana

spalger · 2017-03-28T05:38:30Z

In an effort to make functional/selenium testing more accessible to plugins and external projects, this pr removes intern and replaces it with a custom solution.

Quick info

src/functional_test_runner: source for the bare bones functional test runner. Most functionality via services in config
tests/functional/config.js: Kibana's functional_test_runner config file, plugins will have one just like this
- config file defines
  - which test files to load
  - providers for the the Services and PageObjects
  - config values used by some services
services:
- default:
  - config: use config.get() to read values from the config file)
  - log: use log.debug() and log.error() just like console.log but with more organized output
  - lifecycle: designed for use in other services, calls handlers at certain lifecycle points
    - lifecycle.on(phaseName, handler): register a listener for a specific phase that will be called at that point. The handler can be async and return a promise if it needs to do async work.
    - phases: beforeLoadTests, beforeTests, beforeEachTest, cleanup, phaseStart, phaseEnd
- kibana (see tests/functional/services):
  - remote: leadfoot remote that provides access to automated browser (previously injected into each PageObject)
  - esArchiver: load/unload archives stored with the esArchiver ([esArchiver] combine elasticDump and ScenarioManager #10359)
  - es: elasticsearch client
  - find: helpers for remote findBy* methods that log and manage timeouts (previously PageObjects.common.findBy*)
  - retry: helpers for retrying logic (previously PageObjects.common.try*)
  - testSubjects: helpers for reading/interacting with test subjects (previously PageObjects.common.find/click/findAllTestSubect)
  - docTable: helpers for interacting with doc tables
  - pointSeriesVis: helpers for interacting with point series visualizations

Running the CLI

This is the second script to be added to the root scripts/ directory. Access it with node scripts/functional_test_runner and give it the --help flag to see what it can do.

Debugging Tests

I suggest installing the NIM Chrome Extension, turning on the "auto" setting, and then using the new node.js --inspect flag.

node --debug-brk --inspect scripts/functional_test_runner

Upstream prs:

rhoboat · 2017-04-04T00:22:32Z

crosslink to asciidocs from CONTRIBUTING.md
write functional-testing docs
make sure all the links in the doc are pointing to master
reinstate unit testing before merging
set default loglevel to debug or info, decide
- decided that in jenkins, the default will be debug and the cli tool will run in info

rhoboat · 2017-04-04T01:43:01Z

rhoboat · 2017-04-06T01:39:30Z

23 Tests Failing, all in Management app

rhoboat · 2017-04-06T14:54:09Z

jenkins, test it

rhoboat · 2017-04-06T19:43:59Z

4 Tests Failing, management app

rhoboat · 2017-04-06T21:12:14Z

1 Test failing, management https://kibana-ci.elastic.co/job/elastic+kibana+pull-request+multijob-selenium/3484/consoleFull

rhoboat · 2017-04-06T21:28:50Z

@stacey-gammon I tagged you because I incorporated your changes in here, and specifically for the dashboard tests. It's a big PR, so if you wanted to just look at those, I'd be grateful.

@LeeDr Would like your feedback on the way the tests look and how the functional-test-runner works. Since you're traveling, if you aren't able to take a close look just yet, it's no big deal, I'll tag others, too!

For everyone, I/we will be writing documentation that will help this go live. I can ping you when documentation is ready.

rhoboat · 2017-04-06T21:35:18Z

@spalger Would like to add you as reviewer. I'm not sure if I'm including files and directories that should no longer exist. Can you please check?

…op-level suite

kimjoar · 2017-04-11T22:03:55Z

🎉

epixa · 2017-04-11T22:12:03Z

Awesome! Great job on this @spalger and @archanid.

Let's backport it into 5.x as well.

@gammon

* [functional_test_runner] replace functional testing tools with custom/pluggable solution * [functional_test_runner] Convert unit tests to commonjs format * [functional_test_runner] Fix dashboard test in wrong mode * [functional_test_runner] Add dashboardLandingPage test subject * [functional_test_runner] Get Visualize page object * [functional_test_runner] Fix outdated references * [functional_test_runner] Fix more outdated refs * [functional_test_runner] Remove duplicate tests * [functional_test_runner] Improve test readability * [functional_test_runner] 😞 So many duplicate methods * [functional_test_runner] Move mgmt `before` outside toplevel describe * [functional_test_runner] Settings page obj missing methods * [functional_test_runner] Add improvements from @gammon * [functional_test_runner] Fix return statements in async funcs * [functional_test_runner] Move before() to correct scope * [functional_test_runner] Add after() hooks to remove index patterns * [functional_test_runner] Attempt to fix vertical bar chart tests * [functional_test_runner] Clean up * [functional_test_runner] Reinstate unit tests * [functional_test_runner] Set default loglevel back to info * [functional_test_runner] Replace `context`s with `describe`s * [functional_test_runner] Better error handling * [functional_test_runner] Add in new Tile Map tests * Incorporate changes from master * [functional_test_runner] validate that every test file has a single top-level suite * Update contributing doc with link to full doc * [docs] Spelling and grammar fixes * docs: writing and running functional tests * [docs] Move plugin doc to plugin area * [docs] Housekeeping. Doc in wrong place * [docs] Remove dup doc file * [grunt] Only run mocha_setup when running tests, not every grunt task

@gammon

* [esArchiver] combine elasticDump and ScenarioManager (#10359) * As a part of bringing functional testing to plugins, esArchiver gives these plugins a way to capture and reload es indexes without needing to write a bunch of custom code. It works similarly to the elasticDump and ScenarioManager tools that it replaces. Differences: - Streaming implementation allows for much larger archives - CLI for creating and using archives - Configurable archive location - Stores the data in gzipped files (better for source control, searching, large archives) - Automatically identifies and upgrades Kibana config documents Methods: - `#load(name)`: import an archive - `#loadIfNeeded(name)`: import an archive, but skip the documents what belong to any existing index - `#unload(name)`: delete the indexes stored in an archive CLI operations: - `./bin/es_archiver save <name> [index patterns...]`: save the mapping and documents in one or more indexes that match the wild-card patterns into an the `<name>` archive - `./bin/es_archiver load <name>`: load the mapping and documents from the `<name>` archive * [functional_tests/common/nagivate] check for statusPage * [es_archiver] move bins into new scripts dir * [functional_tests/apps/context] use esArchiver * [esArchiver] general improvements after showing to a few folks - remove auto-upgrading config doc logic (until we have better access to kibana version info) - export unload command - remove preemptive checks in favor of reacting to errors - use type "doc" vs "hit" for doc records (consistency) - wrote a bunch of pending tests to think though and plan * [esArchiver] make log a stream that writes to itself * [esArchiver] fill in stats and archive format tests * [esArchiver] splitup action logic * [esArchiver/cli] fix cli --help output and comment * [esArchiver] remove type-based param coercion * [esArchiver/log] use strings for log levels * [esArchvier] remove unused var * [esArchiver/indexDocRecordsStream] add tests * [esArchive] fill in remaining tests * [esArchiver] fix dem tests * [eslint] remove unused vars * [esArchiver/loadIfNeeded] fix call to load() * [esArchiver] remove loadDumpData helpers * [esArchiver] update archives for 5.x * [functionalTestRunner] replace intern (#10910) * [functional_test_runner] replace functional testing tools with custom/pluggable solution * [functional_test_runner] Convert unit tests to commonjs format * [functional_test_runner] Fix dashboard test in wrong mode * [functional_test_runner] Add dashboardLandingPage test subject * [functional_test_runner] Get Visualize page object * [functional_test_runner] Fix outdated references * [functional_test_runner] Fix more outdated refs * [functional_test_runner] Remove duplicate tests * [functional_test_runner] Improve test readability * [functional_test_runner] 😞 So many duplicate methods * [functional_test_runner] Move mgmt `before` outside toplevel describe * [functional_test_runner] Settings page obj missing methods * [functional_test_runner] Add improvements from @gammon * [functional_test_runner] Fix return statements in async funcs * [functional_test_runner] Move before() to correct scope * [functional_test_runner] Add after() hooks to remove index patterns * [functional_test_runner] Attempt to fix vertical bar chart tests * [functional_test_runner] Clean up * [functional_test_runner] Reinstate unit tests * [functional_test_runner] Set default loglevel back to info * [functional_test_runner] Replace `context`s with `describe`s * [functional_test_runner] Better error handling * [functional_test_runner] Add in new Tile Map tests * Incorporate changes from master * [functional_test_runner] validate that every test file has a single top-level suite * Update contributing doc with link to full doc * [docs] Spelling and grammar fixes * docs: writing and running functional tests * [docs] Move plugin doc to plugin area * [docs] Housekeeping. Doc in wrong place * [docs] Remove dup doc file * [grunt] Only run mocha_setup when running tests, not every grunt task * [eslint] remove use of context()

spalger assigned solostyle Mar 28, 2017

spalger mentioned this pull request Mar 28, 2017

WIP: [functional-tests] replace intern with custom test runner #9853

Closed

spalger added the Team:Core Platform Core services: plugins, logging, config, saved objects, http, ES client, i18n, etc t// label Mar 28, 2017

spalger force-pushed the implement/functional-test-runner branch 3 times, most recently from b60c5a8 to d3be827 Compare March 29, 2017 17:32

rhoboat force-pushed the implement/functional-test-runner branch 2 times, most recently from ee68177 to 89800f1 Compare March 29, 2017 19:43

rhoboat assigned rhoboat and unassigned solostyle Mar 29, 2017

rhoboat force-pushed the implement/functional-test-runner branch 4 times, most recently from d90fbe4 to 5b8ea08 Compare April 3, 2017 21:35

rhoboat force-pushed the implement/functional-test-runner branch 2 times, most recently from c1415a6 to 25e4f9c Compare April 5, 2017 17:51

rhoboat mentioned this pull request Apr 5, 2017

Stabilize dashboard selenium tests #11004

Closed

rhoboat force-pushed the implement/functional-test-runner branch 2 times, most recently from 5bee23d to 73326bf Compare April 6, 2017 00:15

rhoboat force-pushed the implement/functional-test-runner branch from 4cf226f to 5fd1d3f Compare April 6, 2017 21:12

rhoboat requested review from LeeDr and stacey-gammon April 6, 2017 21:23

archanid and others added 16 commits April 11, 2017 16:08

[functional_test_runner] Attempt to fix vertical bar chart tests

cccfbfb

[functional_test_runner] Clean up

38dee88

[functional_test_runner] Reinstate unit tests

8d15b1a

[functional_test_runner] Set default loglevel back to info

ea0175b

[functional_test_runner] Replace contexts with describes

4a49e6e

[functional_test_runner] Better error handling

c126eb8

[functional_test_runner] Add in new Tile Map tests

4e63aef

Incorporate changes from master

c57f64c

docs: writing and running functional tests

ceb9a75

[functional_test_runner] validate that every test file has a single t…

0209314

…op-level suite

Update contributing doc with link to full doc

32cde4b

[docs] Spelling and grammar fixes

7b892ee

[docs] Move plugin doc to plugin area

c68dc2b

[docs] Housekeeping. Doc in wrong place

d21e85e

[docs] Remove dup doc file

f412485

[grunt] Only run mocha_setup when running tests, not every grunt task

d364014

rhoboat force-pushed the implement/functional-test-runner branch from 31e3438 to d364014 Compare April 11, 2017 21:14

rhoboat merged commit 9043476 into elastic:master Apr 11, 2017

epixa added test v5.4.0 labels Apr 11, 2017

spalger mentioned this pull request Apr 11, 2017

[5.x] [functionalTestRunner] replace intern (#10910) #11171

Merged

kobelb mentioned this pull request Apr 12, 2017

Region map visualization #10937

Merged

25 tasks

jbudz mentioned this pull request Apr 17, 2017

Update chromedriver to 2.28 for Chrome 57 #11093

Closed

spalger deleted the implement/functional-test-runner branch August 18, 2020 17:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[functionalTestRunner] replace intern#10910

[functionalTestRunner] replace intern#10910
rhoboat merged 32 commits intoelastic:masterfrom
spalger:implement/functional-test-runner

spalger commented Mar 28, 2017 •

edited by rhoboat

Loading

Uh oh!

rhoboat commented Apr 4, 2017 •

edited

Loading

Uh oh!

rhoboat commented Apr 4, 2017 •

edited

Loading

Uh oh!

rhoboat commented Apr 6, 2017

Uh oh!

rhoboat commented Apr 6, 2017

Uh oh!

rhoboat commented Apr 6, 2017

Uh oh!

rhoboat commented Apr 6, 2017 •

edited

Loading

Uh oh!

rhoboat commented Apr 6, 2017

Uh oh!

rhoboat commented Apr 6, 2017

Uh oh!

kimjoar commented Apr 11, 2017

Uh oh!

epixa commented Apr 11, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

spalger commented Mar 28, 2017 • edited by rhoboat Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rhoboat commented Apr 4, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rhoboat commented Apr 4, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rhoboat commented Apr 6, 2017

Uh oh!

rhoboat commented Apr 6, 2017

Uh oh!

rhoboat commented Apr 6, 2017

Uh oh!

rhoboat commented Apr 6, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rhoboat commented Apr 6, 2017

Uh oh!

rhoboat commented Apr 6, 2017

Uh oh!

kimjoar commented Apr 11, 2017

Uh oh!

epixa commented Apr 11, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

spalger commented Mar 28, 2017 •

edited by rhoboat

Loading

rhoboat commented Apr 4, 2017 •

edited

Loading

rhoboat commented Apr 4, 2017 •

edited

Loading

rhoboat commented Apr 6, 2017 •

edited

Loading