test: visual regression by nickofthyme · Pull Request #394 · elastic/elastic-charts

nickofthyme · 2019-09-26T18:05:12Z

Summary

closes #77

Visual regression tests and dependency upgrades

Add visual testing config

to run the integration tests just run. Don't forget to run yarn to get the latest deps.

yarn jest:integration

These tests are separate from the jest unit tests.

The following devDependencies were used:

Based on this example config/setup.

git ignores the screenshot diffs (i.e.**/__diff_output__/) as this seems to be a local use case. Any diffs created from CI are stored in gcs

Add seeded random number function

Please use the getRandomNumber function in stories/ and for *.test.ts{x} files (there is a lint rule to make sure 😏).

This allows for randomness in story data but allows us to control the randomness when we want, like in visual tests. This is controlled by using the RNG_SEED environment variable.

Visual regression tests use the 'elastic-charts' string add the RNG_SEED.

RNG_SEED='elastic-charts' yarn start

Upgraded `devDependencies`:

jest
typescript
@typescript-eslint/eslint-plugin
@typescript-eslint/parser
ts-jest
ts-loader

None of the upgrades create external breaking changes in the library

Note https://storybook.js.org/docs/testing/automated-visual-testing/#challenges

Prevent no-show charts here

Checklist

Any consumer-facing exports were added to src/index.ts (and stories only import from ../src except for test data & storybook)
Unit tests were updated or added to match the most common scenarios
Each commit follows the convention

Add basic screenshot config and first test. Upgrade typescript, jest and related dev dependencies.

codecov-io · 2019-09-26T18:39:53Z

Codecov Report

Merging #394 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #394      +/-   ##
==========================================
+ Coverage   97.95%   97.96%   +<.01%     
==========================================
  Files          39       39              
  Lines        2887     2899      +12     
  Branches      692      680      -12     
==========================================
+ Hits         2828     2840      +12     
  Misses         52       52              
  Partials        7        7

Impacted Files	Coverage Δ
src/chart_types/xy_chart/utils/dimensions.ts	`100% <100%> (ø)`	⬆️
src/chart_types/xy_chart/utils/axis_utils.ts	`100% <100%> (ø)`	⬆️
src/utils/scales/scale_band.ts	`94.87% <100%> (ø)`	⬆️
.../chart_types/xy_chart/crosshair/crosshair_utils.ts	`97.53% <100%> (ø)`	⬆️
src/utils/bbox/canvas_text_bbox_calculator.ts	`100% <100%> (ø)`	⬆️
...chart_types/xy_chart/utils/stacked_series_utils.ts	`97.5% <100%> (ø)`	⬆️
src/chart_types/xy_chart/store/chart_state.ts	`97.45% <100%> (ø)`	⬆️
src/chart_types/xy_chart/utils/scales.ts	`100% <0%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7db45de...483603c. Read the comment docs.

markov00 · 2019-09-27T09:11:52Z

@nickofthyme thanks to starting this!
We started more or less the same a long time ago, but we stopped the process along the way.
One nice thing was kept from our previous work: we have a folder on our Jenkins CI where we can put all the diffs between the baseline and the current version.
This folder is then accessible from Jenkins Google Cloud Storage:
see

elastic-charts/.ci/jobs/defaults.yml

Lines 68 to 76 in d8f4531

    
           - google-cloud-storage: 
        
               credentials-id: kibana-ci-gcs-plugin 
        
               uploads: 
        
               - classic: 
        
                   file-pattern: stories/__image_diff_snapshots__/**/* 
        
                   storage-location: gs://kibana-ci-artifacts/jobs/$JOB_NAME/$BUILD_NUMBER 
        
                   share-publicly: true 
        
                   upload-for-failed-jobs: true 
        
                   show-inline: true

I think it's enough to change the default diff storage to stories/__image_diff_snapshots__/**/*

nickofthyme · 2019-09-27T14:09:05Z

@markov00 Ok sweet I'll take a look at that! Thanks

nickofthyme · 2019-10-01T14:16:34Z

Waiting to address lfs quota https://github.com/elastic/infra/issues/14886

markov00

This looks great now, I've increased the server launch waiting time a bit as storybook can take a while before loading everything.
I think we can also go and remove the fonts so we don't depends on any particular fonts and we always use the fallback inside our docker container.

As next phase we can have some sort of html report or better a bot that push a comment on the PR with the diffs

package.json

.storybook/config.ts

integration/jest-puppeteer.config.js

markov00

LGTM
please check the squash commit message and align the PR title to that

markov00 · 2019-10-10T20:08:02Z

🎉 This PR is included in version 13.5.2 🎉

The release is available on:

Your semantic-release bot 📦🚀

nickofthyme added 3 commits September 25, 2019 11:28

build(jest): upgrade jest to 24.9.0

405a7d0

build: screenshot testing config and upgrade dev deps

2099427

Add basic screenshot config and first test. Upgrade typescript, jest and related dev dependencies.

build: track *.png files in integration directory using Git LFS

75797d9

nickofthyme added :test Missing or to be fixed test visual testing issues related to visual testing labels Sep 26, 2019

nickofthyme requested review from markov00, monfera and rshen91 September 26, 2019 18:05

chore: remove all screenshots to start fresh

261cb04

nickofthyme force-pushed the tests/visual-regression branch from fbf6919 to a198b70 Compare September 26, 2019 23:37

nickofthyme force-pushed the tests/visual-regression branch 3 times, most recently from 48cecd2 to b331814 Compare September 29, 2019 17:37

test: run all base storybook visual screenshots

426f1dc

nickofthyme force-pushed the tests/visual-regression branch from 1edf75c to 426f1dc Compare October 1, 2019 15:49

nickofthyme added 4 commits October 1, 2019 10:49

Merge branch 'master' into tests/visual-regression

0f7b42d

ci: update gcs file upload storage path

7c2b16d

refactor: remove lfs tracking

7ae39f4

ci: update job to include visual tests

dc31d9f

nickofthyme marked this pull request as ready for review October 3, 2019 14:41

nickofthyme added 6 commits October 3, 2019 10:07

ci: tweak NODE_ENV

49b7705

fix: ci scripts

c5447fe

ci: test OpenHandles

cf9aedb

ci: test visual regression changes

ff93505

ci: test ci tweak

b916d65

fix: control randomness in stories

b55eca7

nickofthyme and others added 10 commits October 7, 2019 11:04

fix: timezone ci issue

2fa55a5

ci: run puppeteer in docker on ci

1a93bd0

fix: add back sandbox for now

b84c95c

WIP

d66153e

fix: timezone and font issues

5707e69

test: update all visual screenshots

6862eef

fix: up timeout

e1181a4

Merge branch 'master' into tests/visual-regression

9ae9c5c

fix: add Marco's fixes

f743df2

fix: increase server launch timeout

50ef908

markov00 reviewed Oct 8, 2019

View reviewed changes

nickofthyme added 12 commits October 8, 2019 09:36

fix: remove failed attempted code

6ef67b9

fix: pr comments

9a32799

fix: get chart rotation knob param issue

5a7431d

WIP

593d06c

fix: add tests for test stories, clean helper methods

b59879d

fix: attempt to prevent no-show charts

f5d2725

Merge branch 'master' into tests/visual-regression

2b3eea8

test: update screenshots from master changes

4498e8e

fix: pr comments

aaad5c7

fix: remove debugger

2ca6b90

fix: update screenshot

523a695

fix: change data attr name

483603c

nickofthyme requested a review from markov00 October 10, 2019 13:21

markov00 approved these changes Oct 10, 2019

View reviewed changes

nickofthyme merged commit 59c39bd into elastic:master Oct 10, 2019

nickofthyme deleted the tests/visual-regression branch October 10, 2019 13:30

nickofthyme changed the title ~~tests: visual regression~~ test: visual regression Oct 10, 2019

markov00 added the released Issue released publicly label Oct 10, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: visual regression#394

test: visual regression#394
nickofthyme merged 43 commits intoelastic:masterfrom
nickofthyme:tests/visual-regression

nickofthyme commented Sep 26, 2019 •

edited

Loading

Uh oh!

codecov-io commented Sep 26, 2019 •

edited

Loading

Uh oh!

markov00 commented Sep 27, 2019 •

edited

Loading

Uh oh!

nickofthyme commented Sep 27, 2019

Uh oh!

nickofthyme commented Oct 1, 2019

Uh oh!

markov00 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

markov00 left a comment

Uh oh!

markov00 commented Oct 10, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

nickofthyme commented Sep 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Add visual testing config

Add seeded random number function

Upgraded devDependencies:

Checklist

Uh oh!

codecov-io commented Sep 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

markov00 commented Sep 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nickofthyme commented Sep 27, 2019

Uh oh!

nickofthyme commented Oct 1, 2019

Uh oh!

markov00 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

markov00 left a comment

Choose a reason for hiding this comment

Uh oh!

markov00 commented Oct 10, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nickofthyme commented Sep 26, 2019 •

edited

Loading

Upgraded `devDependencies`:

codecov-io commented Sep 26, 2019 •

edited

Loading

markov00 commented Sep 27, 2019 •

edited

Loading