-
-
Notifications
You must be signed in to change notification settings - Fork 346
Visual Regression Tests implemented #676
Description
I just implemented Visual Regression Testing, allowing you to quickly generate png images, and automatically detect visual changes in OSMD renders of all samples between code changes.
This is great for testing changes to commit. Now we don't have to manually check lots of samples visually anymore.
Grey is what both images have in common (unchanged).
Red is what the blessed image has and is missing in the current image.
Black is what the current image has and didn't exist in the blessed image.
This way I discovered that I introduced a bug that prevented notes from being invisible/transparent (the duplicate rests), while some stems were invisible (see above).
So when you make changes to OSMD, you can automatically find out whether there were good or bad side effects.
The procedure is explained in visual_regression.sh: (Linux/Mac only for now)
# do OSMD visual regression tests: (Linux/Mac only for now)
# clean images if you ran the tests in the past:
rm -rf visual_regression/blessed visual_regression/current
# set your OSMD to the previous state to compare to, e.g. by checking out develop in a new folder
npm run build # if necessary
npm run generate:blessed
# this can take 1-2 minutes
# then apply your changes and do this:
npm run build
npm run generate:current
# run visual regression tests, creating images in visual_regression/diff.
# this can take 3+ minutes
npm run test:visualIf there are no visual regressions, you can skip deleting and regenerating the blessed images next time, or if there are new samples, you can update the blessed images:
rm -rf visual_regression/blessed
mv visual_regression/current visual_regression/blessedNow you'll get reports for visual differences in the console, and there will be diff and current/blessed images in the visual_regression/diff folder if there were changes above threshold.
You can also run visual tests for single samples: [edit: this is out of date, use generateImages_browserless.js]
node test/Util/generateDiffImagesPuppeteerLocalhost.js ./test/data ./visual_regression/current mySampleFileNameRegExPrefix.*
sh ./test/Util/visual_regression.sh ./visual_regression mySampleFileNamePrefix
# note that javascript expects a RegEx, while the shell script accepts a prefix x that will match 'x*' in the shell. We may use RegExes for both later on.ps: the code is adapted with few core changes from Vexflow, so thanks to them! The bigger challenge was generating pngs in the first place.
