Visual Regression Tests allow you to quickly generate png images of all samples, and automatically detect visual changes when you made changes to OSMD. You can simply get a visual diff of your current generated images to the 'blessed' ones from a known good state. Now we don't have to manually detect changes in many samples anymore.

Grey is what both images have in common (unchanged). Red is what the blessed image has and is missing in the current image. Black (with a red border) is what the current image has and didn't exist in the blessed image.

The procedure is explained in visual_regression.sh: (Linux/Mac only for now)

# one line testing command (assuming you already have blessed images (run npm generate:blessed)):
npm run test:visual:build

# alternatively, run the steps one by one:

# set your OSMD to the previous state to compare to, e.g. by checking out develop in a new folder
# run npm install if you haven't yet done so (only needs to be done once per package.json change)

# need to run an actual build every time if using browserless version, npm start doesn't update this.
npm run build
# npm run build:webpack # skip linting
npm run generate:blessed

# then apply your changes (switch branch) and do this:
npm run build
npm run generate:current
# you can also use faster skyline calculation methods, though in rare cases the results may differ slightly:
# npm run generate:blessed:webgl # needs optional gl package installed
# generate:current:batch

# run visual regression tests
npm run test:visual
# this can take 3+ minutes
# Linux/Mac/WSL2 only for now

Note: the visual regression test script doesn't work on Windows without WSL, and not in WSL1, but it seems to work in WSL2. [I personally haven't upgraded from WSL1 to WSL2, because it broke rsync for me, so I use a virtual machine (VirtualBox) with Linux for regression tests - sschmidTU]

Now you'll get reports for visual differences in the console, and there will be diff and current/blessed images in the visual_regression/diff folder if there were changes above threshold.

You can also run visual tests for single samples:

node test/Util/generateImages_browserless.js ../../build ./test/data ./visual_regression/current 0 0 mySampleFileNameRegExPrefix.* # 0 0 width and height makes the page endless, not dividing it into multiple pages/files
sh ./test/Util/visual_regression.sh ./visual_regression mySampleFileNamePrefix
# note that javascript expects a RegEx, while the shell script accepts a prefix x that will match 'x*' in the shell. We may use RegExes for both later on.

ps: the code is adapted with few core changes from Vexflow, so thanks to them! The bigger challenge was generating pngs in the first place.

See #676, where this wiki entry is adapted from, for more details and related code/commits.

Note: if you run into difficulties with the npm task, it may be due to running with 'sh'. You might need to run in bash directly (from project root):

chmod +x ./test/Util/visual_regression.sh
./test/Util/visual_regression.sh ./visual_regression

We have a test suite of about 150 tests using Karma, Mocha and Chai. You can find these in the test/ folder.

• npm test executes all tests.

If you get the error No binary for ChromeHeadless browser on your platform., you need to install Chrome or Chromium, then set e.g. export CHROME_BIN='/usr/bin/chromium'.
Note that if you're using Windows and OSMD is not on the C partition, npm test will currently probably fail due to bugs in karma-webpack 5.0.0. - try to move OSMD to the C partition, or use a Linux VM to run the tests.

These tests check that all major parts of OSMD (still) do what they are supposed to do.
Ideally you should run these for every major code change before commiting.

Our continuous integration frameworks also run these for every commit (and PR) to the repository.

If you add major functionality, it would be nice to also add tests for it.

No test should fail. Except sometimes a timeout may be too short, we're looking to tune that.

To run only a single specific test, open the test file, and replace it.("myTest") with it.only("myTest").
Or use describe.only for a whole test suite.

Then run npm test.

Of course, this change should later be undone and never committed.
You can also temporarily skip a test with it.skip("test name"), but that should ideally not be committed either.

1. Be sure that you have the right to publish the xml file (check for copyright).
2. add the xml file to /test/data
3. in /test/Common/FileIO/Xml_Test.ts add the filename to the xmlTestset: string[].