Converting this to draft for now. I'd like to test that creating a warning/error in the docs build makes the check fail in an obvious way - it's not clear to me at the moment that's the case.

Not exactly what I expected: https://dev.azure.com/pyqtgraph/pyqtgraph/_build/results?buildId=1079&view=results

I guess because continueOnError: true is set, failures are reported (to github) as successes. I was thinking ideally we would see in the github interface that the docs build failed but still run the tests. This seems to imply the same problem exists for the other pre_test jobs (except build_wheel).

I think there are a few options:

• Just remove continueOnError. I'm still thinking this is not ideal since getting a PR to pass code tests is usually higher priority than docs.
• Add condition: succeededOrFailed() to the test stage and remove continueOnError for the build_docs job. This is also probably not what we want because failing the wheel build should prevent running tests it seems.
• Add the docs_build job to the test stage or to a new post_test stage. I'm leaning toward this option, though I don't know what to do about reporting issues with the other pre_test jobs.

Hey, I think this PR should definitely be merged as I don't want to find out our docs are broken just by browsing RTD...

While running the docs, I did get this warning

/Users/ogi/Developer/pyqtgraph/doc/source/conf.py:142: RemovedInSphinx40Warning: The app.add_stylesheet() is deprecated. Please use app.add_css_file() instead.

Yeah, haven't revisited this in a while. I noticed the same warning and I believe it's been fixed in sphinx_rtd_theme 0.5.0. Things like that are why I'm a little hesitant to allow docs warnings to cause full CI failure, but a lot of sphinx issues are reported only as warnings.

Any thoughts on the options above? I still don't really understand why continueOnError results in a pass for a task, but I guess I'm still leaning toward putting the docs as the last step and turning off continueOnError so it's reported as a failure. The check_diff_size and style_check tasks aren't as important right now.

Taking off WIP status as I'd like to get this merged in, but I'm looking at this a bit more, I too am concerned about azure pipelines reporting failures, but allowing the test suite to continue on...

@ixjlyons sorry it's taken me so long to read into this, but I'm thinking your second option condition: succeededOrFailed() to the test stage is probably what we want. If the wheel fails to build, we'll get an error, and we shouldn't be surprised that the tests fail then ... I'll implement this diff...

now that I got the test stage to work on succeededOrFailed() I'm actually thinking of removing that condition... primarily reason is if you get a docs/linting failure, you can usually fix it quick, but if you're having to wait out the test suite, that's ~15 minutes before the CI will give you another go (I think we have it setup to stop running if a new push comes in).

EDIT: ok, we have it setup so the CI run aborts if a new commit is pushed on that branch;...k, I'm good w/ the current succeededOrFailed();...

I could make another intermediate stage for the wheel to be build, where to run tests we need the wheel to pass, but not necessarily the other pre-test steps... I think that would satisfy all conditions,

I think this PR is ready for merging, the CI is complaining about PColorMeshItem.py which wasn't even in this branch (guess it's checking against the current status of the master branch?)

I'm going to merge this and then tackel the PColorMesh docs issue, thanks @ixjlyons for getting this PR going!