Starting on Saturday, Windows unit tests started sometimes failing. Debugging this was tricky since the test failure was intermittent, only happening on Windows, and the unit test in question has all of stdin, stdout, and stderr mocked out making it hard to use a debugger. I think this all reenforces the idea we've been talking about lately that Certbot testing should be saner/easier.

Check out the Azure Pipeline history if you're curious what all I tried, but the first thing I noticed is that the logging messages in the test failure are very odd. https://dev.azure.com/certbot/certbot/_build/results?buildId=6339&view=logs&j=ed6d8c23-2913-5760-0bd3-640c32c32198&t=f8e8d435-fc2b-5e49-c7a1-d4820401e523&l=958 says the Apache plugin was selected and then https://dev.azure.com/certbot/certbot/_build/results?buildId=6339&view=logs&j=ed6d8c23-2913-5760-0bd3-640c32c32198&t=f8e8d435-fc2b-5e49-c7a1-d4820401e523&l=1038 says we crashed because a plugin wasn't specified. This is especially odd because the failing test doesn't request Apache and the renewal configuration file used in that test specifies standalone.

Checking the commit history, I saw that #9355 landed on Friday (the day before test failures started) and does some mocking causing Apache to be selected. While continuing to dig around, I made a branch that reverted this commit and ran the tests successfully 5 times which is noteworthy because the tests otherwise almost always fails. You can see this at in the Pipeline history where the branch being tested was test-windows-failure-no-reconfigure.

Something/everything below this line is wrong

After noticing that, I looked into how the Apache mocks added in that PR get cleaned up and noticed that we called super().tearDown() before doing our own tear down. Out of the tests in this file using this pattern of creating mocks in setUp and cleaning them up in tearDown, only this test does this. I tried flipping the order and successfully ran the test three times at https://dev.azure.com/certbot/certbot/_build/results?buildId=6349&view=results. (I kept canceling the build after the Certbot unit tests passed and immediately rerunning it.)

It's not immediately clear to me why this matters as the default implementation of unittest.tearDown does nothing, however, I suspect that some combination of things like pytest and pytest-xdist hook into or modify the standard unittest.TestCase.tearDown method and by calling it first, our cleanup code isn't always properly being called. pytest does have code modifying tearDown methods although that particular line of code seems unlikely to be the offender to me.

Regardless of the exact cause here, this seems to reliably fix it as I have not been able to reproduce the failure with this change.