Skip to content

Reset set_by_cli between each test#9567

Merged
alexzorin merged 1 commit intomasterfrom
reset-set-by-cli
Feb 8, 2023
Merged

Reset set_by_cli between each test#9567
alexzorin merged 1 commit intomasterfrom
reset-set-by-cli

Conversation

@bmw
Copy link
Copy Markdown
Member

@bmw bmw commented Feb 8, 2023

I now believe tests started failing after #9355 was merged because the annoying and sneaky global/static variable certbot._internal.cli.set_by_cli.detector was being reset as part of _call at the beginning of each test and was not (also) being reset after each test. The tests introduced in that PR was not the only or first use of this pattern in main_test.py. I think this pattern meant that depending on the exact test order (which I believe will vary by machine/run a bit due to our use of --numprocesses auto to spread tests across processes), the value of certbot._internal.cli.set_by_cli.detector may still have been set to something when other tests ran affecting their result.

In particular, building off my explanation at the beginning of #9564, set_by_cli('authenticator') was returning True claiming the value was set to Apache, however, Apache wasn't actually requested on the command line in the failing test. I believe this caused this code to leave config.authenticator unset which then caused us to blow up later because we didn't know what plugin to use.

This PR fixes this by making use of a global pytest fixture that resets cli.set_by_cli.detector to None between every test. I initially just put this in main_test.py as its the only place we currently do this in our tests, but I personally think it's useful to do it more globally to avoid hitting this in the future in places like the unit tests for the cli module or other modules that use it such as client or renewal.

This PR introduces the first import of pytest in our shipped code although I think/hope there will be many to come and I don't think it's a problem because the Certbot test extras depend on pytest, we claimed this dependency years ago, and instruct packagers to run our tests using pytest.

The last thing I want to say in my wall of text is I think us hitting this problem is another good reason we should prioritize cleaning up our testing and/or CLI parsing.

@alexzorin
Copy link
Copy Markdown
Collaborator

If we are going to rely on pytest features from now on, do we need to do anything about:

if __name__ == '__main__':
unittest.main() # pragma: no cover

Personally I am still in the habit of occasionally running tests directly via python main_test.py, which it looks like is going to no longer work, with possibly subtle breakages.

@bmw
Copy link
Copy Markdown
Member Author

bmw commented Feb 8, 2023

Ooo nice catch! I agree we should do something here. Do you mind if we do it in another PR though so we can get tests on master working again?

Also, do you have a preference whether we just delete those lines so python main_test.py does nothing or would you like to maybe try something like this? In general, I'd prefer not to break people's workflows so I'm somewhat tempted to try it, however, I think doing this is pretty unconventional and may be more trouble than it's worth.

@alexzorin
Copy link
Copy Markdown
Collaborator

The pytest documentation does seem to offer it as a legitimate option: https://docs.pytest.org/en/7.1.x/how-to/usage.html#calling-pytest-from-python-code, but I'd be fine with deleting it too. Either way we get rid of the half-way state of "it runs but things are randomly broken".

@bmw
Copy link
Copy Markdown
Member Author

bmw commented Feb 8, 2023

Cool. I'll play with it.

I saw those docs too, but noticed that they don't pass __file__ so I think their version is the equivalent of running pytest with no arguments and will run tests on everything under your CWD.

EDIT: It's mimicking unittest's behavior that seems a bit unconventional to me.

Copy link
Copy Markdown
Collaborator

@alexzorin alexzorin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, seems to work perfectly with #9569 cherry-picked on top of it. LGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants