All of our test farm tests (the test_*.sh scripts in https://github.com/certbot/certbot/tree/08cea381c8fa9e2f57de8066a8a646dde0062a9b/tests/letstest/scripts) rely on certbot-auto to some degree. Before we deprecate certbot-auto on any tested platforms, we'll need to update the tests to stop using certbot-auto.

For tests specifically for certbot-auto, I think we should leave them unchanged but only run them on systems where certbot-auto is still supported. To do this, I think we should make something like an auto_targets.yml file in tests/letstest containing only AMIs where certbot-auto is still supported. We'd then need to update the release instructions and Azure/tox to make sure this file is used.

For the other tests, I think we should rewrite them in a way not to use certbot-auto. To do this, I'd recommend creating a script in tests/letstest that bootstraps Certbot on the OSes we run tests on. This is only currently only Debian and Red Hat based OSes and the logic will probably have to be somewhat similar to certbot-auto, although we should install Python 3. I'd use a combination of the packages installed at https://certbot.eff.org/docs/contributing.html#running-a-local-copy-of-the-client and the logic in certbot-auto as a starting point. The instructions in our dev docs may not work out of the box on older, LTS systems.

This script, or at least part of it, would need to be written in bash in order to install Python. This script could then be reused in all of the other test scripts. (We could probably also easily extend this script in the future to be used as part of our developer instructions, but for now, I think we should just focus on the test farm test use case and maybe extend it later.)

(Another option here would be to create new AMIs for each OS which has dependencies already installed, but I think that'll be a lot more work in both the short and long term than writing a script.)