These tests can be see at https://github.com/certbot/certbot/tree/master/tests/letstest and in my and others opinions, they have some issues:
- They rely on Amazon EC2 and until recently, our specific AWS account. I think it'd be better to either use Digital Ocean which kindly donates resources to EFF or a free and/or hosting provider agnostic solution like Docker or Vagrant.
- They are slow and fail spuriously all the time. Failures are often due to being unable to create more EC2 instances for your account, but you'd never know that. Error is
NameError: name 'instances' is not defined.
Ideally, I think we should try to use software written by someone else for this if it exists.
If not, we can come up with our own solution and there's tradeoffs to be made between speed, which OSes we can test on, cost, etc. We eventually would like to run these tests in Travis (see #2189).
For what it's worth, a long time ago, I was given a time box by my manager at the time to try and rewrite these to use Docker. Unfortunately, I ran out of time, but that work can be seen at https://github.com/certbot/certbot/commits/letstest2. They were largely finished, but I also became less sure that was the correct approach. By using Docker, you have to download the docker image for every OS you want to test against plus the boulder (or pebble) image. This download was in the gigabytes. Perhaps this would work in Travis with our test-everything branch, but I'm not sure.
@joohoi, I believe you've thought about this issue so if you have thoughts, please share when you get the time.
@adferrand, this also seems to me to be right up your alley so if you have thoughts I'd be especially interested in them as well.
These tests can be see at https://github.com/certbot/certbot/tree/master/tests/letstest and in my and others opinions, they have some issues:
NameError: name 'instances' is not defined.Ideally, I think we should try to use software written by someone else for this if it exists.
If not, we can come up with our own solution and there's tradeoffs to be made between speed, which OSes we can test on, cost, etc. We eventually would like to run these tests in Travis (see #2189).
For what it's worth, a long time ago, I was given a time box by my manager at the time to try and rewrite these to use Docker. Unfortunately, I ran out of time, but that work can be seen at https://github.com/certbot/certbot/commits/letstest2. They were largely finished, but I also became less sure that was the correct approach. By using Docker, you have to download the docker image for every OS you want to test against plus the boulder (or pebble) image. This download was in the gigabytes. Perhaps this would work in Travis with our test-everything branch, but I'm not sure.
@joohoi, I believe you've thought about this issue so if you have thoughts, please share when you get the time.
@adferrand, this also seems to me to be right up your alley so if you have thoughts I'd be especially interested in them as well.