[2018-11-01 11:02:29] <bmw> adferrand[m]: One thing I've been thinking about lately is how to better test Certbot's DNS plugins.
[2018-11-01 11:02:49] <bmw> I was curious how your project and Lexicon handle this. Do you maintain accounts with all these different providers?
[2018-11-01 11:30:15] <adferrand[m]> In Lexicon, we use vcrpy. It works with cassettes, which are yaml files that describes a serie of http request and their response. First, a dns provider developer executes the tests against the real dns api, because at this time, the developer has a valid account on it.
[2018-11-01 11:30:46] <adferrand[m]> During this test, cassettes are recorded, and committed in the repo.
[2018-11-01 11:31:57] <adferrand[m]> In all subsequent tests execution, in particular CI, vcrpy will mock transparently any requests emitted by python (vcrpy supports the most used http libraries).
[2018-11-01 11:32:57] <adferrand[m]> If for a given test method, with its related cassette, the request corresponds to the recorded one, vcrpy will return the recorded response.
[2018-11-01 11:46:05] <bmw> Interesting.
[2018-11-01 11:46:28] <bmw> One thing we've struggled with is how to initially test these plugins as well as to regularly run tests to ensure they still work with the provider's current API.
[2018-11-01 11:47:31] <bmw> We were planning to set up accounts, but doing this for every plugin someone writes based on Lexicon isn't very sustainable.
[2018-11-01 12:53:34] <adferrand[m]> Indeed, you cannot. And if I construct a universal lexicon plug-in, it is dozens of accounts to maintain potentially. Here with Lexicon, it is possible.
[2018-11-01 12:55:04] <adferrand[m]> The only limitation, is if an actual API changes its specifications, because cassettes are not relevant anymore, and you do not know it until someone executes again a real request. But APIs of this kind
[2018-11-01 12:55:46] <adferrand[m]> are really stable. It should occurs only from time to time, I think it is doable to correct cassettes when needed.
[2018-11-01 13:08:50] <bmw> Sure.
[2018-11-01 13:09:42] <bmw> The only thing I don't like about it is it would likely be our users who run the real request and hit the bug when things change, but maybe that's not the end of the world.
[2018-11-01 13:34:40] <adferrand[m]> Yes indeed, I was already thinking about this to make something with it. 
[2018-11-01 13:35:42] <adferrand[m]> Because currently, in fact tests on dns plugind are mocking every actual call to lexicon, right?
[2018-11-01 13:36:51] <adferrand[m]> And there is already integrations tests in Lexicon that checks an acme challenge can be written in a txt record.
[2018-11-01 13:37:49] <adferrand[m]> So I thought this particular cassette could be reused during integrations tests on dns certbot plug-in that are based on lexicon.
[2018-11-01 13:38:43] <adferrand[m]> Txt deletion is not done on Lexicon, but this could stay a complete mock of Lexicon call, it is not critical.
[2018-11-01 13:39:14] <adferrand[m]> (I mean a test on txt deletion)
[2018-11-01 13:41:30] <adferrand[m]> And anyway, about your concerns on regressions that could be found only by users. It is already the case in fact, as everything is mocked, so you do not lose a lot in fact.
[2018-11-01 13:42:23] <adferrand[m]> At least with vcrpy, you will execute actual code of Lexicon providers code, and find regressions concerning the interface between Lexicon and Certbot.