What's a VCR test? And what's a golden test? I couldn't find anything from our docs on this. How do we run them?

Yeah great question!! @adeola-ak actually started a docs PR about this a while ago, but we never got around to finishing it. We should really do that!

VCR tests are, like the name suggests (yes, that VCR! Video Cassette Recorders!), tests that record HTTP interactions in order to replay them later as a way of testing that the behavior and outputs of a method that interacts with an external API stay consistent and match expectations so long as the "inputs" from the external API are the same. For our external service client methods, the things that actually send requests to code host servers, VCR tests are essential.

So the way they work is as follows:

• You create a special HTTP client that is capable of intercepting HTTP interactions and recording them to or reading them from files (the "cassettes"). In our case, we've built common utilities to enable this in the package internal/httptestutil based on a Go library for VCR testing.
• You write a test mostly as normal, invoking your method that would talk to an external API with a certain pre-determined set of inputs and then asserting things about the results. But you make sure your method is using the special HTTP client now.
  • For the tests I wrote for the GitHub V3 client, this setup is accomplished by newV3TestClient helper method.
• You run the test once in "update" mode, to record the interactions. Usually these tests will require you have a couple environment variables set from the shell environment in which you are running the tests, such as a GITHUB_TOKEN to authenticate requests to GitHub with. To target which tests you want to record, you provide an -update flag and a path to the cassette files you wish to be recorded over. The path doesn't have to be a full path, either, so for example if I ran:
  go test ./internal/extsvc/github -run TestV3Client_GetRef -update TestV3Client_GetRef
  then the test would record both TestV3Client_GetRef_success and TestV3Client_GetRef_failure.
  • This will instruct the special HTTP client to actually issue real requests to the API and then record the details of any HTTP interactions (both request and response objects) made over the course of the test as YAML files in the testdata/vcr directory.
  • It will also record a result struct of my choosing as JSON into the testdata/golden directory. This represents the output I hope to get from my method, given these precise API interactions.
• For subsequent runs, you can now omit the -update flag (so running it just like any other test), and this time the special HTTP client will "replay" the recorded interactions, rather than talking to the external API. This means we're still able to test with "real" responses, but the test is more deterministic and reliable, since it isn't subject to network instability, dynamic parts of the response (such as timestamps), or environment. The test will also make sure the result at the end of it is exactly the same as the result from testdata/golden.

Got it, that's really neat. It looks like pretty much just the batches team has been using VCR tests. I wonder if there's a standard coverage we want with VCR tests?

Yeah, I think we tend to use it the most because we have the most in quantity and most complex interactions with code hosts out of any part of the application, and often we depend very closely on the response for things to work out right for the end user in the UI. But technically these internal/extsvc packages are for all of Sourcegraph, not just Batch Changes, so there is pretty good coverage across even the non-BC client interactions at least. 😄 I think it's wise to write VCR tests for any external API interactions that we're critically dependent upon. We could definitely express that sentiment in the docs.