For the customer use case, it's nice to have a flexible tool for this, but I worry that it's too complex/time-intensive to expect them to write their own tests. Would we ship this with a built-in test suite that we recommend?

For the customer use case, it's nice to have a flexible tool for this, but I worry that it's too complex/time-intensive to expect them to write their own tests. Would we ship this with a built-in test suite that we recommend?

We will provide examples that customers can edit and insert their own repo, token, external services details. Does that address your concerns ? Customers don't need to use this if they don't feel a need for it. But I think the simple syntax and the provided docs and examples will make this not be an issue imho.

This is great stuff! But I have the similar concern as @dadlerj that writing this kind of tests actually requires the customer/admin being proficient of how Sourcegraph actually works internally?

@unknwon i'm not sure i understand your comment: why would admins need to know the internals. the scripting only exposes operations that one would do in the UI or in the graphql API console.

My POV:

As long as (eventually) no customers need to learn this command language, I'm happy. If the user experience is src validate <default-smoke-test-script> that they can run after upgrading, then I'd be thrilled! It's nice if really active/enterprising customers can then author their own to expand it, but they shouldn't have to author their own to get a basic test done.

@dadlerj the problem is that they do have to provide which repos they want to add and which search queries they want to execute (etc). this varies from customer to customer. we could have them all use one of our sourcegraph-testing repos, but some of the customers don't even want or can't access github.com. they also need to provide their own github token even if they do have access to github.com.

that's why i thought this is a good compromise. it's simple, restricted enough but a good place to pass this information into the validation. and flexible enough that customers can use their own setup and tokens and decide which verifications and steps are important to them.

we could narrow it down even further with them providing a repo name, the github token etc in a config file. but explaining and documenting these pieces of information gets you almost to this scripting.

I support the idea of enabling customers to write their own validation scripts to exercise Sourcegraph after updating and think this is a great start. Here are my concerns:

• Is skylark the right language to commit to? I don't know anything about it beyond that it's Python-like and it's the language Google uses for Blaze/Bazel. It does not seem widely used outside of that.
• Should we first test this out more thoroughly ourselves before asking customers to invest time in writing their own validation scripts? Once customers start using it, we are on the hook for supporting it, which means maintaining it in the long-run or providing excellent assistance for migrating customers off of it if we abandon it in the future.

If the goal here is just to rapidly prototype this as a proof-of-concept and use it ourselves, then I would be okay approving this under the condition that it is very very clear that it is experimental and unsupported, and it could disappear or change in a breaking fashion at any time.

I would also like to see more examples of tests written using this subcommand that we use ourselves in our testing/release pipeline. That would help me determine if this is worth shipping to customers.

@beyang i agree with all your points. this is very much experimental. i needed a vehicle for deployment and chose src-cli. customers won't hear about it until we're sure we want to go down this road. if they discover it by accident then it's clearly marked as experimental. we won't advertise it.

i'm also not sure about the scripting language choice. i wanted something simple with a familiar syntax. we can plug in a different choice without too much effort.

i will use deploy-sourcegraph as the use case to kick the tires on it and test the approach.

@uwedeportivo I understand that the usefulness is reduced in what you intended for this tool, but I think that falls outside of a validation (like goss, inspec, serverspec, etc) and more running a live test on the machine.

I see value in that live test, but I would say we should be quite verbose about that, like warn that this will add and remove config to ensure no one creates an unintended side effect.

On the code itself, if possible I would reuse the code (by extracting it from the command) we already have for creating repos, removing repos, etc.

sounds good. i'll call it out more and try to re-use what's there (there isn't much for adding and removing)

What's the status here? I get pinged by the GitHub bot in Slack about this PR :)

@mrnugget it landed :-)

@uwedeportivo Can you add a CHANGELOG entry?