I think the confusion is caused by us only half-way implementing #11095. I would personally prefer if we added node scripts/ for each of the test styles rather than adding more yarn scripts...

@spalger I guess I'm a little disappointed to see that we're choosing a non standard route.

I have nothing against executing tests via node scripts, but discoverability nears zero when we don't list these scripts in package.json. Having everything listed under scripts in package.json gives me a single place for me to see what I can run, we can use naming patterns to group things (e.g. test: prefixes shows me at a glance which kinds of tests I can execute), and Matt's example of using a comment to describe a script really helps explain the role of each script to me. As Matt points out, it's also a common convention that most JS developers are familiar with.

Can I suggest that we create parity between all node scripts and package.json scripts by creating a script alias for each existing node script, and as we add node scripts in the future we add a script alias to package.json?

I would also add that integration with things like VSCode for automating test running is gained very easily with npm scripts. Just a developer DX consideration 😊

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

// cc @silne30

Is the problem here primarily documentation? It sounds like having them scripts defined in the package.json is a form of documentation and discoverability.

I am fine with CJ's suggestion of creating aliases for the node scripts, but I don't think we need to have parity with each one. There are some which aren't common enough to justify clobbering run scripts and therefore reducing the discoverability. We should really be focused on improving the documentation and not relying on something like package.json where comments are not even supported.

I think the issue here is a matter of package.json being a standard. Needing to have docs around a nonstandard way of doing things is fine so long as there in value in doing things a non-standard way.
As for not needing parity... then a choice needs to be made for each script, one often based on a given persons opinion and perspective at that moment. I think exposing things here is better. If it's too much noise, perhaps that's a sort of code smell of our systems, not of the exposing of them? Just thinking out loud here 😊

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

@tylersmalley I think I've addressed your concerns.