Swift: add consistency check and accept results for the moment #13277
Conversation
|
@geoffw0 do we need a change note for this, because of the newly added |
|
It's probably a good idea because users may notice this, yeah. If we don't expect users to want to use the consistency pack, the change note can be very brief. |
| | file://:0:0:0:0 | ... = ... | | ||
| | file://:0:0:0:0 | ... = ... | | ||
| | file://:0:0:0:0 | ... = ... | | ||
| | file://:0:0:0:0 | ... = ... | |
There was a problem hiding this comment.
How do these tests work? I can only see the .expected files in each test directory.
There was a problem hiding this comment.
I learnt yesterday there is a --consistency-queries flag to codeql test run:
--consistency-queries=<dir>
[Advanced] A directory with consistency queries that will be run for each test database. These queries should not produce any output (except when they find a problem) unless the test directory includes
a CONSISTENCY subdirectory with a .expected file. This is mostly useful for testing extractors.
The effect is that the codeql test runner will behave as if all the queries in the provided directory were present under a CONSISTENCY directory besides any test it runs. So it will produce .actual files in case of failures, you can run codeql test accept, and --learn produces the .expected files directly.
I was afraid the CI feedback time would take a hit, but actually the ql tests took 9 minutes on macOS and 2 (!) minutes on linux, which is well within the bottleneck of integration tests. So I don't think there is any harm in leaving these consistency checks on (and maybe even adding more). If this ever gets to be a problem, we can always only turn them on on linux that has CPU power to spare. |
No description provided.