Implementation Agnostic Testing#1946
Conversation
|
It seems this PR's develop has diverged from Edit: it seems fixing the history was simple. |
18c5ff4 to
aaec9f9
Compare
* CLI application args * Outline of test execution (test file parsing, testing driver, etc) * Test results/reporting not implemented yet * Test file coverage not implemented yet
|
Progress update: I have implemented a "coverage" subcommand to show the spec test coverage against the spec document. |
|
Progress update: The tester is now runnable, however I do not have a working test file yet. Below is a screenshot of the report generated by the script: The script returns a list of XPath-like selections that yielded incorrect results. |
wendellpiez
left a comment
There was a problem hiding this comment.
Eager to see this work! very cool.
| The specification format will remain unchanged for now. | ||
| There is an argument for the format to be replaced or simplified in the future, but the use of `@id` attributes for sections and requirements make linking a test to a example simple. | ||
|
|
||
| ### Test Suite Data Format |
There was a problem hiding this comment.
Non-blocking: as an aside, did you consider/review the Test Anything Protocol (TAP) or any similar efforts while designing the MVP JSON test data format you are proposing?
There was a problem hiding this comment.
I have not heard of this protocol, though it looks like compelling follow-up work. The format seems simple enough to adapt the current output to (perhaps behind a CLI flag). The current output format is arbitrarily chosen as it's "easy enough" to read both as a human and to grep for basic automation.
Compton-US
left a comment
There was a problem hiding this comment.
I think this is worth moving forward and trying, noting the comment that the JSON schema for the tests would be a future specification.
|
@Compton-NIST how about a little metaschema for the test harness format? We don't really need an XML capability (since XSLT can read JSON if need be) but it might be a fun little exercise. |
* Deleted duplicate `metaschema_datatypes` file * Added spec test adr and prototype spec test file * Spec test harness and minimal example
Committer Notes
Fixes #1771
Side-effects:
@levelto a few spec requirementsAll Submissions:
By submitting a pull request, you are agreeing to provide this contribution under the CC0 1.0 Universal public domain dedication.
(For reviewers: The wiki has guidance on code review and overall issue review for completeness.)
Changes to Core Features: