[experiment] Disable all property-based tests#8575
[experiment] Disable all property-based tests#8575sholderbach wants to merge 3 commits intonushell:mainfrom
Conversation
Primarily to study the impact on the coverage I disabled the property-based tests. Touches `quickcheck` and `proptest` tests (do the same thing slightly differently under the hood.) For now just commented out. If we can stop the flukes and keep the general coverage we would want to move the proptests into a separate scheduled fuzz run.
dare you to `-- -D warnings`
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #8575 +/- ##
=======================================
Coverage 68.50% 68.51%
=======================================
Files 624 624
Lines 100662 100662
=======================================
+ Hits 68962 68971 +9
+ Misses 31700 31691 -9 |
See if the faster quickcheck test is providing the observed coverage
|
Caveat: the indirect coverage tracking aof the parser didn't provide me with a nice diff on codecov.io. What we do with the smaller parser proptest is possibly a bit separate (its removal appeared to show a small drop in coverage) Longer term goal: move all fuzzing into a separate runner that runs periodically. I am not sure if we can achieve that with feature gating (as the dev-dependencies shouldn't be compiled when not needed) alternatively we need to move that to a separate unpublished crate and only use public API surface, |
The two tests `to_nuon_from_nuon` and `to_nuon_from_nuon_string` were taking multiple seconds and have since been superseded by more explicit unit tests. Compared to the time cost for devs and CI they seldomly returned explicit problems. One failure only popped up after months, as a sampled failure (nushell#7564). Fuzzing should move to a separate worker and be removed from the main test suite. See nushell#8575 for experimentation around the impact on our test coverage.
# Description The two tests `to_nuon_from_nuon` and `to_nuon_from_nuon_string` were taking multiple seconds and have since been superseded by more explicit unit tests. Compared to the time cost for devs and CI they seldomly returned explicit problems. One failure only popped up after months, as a sampled failure (#7564). # User-Facing Changes none # Tests + Formatting Fuzzing should move to a separate worker and be removed from the main test suite. See #8575 for experimentation around the impact on our test coverage.
Description
Primarily to study the impact on the coverage I disabled the
property-based tests.
Touches
quickcheckandproptesttests (do the same thing slightlydifferently under the hood.)
For now just commented out. If we can stop the flukes and keep the
general coverage we would want to move the proptests into a separate
scheduled fuzz run.
User-Facing Changes
None
Tests + Formatting
Yeet the proptests from the earth (or the regularly run CI/coverage tests)