We should ensure that all estimators (that have parameters) possess at least two test parameter sets.
The two (or more) parameter sets should:
- be fast to run together -
fit is the bottleneck (so we should not overdo it with too many parameter sets)
- cover substantially different settings for all the important parameters, i.e., substantially different typical cases and/or important edge cases
Recipe:
- search for estimators which have parameters but only a single test parameter set. These are estimators with no
get_test_params implemented, or get_test_params returning only a single dictionary instead of a list of two (or more) dictionaries.
- post here in this issue which estimator you picked (to avoid duplication of work)
- come up with a parameter set satisfying the above constraints and add it to the return (should be list of two or more dictionaries)
- make a PR, this should add the parameter set in
get_test_params, and remove the estimator from EXCLUDED_TESTS_BY_TEST in sktime.tests._config
- if the estimator has a
"results_comparison" parameter set, ignore that - add new parameters only to the unconditional parameter set
An example PR that adds second parameter sets for some estimators can be found here: #3428
Finding some estimators that have only one parameter set can be done quickly by checking the variable EXCLUDED_TESTS_BY_TEST in sktime.tests._config, estimators which do not comply with the "number of test parameter sets" requirements are skipping "test_get_test_params_coverage".
Alternatively, locally running code which does this:
from sktime.registry import all_estimators
all_ests = all_estimators()
ests = [x for x in all_ests if (len(x[1].get_test_params())<2 or isinstance(x[1].get_test_params(), dict)) and len(x[1].get_param_names())>0]
[x[0] for x in ests if len(x[1].get_param_names()) > len(x[1].get_class_tag("reserved_params", []))]Current output:
We should ensure that all estimators (that have parameters) possess at least two test parameter sets.
The two (or more) parameter sets should:
fitis the bottleneck (so we should not overdo it with too many parameter sets)Recipe:
get_test_paramsimplemented, orget_test_paramsreturning only a single dictionary instead of a list of two (or more) dictionaries.get_test_params, and remove the estimator fromEXCLUDED_TESTS_BY_TESTinsktime.tests._config"results_comparison"parameter set, ignore that - add new parameters only to the unconditional parameter setAn example PR that adds second parameter sets for some estimators can be found here: #3428
Finding some estimators that have only one parameter set can be done quickly by checking the variable
EXCLUDED_TESTS_BY_TESTinsktime.tests._config, estimators which do not comply with the "number of test parameter sets" requirements are skipping"test_get_test_params_coverage".Alternatively, locally running code which does this:
Current output:
ClaSPTransformerCNTCNetwork,CanonicalIntervalForest,CircularBinarySegmentation,ColumnEnsembleClassifierColumnTransformerColumnwiseTransformerDOBINDWTTransformerDistFromAlignerDistanceFeaturesDontUpdateDummyRegressorFeatureSelectionGreedyGaussianSegmentationInceptionTimeNetworkIndividualBOSSIndividualTDELogTransformerMCDCNNClassifierMCDCNNNetworkMCDCNNRegressorMLPNetworkMatrixProfileOnlineEnsembleForecasterPAAPCATransformerPlateauFinderRandomIntervalFeatureExtractorRandomIntervalSegmenterRandomIntervalsRandomSamplesAugmenterReducerTransformShapeletTransformSlidingWindowSegmenterSlopeTransformerStackingForecasterSupervisedTimeSeriesForestTapNetNetworkTimeBinnerTruncationTransformerUnobservedComponentsWhiteNoiseAugmenter