[BUG] KNN with msm distance give exact same scores with different hyper-parameters #589
Description
Describe the bug
I am running a cross validation using KNN with msm distance on the GunPoint dataset.
I added the c hyper-parameter (a metric specific hyper-parameter) hoping that it might help attain better results. But I get identical results for all the cross validation runs on all split runs.
To Reproduce
Run
from sklearn.model_selection import RandomizedSearchCV
from sktime.classification.distance_based._time_series_neighbors import KNeighborsTimeSeriesClassifier
import pandas as pd
clf = KNeighborsTimeSeriesClassifier(n_neighbors=1, metric='msm')
hyper_param = {
'algorithm': ['brute'],
'weights': ['uniform', 'distance'],
'metric_params': [{'c': 0.01},{'c': 0.1},{'c': 1},{'c': 10},{'c': 100}]
}
search = RandomizedSearchCV(
estimator=clf,
param_distributions= hyper_param,
n_iter= 50,
scoring='balanced_accuracy',
n_jobs=-1,
random_state=0,
verbose=0,
cv=5
)
search.fit(X_train, y_train)
print(pd.DataFrame(search.cv_results_))
mean_fit_time std_fit_time mean_score_time std_score_time param_weights param_metric_params ... split2_test_score split3_test_score split4_test_score mean_test_score std_test_score
0 0.012613 0.006310 0.018742 0.006238 uniform {'c': 0.01} ... 0.6 0.5 0.4 0.468333 0.10651
1 0.021862 0.007658 0.015628 0.009887 distance {'c': 0.01} ... 0.6 0.5 0.4 0.468333 0.10651
2 0.009365 0.007647 0.021869 0.012500 uniform {'c': 0.1} ... 0.6 0.5 0.4 0.468333 0.10651
3 0.015631 0.009879 0.006250 0.007655 distance {'c': 0.1} ... 0.6 0.5 0.4 0.468333 0.10651
4 0.006544 0.008027 0.012788 0.006418 uniform {'c': 1} ... 0.6 0.5 0.4 0.468333 0.10651
5 0.007685 0.006995 0.007687 0.006998 distance {'c': 1} ... 0.6 0.5 0.4 0.468333 0.10651
6 0.003130 0.006259 0.009377 0.007657 uniform {'c': 10} ... 0.6 0.5 0.4 0.468333 0.10651
7 0.006248 0.007652 0.003125 0.006250 distance {'c': 10} ... 0.6 0.5 0.4 0.468333 0.10651
8 0.006246 0.007650 0.012500 0.006250 uniform {'c': 100} ... 0.6 0.5 0.4 0.468333 0.10651
9 0.003124 0.006248 0.009379 0.007658 distance {'c': 100} ... 0.6 0.5 0.4 0.468333 0.10651
[10 rows x 15 columns]Expected behavior
Results of training process should not be identical
Additional context
Versions
Details
System:
python: 3.7.4 (tags/v3.7.4:e09359112e, Jul 8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)]
executable:~.virtualenvs\Code-h9r8g-fJ\Scripts\python.exe
machine: Windows-10-10.0.18362-SP0
Python dependencies:
pip: 20.3.3
setuptools: 51.1.1
sklearn: 0.24.0
sktime: 0.5.0
statsmodels: 0.12.1
numpy: 1.19.4
scipy: 1.5.4
Cython: 0.29.21
pandas: 1.2.0
matplotlib: 3.3.3
joblib: 1.0.0
numba: 0.52.0
pmdarima: None
tsfresh: 0.17.0