Add SamplerTestCase class in optuna.testing package#6424
Add SamplerTestCase class in optuna.testing package#6424sawa3030 merged 11 commits intooptuna:masterfrom
SamplerTestCase class in optuna.testing package#6424Conversation
|
@kAIto47802 Could you review this PR? |
| restored_sampler = pickle.loads(pickle.dumps(sampler)) | ||
| assert sampler._rng.rng.bytes(10) == restored_sampler._rng.rng.bytes(10) | ||
|
|
||
| def test_before_trial(self) -> None: |
There was a problem hiding this comment.
I noticed that test_before_trial, test_after_trial, test_after_trial_pruning,
test_after_trial_failing, test_after_trial_failing_in_after_trial,
test_after_trial_with_study_tell, and test_sample_relative aren’t really testing sampler
implementations themselves. I think they’re more about verifying the study’s behavior, so it feels a bit inconsistent to treat them as peers of the other
sampler-parameterized tests in TestSampler / SamplerTestCase.
Would it make sense to move these tests out of TestSampler?
There was a problem hiding this comment.
Thank you for your suggestion.
I moved the tests as follows.
test_before_trial->test_trial.py- Since
before_trialis called inTrial.
- Since
test_after_trial_*->test_study.py- Since
after_trialis called instudy/_tell.py.
- Since
|
If my understanding is correct, testing a third-party sampler currently requires implementing fixtures like this: import pytest
from optuna.testing.pytest_samplers import SamplerTestCase
class TestTurboSampler(SamplerTestCase):
@pytest.fixture
def sampler_class(self):
return TuRBOSampler()
@pytest.fixture
def relative_sampler_class(self):
pytest.skip("Relative Sampler cannot be tested since TuRBOSampler requires startup trials.")
@pytest.fixture
def multi_objective_sampler_class(self):
pytest.skip("TurboSampler does not support multi-objective optimization.")
@pytest.fixture
def single_only_sampler_class(self):
return TuRBOSampler()
Would it be possible (or reasonable) to make this more straightforward, for example: import pytest
from optuna.testing.pytest_samplers import SamplerTestCase
@pytest.fixture
def sampler():
return TurboSampler(...)
class TestTurboSampler(SamplerTestCase):
SUPPORTS_RELATIVE = False
SUPPORTS_MULTI_OBJECTIVE = False
SUPPORTS_SINGLE_ONLY = TrueI’m not entirely sure if this would fit the current design, so I’d like to take a bit of time to think through the implementation. |
|
I divided the test cases into four scenarios: from collections.abc import Callable
import pytest
import optuna
from optuna.samplers import BaseSampler
from optuna.testing.pytest_samplers import BasicSamplerTestCase
from optuna.testing.pytest_samplers import MultiObjectiveSamplerTestCase
from optuna.testing.pytest_samplers import RelativeSamplerTestCase
from optuna.testing.pytest_samplers import SingleOnlySamplerTestCase
# Example 1
class TestSampler1(BasicSamplerTestCase, MultiObjectiveSamplerTestCase):
@pytest.fixture
def sampler(self) -> Callable[[], BaseSampler]:
return optuna.samplers.TPESampler
# Example 2
class TestSampler2(RelativeSamplerTestCase):
@pytest.fixture
def sampler(self) -> Callable[[], BaseSampler]:
return lambda: optuna.samplers.TPESampler(multivariate=True, n_startup_trials=0)
# Example 3
class TestSampler3(SingleOnlySamplerTestCase):
@pytest.fixture
def sampler(self) -> Callable[[], BaseSampler]:
return optuna.samplers.CmaEsSampler
def test_user_defined(self, sampler) -> None:
# Additional test can be defined if necessary.
assert sampler.__name__ == "CmaEsSampler"What do you think? |
c-bata
left a comment
There was a problem hiding this comment.
LGTM. Thank you for your pull requests!
Motivation
Similar to #6369, this PR makes it easier for third-party sampler implementations to use the samplers' tests in Optuna.
Description of the changes
SamplerTestCaseclass inoptuna.testing.pytest_samplers.pythat implements general tests for samplers using the fixtures above.test_samplers.pywithSamplerTestCase.