Skip to content

Add SamplerTestCase class in optuna.testing package#6424

Merged
sawa3030 merged 11 commits intooptuna:masterfrom
y0z:feature/pytest_samplers
Feb 12, 2026
Merged

Add SamplerTestCase class in optuna.testing package#6424
sawa3030 merged 11 commits intooptuna:masterfrom
y0z:feature/pytest_samplers

Conversation

@y0z
Copy link
Copy Markdown
Member

@y0z y0z commented Jan 27, 2026

Motivation

Similar to #6369, this PR makes it easier for third-party sampler implementations to use the samplers' tests in Optuna.

Description of the changes

  • Introduce several fixtures to categorize samplers, such as those supporting relative sampling, multi-objective optimization, etc.
  • Introduce the SamplerTestCase class in optuna.testing.pytest_samplers.py that implements general tests for samplers using the fixtures above.
  • Refactoring test_samplers.py with SamplerTestCase.

@y0z y0z added the optuna.testing Related to the `optuna.testing` submodule. This is automatically labeled by github-actions. label Jan 27, 2026
@c-bata
Copy link
Copy Markdown
Member

c-bata commented Jan 27, 2026

@kAIto47802 Could you review this PR?

@y0z y0z assigned sawa3030 and unassigned kAIto47802 Jan 29, 2026
restored_sampler = pickle.loads(pickle.dumps(sampler))
assert sampler._rng.rng.bytes(10) == restored_sampler._rng.rng.bytes(10)

def test_before_trial(self) -> None:
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed that test_before_trial, test_after_trial, test_after_trial_pruning,
test_after_trial_failing, test_after_trial_failing_in_after_trial,
test_after_trial_with_study_tell, and test_sample_relative aren’t really testing sampler
implementations themselves. I think they’re more about verifying the study’s behavior, so it feels a bit inconsistent to treat them as peers of the other
sampler-parameterized tests in TestSampler / SamplerTestCase.

Would it make sense to move these tests out of TestSampler?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your suggestion.
I moved the tests as follows.

  • test_before_trial -> test_trial.py
    • Since before_trial is called in Trial.
  • test_after_trial_* -> test_study.py

@sawa3030
Copy link
Copy Markdown
Collaborator

sawa3030 commented Jan 30, 2026

If my understanding is correct, testing a third-party sampler currently requires implementing fixtures like this:

import pytest
from optuna.testing.pytest_samplers import SamplerTestCase

class TestTurboSampler(SamplerTestCase):
    @pytest.fixture
    def sampler_class(self):
        return TuRBOSampler()

    @pytest.fixture
    def relative_sampler_class(self):
        pytest.skip("Relative Sampler cannot be tested since TuRBOSampler requires startup trials.")
    
    @pytest.fixture
    def multi_objective_sampler_class(self):
        pytest.skip("TurboSampler does not support multi-objective optimization.")

    @pytest.fixture
    def single_only_sampler_class(self):
        return TuRBOSampler()
    

Would it be possible (or reasonable) to make this more straightforward, for example:

import pytest
from optuna.testing.pytest_samplers import SamplerTestCase

@pytest.fixture
def sampler():
    return TurboSampler(...)

class TestTurboSampler(SamplerTestCase):
    SUPPORTS_RELATIVE = False 
    SUPPORTS_MULTI_OBJECTIVE = False
    SUPPORTS_SINGLE_ONLY = True

I’m not entirely sure if this would fit the current design, so I’d like to take a bit of time to think through the implementation.

@y0z
Copy link
Copy Markdown
Member Author

y0z commented Feb 3, 2026

I divided the test cases into four scenarios: BasicSamplerTestCase, MultiObjectiveSamplerTestCase, RelativeSamplerTestCase, and SingleOnlySamplerTestCase. Third-party developers can easily use the test cases as follows by defining a sampler fixture and inheriting the scenarios they need.

from collections.abc import Callable
import pytest

import optuna
from optuna.samplers import BaseSampler
from optuna.testing.pytest_samplers import BasicSamplerTestCase
from optuna.testing.pytest_samplers import MultiObjectiveSamplerTestCase
from optuna.testing.pytest_samplers import RelativeSamplerTestCase
from optuna.testing.pytest_samplers import SingleOnlySamplerTestCase

# Example 1
class TestSampler1(BasicSamplerTestCase, MultiObjectiveSamplerTestCase):
    @pytest.fixture
    def sampler(self) -> Callable[[], BaseSampler]:
        return optuna.samplers.TPESampler

# Example 2
class TestSampler2(RelativeSamplerTestCase):
    @pytest.fixture
    def sampler(self) -> Callable[[], BaseSampler]:
        return lambda: optuna.samplers.TPESampler(multivariate=True, n_startup_trials=0)

# Example 3
class TestSampler3(SingleOnlySamplerTestCase):
    @pytest.fixture
    def sampler(self) -> Callable[[], BaseSampler]:
        return optuna.samplers.CmaEsSampler

    def test_user_defined(self, sampler) -> None:
        # Additional test can be defined if necessary.
        assert sampler.__name__ == "CmaEsSampler"

What do you think?

@y0z
Copy link
Copy Markdown
Member Author

y0z commented Feb 9, 2026

@c-bata @sawa3030

I've split the changes into two PRs. Please review and merge the first one (#6429) before reviewing this PR.

Copy link
Copy Markdown
Member

@c-bata c-bata left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thank you for your pull requests!

@c-bata c-bata removed their assignment Feb 12, 2026
@c-bata c-bata added the test Unit test. label Feb 12, 2026
@c-bata c-bata added this to the v4.8.0 milestone Feb 12, 2026
Copy link
Copy Markdown
Collaborator

@sawa3030 sawa3030 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@sawa3030 sawa3030 removed their assignment Feb 12, 2026
@sawa3030 sawa3030 merged commit 8d98bf3 into optuna:master Feb 12, 2026
13 checks passed
@y0z y0z deleted the feature/pytest_samplers branch February 12, 2026 06:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optuna.testing Related to the `optuna.testing` submodule. This is automatically labeled by github-actions. test Unit test.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants