Skip to content

Refactor SearchSpace in GP#6197

Merged
y0z merged 28 commits intooptuna:masterfrom
kAIto47802:refactor-gp-search-space
Jul 31, 2025
Merged

Refactor SearchSpace in GP#6197
y0z merged 28 commits intooptuna:masterfrom
kAIto47802:refactor-gp-search-space

Conversation

@kAIto47802
Copy link
Copy Markdown
Collaborator

@kAIto47802 kAIto47802 commented Jul 9, 2025

Motivation

Note

This change is necessary to merge multi-objective constrained optimization in a clean manner.

Currently, in the SearchSpace used for GP, variables such as scale_types, bounds, and step are being passed around in too many places. Furthermore, optuna/_gp/search_space.py mixes functions that operate on individual dimensions with those that handle the entire space. This makes it difficult to understand the code and maintain it.

To address this, I refactor the code in optuna/_gp/search_space.py by turning the scattered functions into methods of SearchSpace.

Description of the changes

  • Turn the scattered functions into methods of SearchSpace, making the code more organized and easier to maintain.

Note

Since the search space itself and the sampled points within the space are conceptually distinct, I designed the implementation so that points are passed explicitly as arguments to methods, such as get_normalized_params and get_unnormalized_params.

Sanity check

I verified that the original master (c44f263633ef01c08e44c8fbfdced85fb4131fb7) branch and this PR (58b31be40a5b3f72b02c2a85de05bfe0783306fc) produce exactly the same output using the following script:

from collections.abc import Callable, Sequence

import numpy as np

import optuna
from optuna.study._study_direction import StudyDirection
from optuna.study.study import ObjectiveFuncType


def single_objective(trial: optuna.Trial) -> float:
    return sum(multi_objective(trial))


def multi_objective(trial: optuna.Trial) -> tuple[float, float]:
    a = trial.suggest_float("a", -10.0, 20.0)
    b = trial.suggest_float("b", 1.0, 100.0, log=True)
    c = trial.suggest_float("c", -10.0, 20.0, step=0.1)
    d = trial.suggest_int("d", 3, 10)
    e = trial.suggest_int("e", 1, 100, log=True)
    f = trial.suggest_categorical("f", ["x", "y", "z"])
    trial.set_user_attr("a+b", a + b)
    trial.set_user_attr("c+d", c + d)
    return a + b + c, d + e + ord(f)


def constraints(trial: optuna.trial.FrozenTrial) -> tuple[float, float]:
    ab = trial.user_attrs["a+b"]
    cd = trial.user_attrs["c+d"]
    return ab - 10.0, cd - 5.0


def experiment(
    objective: ObjectiveFuncType,
    directions: Sequence[str | StudyDirection] | None = None,
    constraints_func: Callable[[optuna.trial.FrozenTrial], Sequence[float]] | None = None,
) -> None:
    sampler = optuna.samplers.GPSampler(
        n_startup_trials=1, seed=42, constraints_func=constraints_func
    )
    study = optuna.create_study(sampler=sampler, directions=directions)
    study.optimize(objective, n_trials=30)

    for trial in study.trials:
        print(f"Trial {trial.number}: {trial.values} (params: {trial.params})")


def main() -> None:
    print("Single Objective Optimization:")
    experiment(single_objective)
    print("Constrained Optimization:")
    experiment(single_objective, constraints_func=constraints)
    print("Multi-objective Optimization:")
    experiment(multi_objective, directions=["minimize"] * 2)


if __name__ == "__main__":
    main()
#!/bin/bash

git switch master
master=$(git rev-parse HEAD)
python check.py > ${master}.txt

git switch refactor-gp-search-space
refactor=$(git rev-parse HEAD)
python check.py > ${refactor}.txt

if cmp -s ${master}.txt ${refactor}.txt; then
    echo "Both branches produce the same output."
    exit 0
else
    echo "The outputs differ between the branches."
    exit 1
fi

@kAIto47802 kAIto47802 changed the title Refactor gp search space Refactor SearchSpace in GP Jul 9, 2025
@kAIto47802 kAIto47802 marked this pull request as draft July 9, 2025 07:44
@kAIto47802 kAIto47802 force-pushed the refactor-gp-search-space branch from c65253b to e9198d8 Compare July 9, 2025 07:48
@kAIto47802 kAIto47802 force-pushed the refactor-gp-search-space branch from e9198d8 to fac7ad2 Compare July 9, 2025 10:11
@kAIto47802 kAIto47802 force-pushed the refactor-gp-search-space branch from ead2e64 to 9b3e191 Compare July 9, 2025 10:43
kAIto47802 and others added 2 commits July 11, 2025 17:34
Co-authored-by: Shuhei Watanabe <47781922+nabenabe0928@users.noreply.github.com>
@nabenabe0928 nabenabe0928 added the code-fix Change that does not change the behavior, such as code refactoring. label Jul 13, 2025
@github-actions
Copy link
Copy Markdown
Contributor

This pull request has not seen any recent activity.

@github-actions github-actions bot added the stale Exempt from stale bot labeling. label Jul 20, 2025
@nabenabe0928 nabenabe0928 removed the stale Exempt from stale bot labeling. label Jul 23, 2025
kAIto47802 and others added 5 commits July 23, 2025 18:21
Co-authored-by: Shuhei Watanabe <47781922+nabenabe0928@users.noreply.github.com>
Co-authored-by: Shuhei Watanabe <47781922+nabenabe0928@users.noreply.github.com>
@kAIto47802 kAIto47802 marked this pull request as ready for review July 23, 2025 10:09
@nabenabe0928 nabenabe0928 self-assigned this Jul 24, 2025
self,
trials: list[FrozenTrial],
) -> np.ndarray:
values = np.zeros((len(trials), len(self._optuna_search_space)), dtype=np.float64)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] In Optuna, dtype=float is more conventional, but it is a nit picking.
https://numpy.org/doc/stable/user/basics.types.html

For a huge array, np.empty is better, but the size of this array would not be that huge for GPSampler probably.

Suggested change
values = np.zeros((len(trials), len(self._optuna_search_space)), dtype=np.float64)
values = np.empty((len(trials), len(self._optuna_search_space)), dtype=float)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the suggestion. I was just following the original implementation in order to minimize the difference from the original one. However, I agree with the points you mentioned, and I wanted to update.
Since the same thing also holds for the contractor, I will also update it if we update here.

Copy link
Copy Markdown
Contributor

@nabenabe0928 nabenabe0928 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!
I left only minor comments:)

kAIto47802 and others added 5 commits July 25, 2025 15:29
Co-authored-by: Shuhei Watanabe <47781922+nabenabe0928@users.noreply.github.com>
Co-authored-by: Shuhei Watanabe <47781922+nabenabe0928@users.noreply.github.com>
Co-authored-by: Shuhei Watanabe <47781922+nabenabe0928@users.noreply.github.com>
Co-authored-by: Shuhei Watanabe <47781922+nabenabe0928@users.noreply.github.com>
@nabenabe0928 nabenabe0928 added this to the v4.5.0 milestone Jul 30, 2025
Copy link
Copy Markdown
Collaborator

@sawa3030 sawa3030 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies for the delay in reviewing. I’ve confirmed that _ScaleType is now only used in search_space.py, and the behavior remains consistent. LGTM!

@y0z y0z merged commit c717655 into optuna:master Jul 31, 2025
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

code-fix Change that does not change the behavior, such as code refactoring.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants