Benchmark of Multi-objective Unconstrained Optimization

Benchmark setup

(This benchmark follows the evaluation procedure used in #6430)
I compared the current master implementation against the Kriging Believer strategy introduced in this PR.

Tested settings

• Benchmark problems: 5×WFG (function1 to 5)
• n_trials: [25, 50, 75, 100]
• batch_size: [5, 10, 50]
• n_objective: [2, 4]
• Repeats: 10 independent runs per (problem, n_trials, batch_size, strategy), using different random seeds

Evaluation

For each (benchmark problems, n_trials, batch_size), I compared hypervolume between master branch and this PR using a one-sided Mann–Whitney U test (α = 0.10).

If strategy i is significantly better than strategy j (j != i), I count it as a “win” for i. Total wins are then aggregated over all (n_trials, problems).

NOTE: The benchmark script is available here

Results (total wins)

Overall, these results suggest that this PR improves HV compared to master under the tested settings.

Let me open this PR when #6430 is merged

Benchmark of Single-objective Constrained Optimization

Benchmark setup

(This benchmark follows the evaluation procedure used in #6430)
I compared the current master implementation against the Kriging Believer strategy introduced in this PR.

Tested settings

• Benchmark problems: two simulation problems from "Bayesian Optimization with Inequality Constraints" (Gardner, Jacob R., et al., ICML 2014):
  • Objective: cos(2x) cos(y) + sin(x)
    Constraint: cos(x) cos(y) - sin(x) sin(y) <= 0.5
  • Objective: sin(x) + y
    Constraint: sin(x) sin(y) <= -0.95
• n_trials: [25, 50, 75, 100]
• batch_size: [5, 10, 50]
• Repeats: 100 independent runs per (problem, n_trials, batch_size, strategy), using different random seeds

Evaluation

For each (benchmark problems, n_trials, batch_size), I compared best_value between master branch and this PR using a one-sided Mann–Whitney U test (α = 0.10).

If strategy i is significantly better than strategy j (j != i), I count it as a “win” for i. Total wins are then aggregated over all (n_trials, problems).

NOTE: The benchmark script is available here

Results (total wins)

Overall, these results suggest that this PR improves the performance compared to master under the tested settings.

Benchmark of Multi-objective Constrained Optimization

Benchmark setup

(This benchmark follows the evaluation procedure used in #6430)
I compared the current master implementation against the Kriging Believer strategy introduced in this PR.

Tested settings

• Benchmark problems: C2-DTLZ2
• n_trials: [25, 50, 75, 100]
• batch_size: [5, 10, 50]
• Repeats: 10 independent runs per (problem, n_trials, batch_size, strategy), using different random seeds

Evaluation

For each (benchmark problems, n_trials, batch_size), I compared hypervolume between master branch and this PR using a one-sided Mann–Whitney U test (α = 0.10).

If strategy i is significantly better than strategy j (j != i), I count it as a “win” for i. Total wins are then aggregated over all (n_trials, problems).

NOTE: The benchmark script is available here

Results (total wins)

Overall, these results suggest that this PR improves HV compared to master under the tested settings.

@y0z Could you review this PR?

Could you check for mypy errors? https://github.com/not522/optuna/actions/runs/22881585369/job/66385359005
The rest of the PR appears to be a well-organized implementation. I'll check benchmarks.

Could you check for mypy errors? https://github.com/not522/optuna/actions/runs/22881585369/job/66385359005 The rest of the PR appears to be a well-organized implementation. I'll check benchmarks.

Thank you for checking. The mypy errors should be fixed now.
https://github.com/sawa3030/optuna/actions/runs/23037280833

Thank you for the review. I've made the requested updates. PTAL.