Motivation

In certain situations, more sophisticated samplers cannot be applied. Then they fall back to randomly sampling the results. It is highly appreciated that the samplers are programmed in such a robust way. However, in some situations the surprise might appear at the end of a time- and energy-consuming optimization study, when the sampler was actually on par with a random sampler (see, e.g., #6088). In some cases, remodelling the search space can alleviate this issue and users should get the feedback of having a look at it while executing the study - and not at the end by tediously inspecting visuals and statistics on their own.

Description

Samplers should consistently issue warnings when they fall back to the RandomSampler. In this context, the CMA-ES sampler is a great example, see e.g. https://github.com/optuna/optuna/blob/master/optuna/samplers/_cmaes.py#L374. If I had received a similar feedback, e.g., when using BoTorch, that would have saved me quite some time (again, see #6088).

Alternatives (optional)

Let people read the code, run statistical tests at the end of the study and deduce such insights in "sampler performance" on their own. This might be a bit error-prone.

Additional context (optional)

As part of my research, I want to understand the behavior of these algorithms better. So it could be great if the algorithms could inform the user (i.e., me) about its internal steps better, e.g., by keeping records in the database or logging interesting information to debug.