Skip to content

Isolation of optuna integration modules. #4484

@gen740

Description

@gen740

Isolation of optuna integration modules.

Optuna is now isolating optuna.integration module to optuna-integration.
The following is a list of integration modules and their priorities.

Status Module Priority Note
Done AllenNLP high #4579
Done BoTorch low #5230
Done Catalyst high #4644
Done CatBoost high #5198
Done Chainer high #4497
#4370
Done Dask low #5222
Done fast.ai high #5120
Done Keras high #4558
Done LightGBM low #5249
Done MLflow high #5246
Done Weights & Biases high #5237
Done MXNet high #4790
Done pycma high #5236
Done PyTorch(ignite) high #5213
Done PyTorch(lightning) high #5213
Done PyTorch(ddp) high #5213
Done scikit-learn low #5225
Done scikit-optimize high #5234
Done SHAP low #4791
Done skorch high #4663
Done TensorBoard high #5161
Done TensorFlow high #4666
Done tf.keras high #4662
Done XGBoost high #5239

Do not migrate low-priority module because it will have some complex dependencies on tests and others. (Migration of low-priority modules would be done by Committers.)

Migration procedure

Following is the transfer procedure, please follow these instructions

On the Optuna Integration side:

  1. Clone the optuna-integration forked repository
    git clone git@github.com:{YOUR-FOR}/optuna-integration.git
  2. Add the Optuna master branch to the local upstream.
    git remote add upstream https://github.com/optuna/optuna.git
  3. Create the new working branch
    git checkout -b YOUR-BRANCH-NAME.
  4. Check out the relevant file for integration purposes.
    git fetch upstream and git checkout upstream/master -- <path/to/file>
    • The desired files are as follows
      • optuna/integration/<module>
      • tests/integration_tests/<module>
    • Move the file paths from optuna/integration/<module> to optuna_integration/<module> and tests/integration_tests/<module> to tests/<module>.
    • Create a commit for each file, with a commit message that reads

      "Fetched from https://github.com/optuna/optuna/commit/<commit-hash>".

    • <commit-hash> can be obtained using git rev-list upstream/master -- <path/to/file> | head -n 1.
  5. In the desired files, replace optuna._imports with optuna_integration._imports.
  6. Update the required dependencies in pyproject.toml. Especially, please add the corresponding third party library in the all section of the pyproject.toml.
  7. Modify the CI workflow file if necessary. Check the Optuna's CI workflow file of integration, i.e., .github/workflows/tests-integration.yml. If you find the any specific treatment for the modified integration, please add it to the optuna-integration's CI workflow file, i.e., .github/workflows/tests.yml.
  8. Update the imports in tests to use optuna_integration instead of optuna.integration.
  9. Add the new module to docs/source/reference/index.rst (from here).
  10. Add the new module to the relevant section in README.md.
  11. Change all optuna.integration sentences to optuna_integration (if they exist).

On the Optuna side:

  1. Import the package that was migrated from optuna_integration and pass its contents to __all__ magic variable.
    • For example.
from optuna_integration.chainer import ChainerPruningExtension


__all__ = ["ChainerPruningExtension"]
  1. Remove the relevant files from test/integration and related imports from test/integration/test_integration.py.
  2. Remove any relevant tests from pytest tests -m "integration" --ignore in .github/workflows/tests-integration.yml.
  3. Remove the relevant documentation from docs/source/reference/integration.rst.
  4. Remove the integration dependencies from setup.py and pyproject.toml, taking care not to remove dependencies that are still needed.
    • Do not remove the relevant imports from _import_structure variable in optuna/integration/__init__.py, as this would render the from optuna.integration import ... idiom unusable.
  5. Transfer any relevant issues from Optuna to Optuna Integration.
  6. Request that the author of any relevant PRs transfer them from Optuna to Optuna Integration.

These changes should be merged into optuna-integration at first, then merged into optuna.

On the Optuna exmaple side:

  1. Change the corresponding integration's ci yaml to install optuna-integration.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions