Drop Python 3.6 by HideakiImamura · Pull Request #4150 · optuna/optuna

HideakiImamura · 2022-11-08T07:24:38Z

Motivation

The Python 3.6 is EOL and , recently, the latest official docker image of Ubuntu is 22.04, which does not support Python 3.6 anymore. We, the maintainers, have decided to drop Python 3.6 as we do not feel it is appropriate to continue to officially support it any longer. Future releases of Optuna will only be available for Python 3.7 or later. In addition, CI will not be testing with Python 3.6 at all; users who wish to continue using Python 3.6 will have to use versions of Optuna up to 3.0.3.

Description of the changes

Drop Python 3.6 in setup.py and CI.
Resolve some TODOs after Python 3.6 drop.
Fix some formatting errors, which are ignored due to pytest.skip.

not522 · 2022-11-08T07:48:16Z

Please remove the 3.6 badge from README.

setup.py

not522 · 2022-11-08T08:12:25Z

I found the following codes are related to this PR.

"Optuna supports Python 3.6 or newer." in docs/source/installation.rst

optuna/optuna/samplers/_qmc.py

Lines 160 to 167 in 4bc3dfc

    
           version = sys.version_info 
        
           if version < (3, 7, 0): 
        
               version_txt = str(version[0]) + "." + str(version[1]) + "." + str(version[2]) 
        
               message = ( 
        
                   f"`QMCSampler` is not supported for Python {version_txt}. " 
        
                   "Consider using Python 3.7 or later." 
        
               ) 
        
               raise ValueError(message)

optuna/optuna/trial/_frozen.py

Lines 169 to 183 in 4bc3dfc

    
           # Ordered list of fields required for `__repr__`, `__hash__` and dataframe creation. 
        
           # TODO(hvy): Remove this list in Python 3.7 as the order of `self.__dict__` is preserved. 
        
           _ordered_fields = [ 
        
               "number", 
        
               "_values", 
        
               "datetime_start", 
        
               "datetime_complete", 
        
               "params", 
        
               "_distributions", 
        
               "user_attrs", 
        
               "system_attrs", 
        
               "intermediate_values", 
        
               "_trial_id", 
        
               "state", 
        
           ]

HideakiImamura · 2022-11-09T01:45:38Z

@not522 Thanks for the good catch! Fixed.

not522 · 2022-11-09T01:49:00Z

"Optuna supports Python 3.6 or newer." in docs/source/installation.rst

Please update this statement.

codecov-commenter · 2022-11-09T02:17:33Z

Codecov Report

Merging #4150 (10c3c08) into master (8e3e66d) will increase coverage by 0.03%.
The diff coverage is 90.90%.

@@            Coverage Diff             @@
##           master    #4150      +/-   ##
==========================================
+ Coverage   90.03%   90.06%   +0.03%     
==========================================
  Files         161      161              
  Lines       12690    12680      -10     
==========================================
- Hits        11425    11420       -5     
+ Misses       1265     1260       -5

Impacted Files	Coverage Δ
optuna/_transform.py	`94.35% <ø> (-0.09%)`	⬇️
optuna/samplers/_qmc.py	`97.93% <ø> (+2.79%)`	⬆️
optuna/trial/_frozen.py	`97.89% <0.00%> (-0.02%)`	⬇️
optuna/integration/botorch.py	`98.26% <100.00%> (+0.88%)`	⬆️
optuna/storages/_journal/storage.py	`96.94% <100.00%> (-0.02%)`	⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

not522

LGTM!

toshihikoyanase

Thank you for your PR! The change will simplify CI, setup, and tests 👍
I have a question about FrozenTrial.

toshihikoyanase · 2022-11-09T08:32:29Z

Dockerfile

-        else \
-            pip install ${PIP_OPTIONS} -e '.[checking, document, integration]' -f https://download.pytorch.org/whl/torch_stable.html; \
-        fi \
+        pip install ${PIP_OPTIONS} -e '.[checking, document, integration]' -f https://download.pytorch.org/whl/torch_stable.html; \


Good catch!

[Note] We could remove this line when we dropped Python 3.5 support actually.

toshihikoyanase · 2022-11-09T08:37:37Z

optuna/_transform.py

-        # TODO(hvy): Avoid casting to `OrderedDict` when Python 3.6 is no longer supported since
-        # order will be guaranteed.
-        search_space = OrderedDict(search_space)
-


👍

Changed in version 3.7: Dictionary order is guaranteed to be insertion order. This behavior was an implementation detail of CPython from 3.6.

https://docs.python.org/3.7/library/stdtypes.html?highlight=dict#dict

toshihikoyanase · 2022-11-09T08:48:13Z

optuna/storages/_journal/storage.py

            params = {k: distributions[k].to_external_repr(p) for k, p in log["params"].items()}
        if log["datetime_start"] is not None:
-            datetime_start = datetime_from_isoformat(log["datetime_start"])
+            datetime_start = datetime.datetime.fromisoformat(log["datetime_start"])


Caution This does not support parsing arbitrary ISO 8601 strings - it is only intended as the inverse operation of datetime.isoformat().
https://docs.python.org/3.10/library/datetime.html#datetime.datetime.fromisoformat

I confirmed that log["datetime_start"] and log["datetime_complete"] are formatted by the datetime.isoformat function.

e.g.,

optuna/optuna/storages/_journal/storage.py

Lines 184 to 207 in 10c3c08

def create_new_trial(self, study_id: int, template_trial: Optional[FrozenTrial] = None) -> int:

log: Dict[str, Any] = {

"study_id": study_id,

"datetime_start": datetime.datetime.now().isoformat(timespec="microseconds"),

}

if template_trial:

log["state"] = template_trial.state

if template_trial.values is not None and len(template_trial.values) > 1:

log["value"] = None

log["values"] = template_trial.values

else:

log["value"] = template_trial.value

log["values"] = None

if template_trial.datetime_start:

log["datetime_start"] = template_trial.datetime_start.isoformat(

timespec="microseconds"

)

else:

log["datetime_start"] = None

if template_trial.datetime_complete:

log["datetime_complete"] = template_trial.datetime_complete.isoformat(

timespec="microseconds"

)

toshihikoyanase · 2022-11-09T09:11:01Z

optuna/trial/_frozen.py

-    # Ordered list of fields required for `__repr__`, `__hash__` and dataframe creation.
-    # TODO(hvy): Remove this list in Python 3.7 as the order of `self.__dict__` is preserved.
-    _ordered_fields = [
-        "number",
-        "_values",
-        "datetime_start",
-        "datetime_complete",
-        "params",
-        "_distributions",
-        "user_attrs",
-        "system_attrs",
-        "intermediate_values",
-        "_trial_id",
-        "state",
-    ]


Question: The keys of self.__dict__ will be as follows:

['_number', 'state', '_values', '_datetime_start', 'datetime_complete', '_params', '_user_attrs', '_system_attrs', 'intermediate_values', '_distributions', '_trial_id']

We can see some differences:

Some field including number, params, user_attrs and system_attrs will be prefixed with _.

The order of fields is different. For example, _values is the second item of _ordered_field, but it is the third one of self.__dict__.

But, I don't think they affect users because of the following two reasons:

hash value will be changed, but hash values are different process by process originally. So, we cannot compare hash values of different processes even if we use the current master.

I confirmed that trial_str = repr(trial) can be restored by eval(trial_str).

I also confirmed that dumped trial with repr in master can be restored by eval in this PR.

What do you think of it?

Thanks for the insightful review. Actually, the items and order of self.__dict__ and _ordered_field are different each other. But, as you confirmed,

I confirmed that trial_str = repr(trial) can be restored by eval(trial_str).
I also confirmed that dumped trial with repr in master can be restored by eval in this PR.

The impact of this change on users would be minimal. So, we can accept it.

Thank you for your reply. That makes sense.

toshihikoyanase

LGTM! Thank you.

When we release v3.1.0, we need to update the installation section of optuna.org.

Drop python 3.6

391a82e

HideakiImamura added the compatibility Change that breaks compatibility. label Nov 8, 2022

Use ubuntu-latest

0d6f801

github-actions bot added optuna.samplers Related to the `optuna.samplers` submodule. This is automatically labeled by github-actions. optuna.storages Related to the `optuna.storages` submodule. This is automatically labeled by github-actions. labels Nov 8, 2022

not522 self-assigned this Nov 8, 2022

not522 mentioned this pull request Nov 8, 2022

[RFC] Drop Python 3.6 support #3021

Closed

not522 linked an issue Nov 8, 2022 that may be closed by this pull request

[RFC] Drop Python 3.6 support #3021

Closed

not522 reviewed Nov 8, 2022

View reviewed changes

setup.py Outdated Show resolved Hide resolved

HideakiImamura added 2 commits November 8, 2022 16:55

Follow review comments

d1ab2ab

Apply formatter

474672f

Remove remaining TODOs

0d1b129

c-bata mentioned this pull request Nov 9, 2022

Drop Python 3.6 support. CyberAgentAILab/cmaes#130

Merged

Fix by reviews

10c3c08

not522 approved these changes Nov 9, 2022

View reviewed changes

not522 removed their assignment Nov 9, 2022

toshihikoyanase self-assigned this Nov 9, 2022

toshihikoyanase reviewed Nov 9, 2022

View reviewed changes

c-bata mentioned this pull request Nov 9, 2022

Add Dask integration #2023

Merged

2 tasks

toshihikoyanase approved these changes Nov 10, 2022

View reviewed changes

toshihikoyanase added this to the v3.1.0 milestone Nov 10, 2022

toshihikoyanase merged commit 9942783 into optuna:master Nov 10, 2022

c-bata mentioned this pull request Nov 10, 2022

Deselect integration tests in Tests CI #4013

Merged

HideakiImamura mentioned this pull request Jan 4, 2023

Run test_pytorch_distributed.py again #4301

Merged

not522 mentioned this pull request Jan 19, 2023

optuna v3.1.0 conda-forge/optuna-feedstock#50

Merged

3 tasks

HideakiImamura deleted the drop-python-3.6 branch June 9, 2023 02:25

	def create_new_trial(self, study_id: int, template_trial: Optional[FrozenTrial] = None) -> int:
	log: Dict[str, Any] = {
	"study_id": study_id,
	"datetime_start": datetime.datetime.now().isoformat(timespec="microseconds"),
	}

	if template_trial:
	log["state"] = template_trial.state
	if template_trial.values is not None and len(template_trial.values) > 1:
	log["value"] = None
	log["values"] = template_trial.values
	else:
	log["value"] = template_trial.value
	log["values"] = None
	if template_trial.datetime_start:
	log["datetime_start"] = template_trial.datetime_start.isoformat(
	timespec="microseconds"
	)
	else:
	log["datetime_start"] = None
	if template_trial.datetime_complete:
	log["datetime_complete"] = template_trial.datetime_complete.isoformat(
	timespec="microseconds"
	)

Uh oh!

Conversation

HideakiImamura commented Nov 8, 2022

Motivation

Description of the changes

Uh oh!

not522 commented Nov 8, 2022

Uh oh!

Uh oh!

not522 commented Nov 8, 2022

Uh oh!

HideakiImamura commented Nov 9, 2022

Uh oh!

not522 commented Nov 9, 2022

Uh oh!

codecov-commenter commented Nov 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

not522 left a comment

Choose a reason for hiding this comment

Uh oh!

toshihikoyanase left a comment

Choose a reason for hiding this comment

Uh oh!

toshihikoyanase Nov 9, 2022

Choose a reason for hiding this comment

Uh oh!

toshihikoyanase Nov 9, 2022

Choose a reason for hiding this comment

Uh oh!

toshihikoyanase Nov 9, 2022

Choose a reason for hiding this comment

Uh oh!

toshihikoyanase Nov 9, 2022

Choose a reason for hiding this comment

Uh oh!

HideakiImamura Nov 9, 2022

Choose a reason for hiding this comment

Uh oh!

toshihikoyanase Nov 10, 2022

Choose a reason for hiding this comment

Uh oh!

toshihikoyanase left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov-commenter commented Nov 9, 2022 •

edited

Loading