This repository was archived by the owner on Nov 3, 2023. It is now read-only.
This repository was archived by the owner on Nov 3, 2023. It is now read-only.
[raystrategy] multi-stragy in the worker is not consistent #161
Description
In the current ray strategy, since the strategy shows in three places: two are obvious and one is hidden.
one is in the ray launcher:
class RayLauncher(_SpawnLauncher):
def __init__(self, strategy: "RayPlugin") -> None:
self._strategy = strategy
self._start_method = "ray"
self._workers = []
self._futures = []
self._master_addr = NoneAnd later we use these in _wrapped_function_
https://github.com/JiahaoYao/ray_lightning/blob/main/ray_lightning/ray_ddp.py#L241-L242
self._strategy.set_remote(True)
self._strategy.set_global_to_local(global_to_local)The second is an attributed in the trainer.strategy.
The last hidden one is in the
self._futures = [
w.execute.remote(self._wrapping_function, i, self._global_to_local,
trainer, function, args, kwargs, self.tune_queue)
for i, w in enumerate(self._workers)
]ray remote functions create the copy of trainer.
Thus, the actual call of the strategy.teardown is the one from the copies of trainer.
support of the assumption is
printing out the pid of strategy, and it turns out they are different.
Proposal: might removing the redundant use of strategy