This repository was archived by the owner on Nov 3, 2023. It is now read-only.
This repository was archived by the owner on Nov 3, 2023. It is now read-only.
Tune.run() doesn't finish with trainer.test() call. #132
Description
Hi,
I'm using the ray-lightning 0.2.0 version with PL 1.5.9, to use hyperparameters optimization in parallel.
The following script doesn't finish, and trainings are still running even when my test metrics are printed (so the test function is terminated).
If I remove 'trainer.test()' then the script finishes normally.
I have tested this code with ray-lightning 0.1.1 version with PL 1.4.9. and it is OK.
def train(config):
metrics = {"loss": "val_loss"}
callbacks = [TuneReportCallback(metrics, on="validation_end")]
model = md.MyModel(...)
datamodule = ...
trainer = pl.Trainer(
max_epochs=max_epochs,
logger=logger,
callbacks=callbacks,
plugins=[RayPlugin(num_workers=2)])
trainer.fit(model, datamodule)
trainer.test(model, datamodule) # OK without this line
if __name__ == '__main__':
analysis = tune.run(
train,
metric="loss",
mode="min",
config=training_config,
num_samples=1,
resources_per_trial=get_tune_resources(num_workers=2),
name="tune_mnist")
print("\n\nBest hyperparameters found were: ", analysis.best_config)Thanks
python 3.9
pytorch 1.11