-
Notifications
You must be signed in to change notification settings - Fork 7.4k
[AIR][UX] RunConfig can be specified multiple times and it's unclear which one is used #36082
Copy link
Copy link
Open
Labels
P2Important issue, but not time-criticalImportant issue, but not time-criticalUXThe issue is not only about technical bugsThe issue is not only about technical bugstrainRay Train Related IssueRay Train Related IssuetuneTune-related issuesTune-related issues
Description
Problem
Here's an example workflow of going from a single training run to a tuning run. Take note of the duplicate configs.
# Single Train run
trainer = TorchTrainer(
train_fn,
train_loop_config={"lr": 0.01},
scaling_config=air.ScalingConfig(num_workers=2),
run_config=air.RunConfig(storage_path="/storage/path/1", callbacks=[WandbLoggerCallback()]),
)
result = trainer.fit() # <-- This creates a Tuner() under the hood.
# Great, that works! Let's tune the hyperparameters now.
tuner = Tuner(
trainer,
param_space={
# Hmm, what happens to my original train_loop_config?
# My IDE isn't able to autofill these magic dict keys...
"train_loop_config": {"lr": tune.uniform(0, 1)},
"scaling_config": air.ScalingConfig(
num_workers=tune.choice([2, 4])
),
# ScalingConfig can be here, but RunConfig can not.
},
# Why am I able to specify a new RunConfig? Which one gets used?
run_config=air.RunConfig(storage_path="/storage/path/2"),
tune_config=tune.TuneConfig(num_samples=2),
)
results = tuner.fit()
# Why don't I see any wandb logs?
# -> It's because the wandb callback that was defined in the Trainer's RunConfig gets thrown away in favor of the Tuner's RunConfig.We do log an info string, but this log can get lost easily and is still not good UX.
logger.info(
"A `RunConfig` was passed to both the `Tuner` and the "
f"`{trainer.__class__.__name__}`. The run config passed to "
"the `Tuner` is the one that will be used."
)Related Issues and PRs
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
P2Important issue, but not time-criticalImportant issue, but not time-criticalUXThe issue is not only about technical bugsThe issue is not only about technical bugstrainRay Train Related IssueRay Train Related IssuetuneTune-related issuesTune-related issues