Skip to content

[AIR][UX] RunConfig can be specified multiple times and it's unclear which one is used #36082

@justinvyu

Description

@justinvyu

Problem

Here's an example workflow of going from a single training run to a tuning run. Take note of the duplicate configs.

# Single Train run
trainer = TorchTrainer(
    train_fn,
    train_loop_config={"lr": 0.01},
    scaling_config=air.ScalingConfig(num_workers=2),
    run_config=air.RunConfig(storage_path="/storage/path/1", callbacks=[WandbLoggerCallback()]),
)
result = trainer.fit() # <-- This creates a Tuner() under the hood.

# Great, that works! Let's tune the hyperparameters now.

tuner = Tuner(
    trainer,
    param_space={
        # Hmm, what happens to my original train_loop_config?
        # My IDE isn't able to autofill these magic dict keys...
        "train_loop_config": {"lr": tune.uniform(0, 1)},
        "scaling_config": air.ScalingConfig(
            num_workers=tune.choice([2, 4])
        ),
        # ScalingConfig can be here, but RunConfig can not.
    },
    # Why am I able to specify a new RunConfig? Which one gets used?
    run_config=air.RunConfig(storage_path="/storage/path/2"),
    tune_config=tune.TuneConfig(num_samples=2),
)
results = tuner.fit()

# Why don't I see any wandb logs?
# -> It's because the wandb callback that was defined in the Trainer's RunConfig gets thrown away in favor of the Tuner's RunConfig.

We do log an info string, but this log can get lost easily and is still not good UX.

            logger.info(
                "A `RunConfig` was passed to both the `Tuner` and the "
                f"`{trainer.__class__.__name__}`. The run config passed to "
                "the `Tuner` is the one that will be used."
            )

Related Issues and PRs

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Important issue, but not time-criticalUXThe issue is not only about technical bugstrainRay Train Related IssuetuneTune-related issues

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions