…ay-project#57882)

# Summary

The crux of the issue is that in the past, train run status was
synonymous with final worker group status, but now, when there are
pending validations, the worker group is finished but the train run is
not. This leads to confusing situations in which the Train Run is
`FINISHED`, but because there are pending validations, the `controller`
actor is alive and results are inaccessible.

This PR:
* Adds a new `SHUTTING_DOWN` `TrainControllerState` that happens after
the worker group finishes but before the controller shuts everything
down.
* Makes `ValidationManager` logging slightly cleaner.

Like `RESCHEDULING`, `SHUTTING_DOWN` is a hidden state that shows up in
`StateManager` logs and Grafana but not in the state export. We only
want to show terminal states in the state export after `fit()` has
returned and results are accessible. More concretely:
* Finished/errored: The worker group finishes (Train Run is `RUNNING`
but internal state is `SHUTTING_DOWN`), validation finishes (both Train
Run and internal state say `FINISHED` or `ERRORED`), then results are
accessible.
* Aborted: Ideally, the worker group should be aborted and in-flight
validation tasks canceled before the Train Run is `ABORTED`. However,
this PR doesn't change the current behavior, in which the Train Run
might be `ABORTED` before reference counting cleans up the validation
tasks. I will cancel validation tasks before marking the train run
`ABORTED` in a future PR.

I considered polling both the worker group and validations in `_step`
itself, but decided to leave `_step` as a function that only cares about
the worker group.

# Testing

Unit tests

---------

Signed-off-by: Timothy Seah <tseah@anyscale.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: xgui <xgui@anyscale.com>

…ay-project#57882)

# Summary

The crux of the issue is that in the past, train run status was
synonymous with final worker group status, but now, when there are
pending validations, the worker group is finished but the train run is
not. This leads to confusing situations in which the Train Run is
`FINISHED`, but because there are pending validations, the `controller`
actor is alive and results are inaccessible.

This PR:
* Adds a new `SHUTTING_DOWN` `TrainControllerState` that happens after
the worker group finishes but before the controller shuts everything
down.
* Makes `ValidationManager` logging slightly cleaner.
 
Like `RESCHEDULING`, `SHUTTING_DOWN` is a hidden state that shows up in
`StateManager` logs and Grafana but not in the state export. We only
want to show terminal states in the state export after `fit()` has
returned and results are accessible. More concretely:
* Finished/errored: The worker group finishes (Train Run is `RUNNING`
but internal state is `SHUTTING_DOWN`), validation finishes (both Train
Run and internal state say `FINISHED` or `ERRORED`), then results are
accessible.
* Aborted: Ideally, the worker group should be aborted and in-flight
validation tasks canceled before the Train Run is `ABORTED`. However,
this PR doesn't change the current behavior, in which the Train Run
might be `ABORTED` before reference counting cleans up the validation
tasks. I will cancel validation tasks before marking the train run
`ABORTED` in a future PR.

I considered polling both the worker group and validations in `_step`
itself, but decided to leave `_step` as a function that only cares about
the worker group.

# Testing

Unit tests

---------

Signed-off-by: Timothy Seah <tseah@anyscale.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

…ay-project#57882)

# Summary

The crux of the issue is that in the past, train run status was
synonymous with final worker group status, but now, when there are
pending validations, the worker group is finished but the train run is
not. This leads to confusing situations in which the Train Run is
`FINISHED`, but because there are pending validations, the `controller`
actor is alive and results are inaccessible.

This PR:
* Adds a new `SHUTTING_DOWN` `TrainControllerState` that happens after
the worker group finishes but before the controller shuts everything
down.
* Makes `ValidationManager` logging slightly cleaner.

Like `RESCHEDULING`, `SHUTTING_DOWN` is a hidden state that shows up in
`StateManager` logs and Grafana but not in the state export. We only
want to show terminal states in the state export after `fit()` has
returned and results are accessible. More concretely:
* Finished/errored: The worker group finishes (Train Run is `RUNNING`
but internal state is `SHUTTING_DOWN`), validation finishes (both Train
Run and internal state say `FINISHED` or `ERRORED`), then results are
accessible.
* Aborted: Ideally, the worker group should be aborted and in-flight
validation tasks canceled before the Train Run is `ABORTED`. However,
this PR doesn't change the current behavior, in which the Train Run
might be `ABORTED` before reference counting cleans up the validation
tasks. I will cancel validation tasks before marking the train run
`ABORTED` in a future PR.

I considered polling both the worker group and validations in `_step`
itself, but decided to leave `_step` as a function that only cares about
the worker group.

# Testing

Unit tests

---------

Signed-off-by: Timothy Seah <tseah@anyscale.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Aydin Abiar <aydin@anyscale.com>

…ay-project#57882)

# Summary

The crux of the issue is that in the past, train run status was
synonymous with final worker group status, but now, when there are
pending validations, the worker group is finished but the train run is
not. This leads to confusing situations in which the Train Run is
`FINISHED`, but because there are pending validations, the `controller`
actor is alive and results are inaccessible.

This PR:
* Adds a new `SHUTTING_DOWN` `TrainControllerState` that happens after
the worker group finishes but before the controller shuts everything
down.
* Makes `ValidationManager` logging slightly cleaner.

Like `RESCHEDULING`, `SHUTTING_DOWN` is a hidden state that shows up in
`StateManager` logs and Grafana but not in the state export. We only
want to show terminal states in the state export after `fit()` has
returned and results are accessible. More concretely:
* Finished/errored: The worker group finishes (Train Run is `RUNNING`
but internal state is `SHUTTING_DOWN`), validation finishes (both Train
Run and internal state say `FINISHED` or `ERRORED`), then results are
accessible.
* Aborted: Ideally, the worker group should be aborted and in-flight
validation tasks canceled before the Train Run is `ABORTED`. However,
this PR doesn't change the current behavior, in which the Train Run
might be `ABORTED` before reference counting cleans up the validation
tasks. I will cancel validation tasks before marking the train run
`ABORTED` in a future PR.

I considered polling both the worker group and validations in `_step`
itself, but decided to leave `_step` as a function that only cares about
the worker group.

# Testing

Unit tests

---------

Signed-off-by: Timothy Seah <tseah@anyscale.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Future-Outlier <eric901201@gmail.com>

…ay-project#57882)

# Summary

The crux of the issue is that in the past, train run status was
synonymous with final worker group status, but now, when there are
pending validations, the worker group is finished but the train run is
not. This leads to confusing situations in which the Train Run is
`FINISHED`, but because there are pending validations, the `controller`
actor is alive and results are inaccessible.

This PR:
* Adds a new `SHUTTING_DOWN` `TrainControllerState` that happens after
the worker group finishes but before the controller shuts everything
down.
* Makes `ValidationManager` logging slightly cleaner.

Like `RESCHEDULING`, `SHUTTING_DOWN` is a hidden state that shows up in
`StateManager` logs and Grafana but not in the state export. We only
want to show terminal states in the state export after `fit()` has
returned and results are accessible. More concretely:
* Finished/errored: The worker group finishes (Train Run is `RUNNING`
but internal state is `SHUTTING_DOWN`), validation finishes (both Train
Run and internal state say `FINISHED` or `ERRORED`), then results are
accessible.
* Aborted: Ideally, the worker group should be aborted and in-flight
validation tasks canceled before the Train Run is `ABORTED`. However,
this PR doesn't change the current behavior, in which the Train Run
might be `ABORTED` before reference counting cleans up the validation
tasks. I will cancel validation tasks before marking the train run
`ABORTED` in a future PR.

I considered polling both the worker group and validations in `_step`
itself, but decided to leave `_step` as a function that only cares about
the worker group.

# Testing

Unit tests

---------

Signed-off-by: Timothy Seah <tseah@anyscale.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: peterxcli <peterxcli@gmail.com>