Description

Add an annotation to failed pods "already_checked=True" for a task rerun to create a new pod.

Use case / motivation
In the case of pod failures, we WANT to reattach the first time so the failure can be reported.
Instead, we should add an annotation to that failed pod of "already_checked=True". This way the first time the task re-runs it would re-attach and fail, but the second time it would see the annotation and create a new pod instead. #10020 (comment)