Skip to content

daemon: cleanupContainer: don't fail if container is already stopped#46213

Merged
thaJeztah merged 3 commits intomoby:masterfrom
thaJeztah:daemon_remove_errors
Aug 24, 2023
Merged

daemon: cleanupContainer: don't fail if container is already stopped#46213
thaJeztah merged 3 commits intomoby:masterfrom
thaJeztah:daemon_remove_errors

Conversation

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Aug 13, 2023

daemon: cleanupContainer: don't fail if container is already stopped

Saw this failure in a flaky test, and I wondered why we consider this an
error condition;

=== RUN   TestKillWithStopSignalAndRestartPolicies
    main_test.go:32: assertion failed: error is not nil: Error response from daemon: Could not kill running container 668f62511f4aa62357269cd405cff1fbe295b7f6d5011e7cfed434e3072330b7, cannot remove - Container 668f62511f4aa62357269cd405cff1fbe295b7f6d5011e7cfed434e3072330b7 is not running: failed to remove 668f62511f4aa62357269cd405cff1fbe295b7f6d5011e7cfed434e3072330b7
--- FAIL: TestKillWithStopSignalAndRestartPolicies (0.84s)
=== RUN   TestKillWithStopSignalAndRestartPolicies/same-signal-disables-restart-policy
    --- PASS: TestKillWithStopSignalAndRestartPolicies/same-signal-disables-restart-policy (0.42s)
=== RUN   TestKillWithStopSignalAndRestartPolicies/different-signal-keep-restart-policy
    --- PASS: TestKillWithStopSignalAndRestartPolicies/different-signal-keep-restart-policy (0.23s)

In the above;

  1. Error response from daemon: Could not kill running container 668f62511f4aa62357269cd405cff1fbe295b7f6d5011e7cfed434e3072330b7
  2. cannot remove - Container 668f62511f4aa62357269cd405cff1fbe295b7f6d5011e7cfed434e3072330b7 is not running
  3. failed to remove 668f62511f4aa62357269cd405cff1fbe295b7f6d5011e7cfed434e3072330b7

So it looks like the removal fails because we couldn't kill the container
because it was already stopped, which may be a race condition where the first
check shows the container to be running (but may already be in process to be
removed or killed. In either case, we probably shouldn't fail the removal if
the container is already stopped.

This patch adds a isNotRunning() utility, so that we can ignore this case,
and proceed with the removal.

daemon: cleanupContainer: slightly cleanup error messages

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

@thaJeztah thaJeztah force-pushed the daemon_remove_errors branch 2 times, most recently from 0c30262 to d72a568 Compare August 13, 2023 21:17
@thaJeztah thaJeztah marked this pull request as draft August 13, 2023 22:31
@thaJeztah thaJeztah force-pushed the daemon_remove_errors branch 2 times, most recently from 17ceaee to 01eb5a6 Compare August 16, 2023 00:04
@thaJeztah thaJeztah marked this pull request as ready for review August 16, 2023 00:04
@thaJeztah thaJeztah requested a review from tianon as a code owner August 16, 2023 00:04
Saw this failure in a flaky test, and I wondered why we consider this an
error condition;

    === RUN   TestKillWithStopSignalAndRestartPolicies
        main_test.go:32: assertion failed: error is not nil: Error response from daemon: Could not kill running container 668f62511f4aa62357269cd405cff1fbe295b7f6d5011e7cfed434e3072330b7, cannot remove - Container 668f62511f4aa62357269cd405cff1fbe295b7f6d5011e7cfed434e3072330b7 is not running: failed to remove 668f62511f4aa62357269cd405cff1fbe295b7f6d5011e7cfed434e3072330b7
    --- FAIL: TestKillWithStopSignalAndRestartPolicies (0.84s)
    === RUN   TestKillWithStopSignalAndRestartPolicies/same-signal-disables-restart-policy
        --- PASS: TestKillWithStopSignalAndRestartPolicies/same-signal-disables-restart-policy (0.42s)
    === RUN   TestKillWithStopSignalAndRestartPolicies/different-signal-keep-restart-policy
        --- PASS: TestKillWithStopSignalAndRestartPolicies/different-signal-keep-restart-policy (0.23s)

In the above;

1. `Error response from daemon: Could not kill running container 668f62511f4aa62357269cd405cff1fbe295b7f6d5011e7cfed434e3072330b7`
2. `cannot remove - Container 668f62511f4aa62357269cd405cff1fbe295b7f6d5011e7cfed434e3072330b7 is not running`
3. `failed to remove 668f62511f4aa62357269cd405cff1fbe295b7f6d5011e7cfed434e3072330b7`

So it looks like the removal fails because we couldn't kill the container
because it was already stopped, which may be a race condition where the first
check shows the container to be running (but may already be in process to be
removed or killed. In either case, we probably shouldn't fail the removal if
the container is already stopped.

This patch adds a `isNotRunning()` utility, so that we can ignore this case,
and proceed with the removal.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Also remove integration-cli: `DockerAPISuite.TestContainerAPIDeleteConflict`,
which was testing the same conditions as `TestRemoveContainerRunning` in
integration/container.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah
Copy link
Member Author

@neersighted @rumpl ptal 🤗

Copy link
Member

@rumpl rumpl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, do we want to change the Cannot kill container error message too while we're at it?

-Error response from daemon: Cannot kill container: 79a: Container 79a19db9dd2ce7be5d9ca5651fbdd5e2c1b4dbe3a0ec9cff18af9e820ac6f88e is not running
+Error response from daemon: cannot kill container: 79a: container 79a19db9dd2ce7be5d9ca5651fbdd5e2c1b4dbe3a0ec9cff18af9e820ac6f88e is not running

@thaJeztah
Copy link
Member Author

Good one; I see there's a Cannot pause error as well.

Let me do those in a follow-up, because it's possible that docker-py would also need updates for that (need to check)

@thaJeztah thaJeztah merged commit 64f5d9b into moby:master Aug 24, 2023
@thaJeztah thaJeztah deleted the daemon_remove_errors branch August 24, 2023 11:34
@thaJeztah thaJeztah added the kind/bugfix PR's that fix bugs label Aug 24, 2023
@thaJeztah thaJeztah added this to the 25.0.0 milestone Aug 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants