Skip to content

[BUG] Docker daemon crashing or restarting causes integration tests to hang and timeout #972

@invidian

Description

@invidian

Expected Behavior

Tests should exit gracefully right after build under test fails.

Actual Behavior

Last occurrence: #970. Dockerd was segfaulting which was causing test suite to hang and panic on timeout.

This also might be related to #895.

Steps To Reproduce

Patch to reproduce:

diff --git test/linux_target_test.go test/linux_target_test.go
index 83f856f..e9dd1c2 100644
--- test/linux_target_test.go
+++ test/linux_target_test.go
@@ -11,6 +11,7 @@ import (
        "io"
        "io/fs"
        "os"
+       "os/exec"
        "path/filepath"
        "strings"
        "testing"
@@ -918,6 +919,19 @@ index 0000000..5260cb1
                                        withIgnoreCache(targets.IgnoreCacheKeyContainer),
                                )

+                               go func() {
+                                       select {
+                                       case <-ctx.Done():
+                                               return
+                                       case <-time.After(5 * time.Second):
+                                               t.Logf("Restarting docker daemon")
+
+                                               if err := exec.Command("sudo", "systemctl", "restart", "docker").Run(); err != nil {
+                                                       t.Logf("Failed to restart docker daemon: %v", err)
+                                               }
+                                       }
+                               }()
+
                                res := solveT(ctx, t, gwc, sr)

                                ops, err := test.LLBOpsFromState(ctx, resultToState(t, res))

Then run e.g. go test -timeout 1h ./test -v -count 1 -run TestBionic/container/respects_container_cache_key.

Are you willing to submit PRs to contribute to this bug fix?

  • Yes, I am willing to implement it.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions