Skip to content

acceptance: TestDockerCLI_test_demo_multitenant shutdown synchronization #110645

@rafiss

Description

@rafiss

Failed here: https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Ci_Tests_Acceptance/11765319?buildTab=artifacts#

The PR (#110595) is changing the text of an error message in SQL, so it is not related.

I230913 19:30:19.875967 16349 1@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6401  the server is terminating due to a fatal error (see the STORAGE channel for details)
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402  000410.sst:
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 +orig err: open ‹000410.sst›: file does not exist
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 +list err: open ‹./›: file does not exist
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !goroutine 16349 [running]:
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !runtime/debug.Stack()
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !	GOROOT/src/runtime/debug/stack.go:24 +0x65
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(_, {{{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, ...}}, ...})
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !	github.com/cockroachdb/cockroach/pkg/util/log/clog.go:274 +0xb8
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepthInternal({0x793d508, 0xc00ad84ea0}, 0x3, 0x4, 0x3, 0x0?, {0x6364e88, 0x1d}, {0xc00a95c270, 0x3, ...})
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:109 +0x5a5
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !github.com/cockroachdb/cockroach/pkg/util/log.logfDepth(...)
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !	github.com/cockroachdb/cockroach/pkg/util/log/channels.go:39
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !github.com/cockroachdb/cockroach/pkg/util/log.loggerStorage.FatalfDepth(...)
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !	github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/util/log/log_channels_generated.go:2223
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !github.com/cockroachdb/cockroach/pkg/storage.pebbleLogger.Fatalf({{0x793d508?, 0xc00ad84ea0?}, 0x0?}, {0x6364e88?, 0x46a9a7?}, {0xc00a95c270?, 0x57a5ee0?, 0xc006482801?})
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !	github.com/cockroachdb/cockroach/pkg/storage/pebble.go:712 +0x56
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !github.com/cockroachdb/pebble/internal/base.MustExist({0x79b6600, 0xc008385920}, {0xc00e788060, 0xa}, {0x7f93acebfb50, 0xc00a50a918}, {0x78fde60, 0xc00a95c1b0})
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !	github.com/cockroachdb/pebble/internal/base/external/com_github_cockroachdb_pebble/internal/base/filenames.go:172 +0x283
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).vfsOpenForReading(0xc010bc9800, {0x2?, 0x0?}, 0x0?, {0x0?}, {0x0?})
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !	github.com/cockroachdb/pebble/objstorage/objstorageprovider/external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/vfs.go:30 +0x145
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !github.com/cockroachdb/pebble/objstorage/objstorageprovider.(*provider).OpenForReading(0xc010bc9800, {0x793d498, 0xc0000bc070}, 0x5f0fcc0?, {0x93ab949108?}, {0x80?})
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !	github.com/cockroachdb/pebble/objstorage/objstorageprovider/external/com_github_cockroachdb_pebble/objstorage/objstorageprovider/provider.go:225 +0x35a
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !github.com/cockroachdb/pebble.(*tableCacheValue).load(0xc00a95c0c0, {{0xc006482cc8?}, 0xee3b6f?, 0xc00a95c120?}, 0xc005806790, 0xc0047695f8)
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:1056 +0xa3
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !github.com/cockroachdb/pebble.(*tableCacheShard).findNode.func3({0x793d508, 0xc00a95c120})
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:816 +0x3c
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !runtime/pprof.Do({0x793d498?, 0xc0000bc058?}, {{0xc000620780?, 0xc006482b98?, 0xc006482dd8?}}, 0xc006482db0)
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !	GOROOT/src/runtime/pprof/runtime.go:44 +0xa3
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !github.com/cockroachdb/pebble.(*tableCacheShard).findNode(0xc005806790, 0xc010bc8f00, 0xc0047695f8)
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:815 +0x4c5
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !github.com/cockroachdb/pebble.(*tableCacheContainer).withReader(0x100000000003c?, {0xc00d4f5f30?}, 0xc006482ee0)
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_cache.go:199 +0x6d
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !github.com/cockroachdb/pebble.(*DB).loadTableStats(0xc006483080?, 0x3e48?, 0xc00e0f7380?, {0xf?})
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_stats.go:283 +0xe5
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !github.com/cockroachdb/pebble.(*DB).scanReadStateTableStats(0xc00fc2c000, 0xc0066328d0, {0xc00afdd4c0?, 0x0?, 0xc00afdd370?})
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_stats.go:255 +0x3b5
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !github.com/cockroachdb/pebble.(*DB).collectTableStats(0xc00fc2c000)
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_stats.go:110 +0x245
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !created by github.com/cockroachdb/pebble.(*DB).maybeCollectTableStatsLocked
F230913 19:30:19.875998 16349 3@pebble/internal/base/filenames.go:172 ⋮ [n?,s?,pebble] 6402 !	github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/table_stats.go:46 +0xcc

Jira issue: CRDB-31541

Metadata

Metadata

Assignees

Labels

A-storageRelating to our storage engine (Pebble) on-disk storage.C-test-failureBroken test (automatically or manually discovered).T-storageStorage Teambranch-masterFailures and bugs on the master branch.

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions