roachtest: acceptance/version-upgrade failed #43957
Description
I broke the issue filing (fixing in #43956) so these fell off the radar:
https://teamcity.cockroachdb.com/viewLog.html?buildId=1686776&buildTypeId=Cockroach_UnitTests
https://teamcity.cockroachdb.com/viewLog.html?buildId=1687798&buildTypeId=Cockroach_UnitTests
Both are from today and have a node crash of this kind (TLDR: no inbound stream connection)
That error unfortunately just means "the other node didn't connect to us". Could it not be doing that because of some version incompatibility that we introduced?
@solongordon I think this must've become a bug when you rewrote this method in f8faf89? Perhaps it's just a bad idea to use distsql in a migration because if there's a version bump between the nodes some old nodes might refuse the inbound connection (or so I'm imagining this breaks).
F200113 16:42:41.482752 101 server/server.go:1623 [n3] error with attached stack trace:
github.com/cockroachdb/cockroach/pkg/sql.(*internalExecutorImpl).execInternal.func1
/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:477
github.com/cockroachdb/cockroach/pkg/sql.(*internalExecutorImpl).execInternal
/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:574
github.com/cockroachdb/cockroach/pkg/sql.(*internalExecutorImpl).queryInternal
/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:252
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryWithUser
/go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:269
github.com/cockroachdb/cockroach/pkg/sqlmigrations.migrateSystemNamespace
/go/src/github.com/cockroachdb/cockroach/pkg/sqlmigrations/migrations.go:708
github.com/cockroachdb/cockroach/pkg/sqlmigrations.(*Manager).EnsureMigrations
/go/src/github.com/cockroachdb/cockroach/pkg/sqlmigrations/migrations.go:573
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start
/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1617
github.com/cockroachdb/cockroach/pkg/cli.runStart.func3.2
/go/src/github.com/cockroachdb/cockroach/pkg/cli/start.go:698
github.com/cockroachdb/cockroach/pkg/cli.runStart.func3
/go/src/github.com/cockroachdb/cockroach/pkg/cli/start.go:814
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1357
- error with embedded safe details: read-deprecated-namespace-table
- read-deprecated-namespace-table:
- no inbound stream connection
github.com/cockroachdb/cockroach/pkg/sql/flowinfra.init
/go/src/github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow_registry.go:30
runtime.doInit
/usr/local/go/src/runtime/proc.go:5222
runtime.doInit
/usr/local/go/src/runtime/proc.go:5217
runtime.doInit
/usr/local/go/src/runtime/proc.go:5217
runtime.doInit
/usr/local/go/src/runtime/proc.go:5217
runtime.doInit
/usr/local/go/src/runtime/proc.go:5217
runtime.doInit
/usr/local/go/src/runtime/proc.go:5217
runtime.main
/usr/local/go/src/runtime/proc.go:190
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1357
failed to run migration "migrate system.namespace_deprecated entries into system.namespace"
github.com/cockroachdb/cockroach/pkg/sqlmigrations.(*Manager).EnsureMigrations
/go/src/github.com/cockroachdb/cockroach/pkg/sqlmigrations/migrations.go:574
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start
/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1617
github.com/cockroachdb/cockroach/pkg/cli.runStart.func3.2
/go/src/github.com/cockroachdb/cockroach/pkg/cli/start.go:698
github.com/cockroachdb/cockroach/pkg/cli.runStart.func3
/go/src/github.com/cockroachdb/cockroach/pkg/cli/start.go:814
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1357
goroutine 101 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0x6ea9c01, 0xed5ae9501, 0x0, 0x47a6ea0)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/get_stacks.go:25 +0xb8
github.com/cockroachdb/cockroach/pkg/util/log.(*loggerT).outputLogEntry(0x6ea6a60, 0xc000000004, 0x65311ef, 0x10, 0x657, 0xc003d80c00, 0xb52)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:211 +0xa0c
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x47a6da0, 0xc0006b9380, 0x4000000000000004, 0x2, 0x3f66d30, 0x3, 0xc0039a2d38, 0x1, 0x1)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:66 +0x2c9
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x47a6da0, 0xc0006b9380, 0x1, 0x4, 0x3f66d30, 0x3, 0xc0039a2d38, 0x1, 0x1)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:44 +0x8c
github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(...)
/go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:155
github.com/cockroachdb/cockroach/pkg/server.(*Server).Start(0xc0003aa800, 0x47a6da0, 0xc0006bc1e0, 0x0, 0x0)
/go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1623 +0x2b54
github.com/cockroachdb/cockroach/pkg/cli.runStart.func3.2(0xc000852120, 0xc0003de118, 0xc0001f8060, 0x47a6da0, 0xc0006bc1e0, 0x0, 0x2bbd3c01, 0xed5ae94f6, 0x0, 0x0, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/cli/start.go:698 +0x107
github.com/cockroachdb/cockroach/pkg/cli.runStart.func3(0xc0003de118, 0x47a6da0, 0xc0006bc1e0, 0x480d4e0, 0xc00027a580, 0xc000852120, 0xc0001f8060, 0x0, 0x2bbd3c01, 0xed5ae94f6, ...)
/go/src/github.com/cockroachdb/cockroach/pkg/cli/start.go:814 +0x181
created by github.com/cockroachdb/cockroach/pkg/cli.runStart
/go/src/github.com/cockroachdb/cockroach/pkg/cli/start.go:654 +0x9d8