Skip to content

sql: v22.1.3: nil ctx in zigzagJoiner.close #84902

Closed
#91969
#91446
Closed
sql: v22.1.3: nil ctx in zigzagJoiner.close#84902
#91969
#91446
Assignees
yuzefovich
Labels
C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.O-sentryOriginated from an in-the-wild panic report.T-sql-queriesSQL Queries Team
@cockroach-teamcity

Description

@cockroach-teamcity
cockroach-teamcity
opened on Jul 22, 2022

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3447276875/?referrer=webhooks_plugin

Panic message:

conn_executor.go:746: runtime error: invalid memory address or nil pointer dereference
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:746
| [...repeated from below...]
Wraps: (2) while executing: SELECT , , , _ FROM _ WHERE ((( > $1) AND ( = $2)) AND ( = $3)) AND (_ IN (SELECT unnest($4))) ORDER BY _ ASC LIMIT $5
Wraps: (3) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:746
| runtime.gopanic
| GOROOT/src/runtime/panic.go:1038
| runtime.panicmem
| GOROOT/src/runtime/panic.go:221
| runtime.sigpanic
| GOROOT/src/runtime/signal_unix.go:735
| github.com/cockroachdb/cockroach/pkg/util/tracing.SpanFromContext
| github.com/cockroachdb/cockroach/pkg/util/tracing/context.go:25
| github.com/cockroachdb/cockroach/pkg/util/log.getSpanOrEventLog
| github.com/cockroachdb/cockroach/pkg/util/log/trace.go:91
| github.com/cockroachdb/cockroach/pkg/util/log.vEventf
| github.com/cockroachdb/cockroach/pkg/util/log/trace.go:206
| github.com/cockroachdb/cockroach/pkg/util/log.VEventf
| github.com/cockroachdb/cockroach/pkg/util/log/trace.go:232
| github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*zigzagJoiner).close
| github.com/cockroachdb/cockroach/pkg/sql/rowexec/zigzagjoiner.go:523
| github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*zigzagJoiner).ConsumerClosed
| github.com/cockroachdb/cockroach/pkg/sql/rowexec/zigzagjoiner.go:995
| github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBaseNoHelper).InternalCloseEx
| github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:946
| github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBase).InternalCloseEx
| github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:924
| github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBase).InternalClose
| github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:915
| github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*joinReader).close
| github.com/cockroachdb/cockroach/pkg/sql/rowexec/joinreader.go:1057
| github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*joinReader).ConsumerClosed
| github.com/cockroachdb/cockroach/pkg/sql/rowexec/joinreader.go:1053
| github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBaseNoHelper).InternalCloseEx
| github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:946
| github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBaseNoHelper).InternalClose
| github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:934
| github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).Close
| github.com/cockroachdb/cockroach/pkg/sql/colexec/columnarizer.go:296
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).close
| github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:339
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run.func1
| github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:269
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run
| github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:283
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
| github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:260
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:597
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1445
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1471
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1145
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:686
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:143
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:231
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1952
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1954
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1800
Wraps: (4) runtime error: invalid memory address or nil pointer dereference
Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *withstack.withStack (4) runtime.errorString
-- report composition:
runtime.errorString
conn_executor.go:746: *withstack.withStack (top exception)
*safedetails.withSafeDetails: while executing: SELECT , , , _ FROM _ WHERE ((( > $1) AND ( = $2)) AND ( = $3)) AND (_ IN (SELECT unnest($4))) ORDER BY _ ASC LIMIT $5
conn_executor.go:746: *withstack.withStack (1)
(check the extra data payloads)

Stacktrace (expand for inline code snippets):

cockroach/pkg/sql/conn_executor.go

Lines 745 to 747 in 2ca7454

r := recover()
h.ex.closeWrapper(ctx, r)
}()
in pkg/sql.(*Server).ServeConn.func1
GOROOT/src/runtime/panic.go#L1037-L1039 in runtime.gopanic
GOROOT/src/runtime/panic.go#L220-L222 in runtime.panicmem
GOROOT/src/runtime/signal_unix.go#L734-L736 in runtime.sigpanic

cockroach/pkg/util/tracing/context.go

Lines 24 to 26 in 2ca7454

func SpanFromContext(ctx context.Context) *Span {
val := ctx.Value(activeSpanKey{})
if sp, ok := val.(*Span); ok {
in pkg/util/tracing.SpanFromContext

cockroach/pkg/util/log/trace.go

Lines 90 to 92 in 2ca7454

func getSpanOrEventLog(ctx context.Context) (*tracing.Span, *ctxEventLog, bool) {
if sp := tracing.SpanFromContext(ctx); sp != nil {
if !sp.IsVerbose() && !sp.Tracer().HasExternalSink() {
in pkg/util/log.getSpanOrEventLog

cockroach/pkg/util/log/trace.go

Lines 205 to 207 in 2ca7454

} else {
sp, el, ok := getSpanOrEventLog(ctx)
if !ok {
in pkg/util/log.vEventf

cockroach/pkg/util/log/trace.go

Lines 231 to 233 in 2ca7454

func VEventf(ctx context.Context, level Level, format string, args ...interface{}) {
vEventf(ctx, false /* isErr */, 1, level, format, args...)
}
in pkg/util/log.VEventf

cockroach/pkg/sql/rowexec/zigzagjoiner.go

Lines 522 to 524 in 2ca7454

}
log.VEventf(z.Ctx, 2, "exiting zigzag joiner run")
}
in pkg/sql/rowexec.(*zigzagJoiner).close

cockroach/pkg/sql/rowexec/zigzagjoiner.go

Lines 994 to 996 in 2ca7454

func (z *zigzagJoiner) ConsumerClosed() {
z.close()
}
in pkg/sql/rowexec.(*zigzagJoiner).ConsumerClosed

cockroach/pkg/sql/execinfra/processorsbase.go

Lines 945 to 947 in 2ca7454

for _, input := range pb.inputsToDrain[pb.curInputToDrain:] {
input.ConsumerClosed()
}
in pkg/sql/execinfra.(*ProcessorBaseNoHelper).InternalCloseEx

cockroach/pkg/sql/execinfra/processorsbase.go

Lines 923 to 925 in 2ca7454

func (pb *ProcessorBase) InternalCloseEx(onClose func()) bool {
closing := pb.ProcessorBaseNoHelper.InternalCloseEx(onClose)
if closing {
in pkg/sql/execinfra.(*ProcessorBase).InternalCloseEx

cockroach/pkg/sql/execinfra/processorsbase.go

Lines 914 to 916 in 2ca7454

func (pb *ProcessorBase) InternalClose() bool {
return pb.InternalCloseEx(nil /* onClose */)
}
in pkg/sql/execinfra.(*ProcessorBase).InternalClose

cockroach/pkg/sql/rowexec/joinreader.go

Lines 1056 to 1058 in 2ca7454

func (jr *joinReader) close() {
if jr.InternalClose() {
if jr.fetcher != nil {
in pkg/sql/rowexec.(*joinReader).close

cockroach/pkg/sql/rowexec/joinreader.go

Lines 1052 to 1054 in 2ca7454

// The consumer is done, Next() will not be called again.
jr.close()
}
in pkg/sql/rowexec.(*joinReader).ConsumerClosed

cockroach/pkg/sql/execinfra/processorsbase.go

Lines 945 to 947 in 2ca7454

for _, input := range pb.inputsToDrain[pb.curInputToDrain:] {
input.ConsumerClosed()
}
in pkg/sql/execinfra.(*ProcessorBaseNoHelper).InternalCloseEx

cockroach/pkg/sql/execinfra/processorsbase.go

Lines 933 to 935 in 2ca7454

func (pb *ProcessorBaseNoHelper) InternalClose() bool {
return pb.InternalCloseEx(nil /* onClose */)
}
in pkg/sql/execinfra.(*ProcessorBaseNoHelper).InternalClose

cockroach/pkg/sql/colexec/columnarizer.go

Lines 295 to 297 in 2ca7454

}
c.InternalClose()
return nil
in pkg/sql/colexec.(*Columnarizer).Close

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 338 to 340 in 2ca7454

for _, toClose := range f.input.ToClose {
if err := toClose.Close(ctx); err != nil {
lastErr = err
in pkg/sql/colflow.(*BatchFlowCoordinator).close

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 268 to 270 in 2ca7454

defer func() {
if err := f.close(ctx); err != nil && status != execinfra.ConsumerClosed {
f.pushError(err)
in pkg/sql/colflow.(*BatchFlowCoordinator).Run.func1

cockroach/pkg/sql/colflow/flow_coordinator.go

Lines 282 to 284 in 2ca7454

// tree might not be setup properly.
return
}
in pkg/sql/colflow.(*BatchFlowCoordinator).Run

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 259 to 261 in 2ca7454

log.VEvent(ctx, 1, "running the batch flow coordinator in the flow's goroutine")
f.batchFlowCoordinator.Run(ctx)
}
in pkg/sql/colflow.(*vectorizedFlow).Run

cockroach/pkg/sql/distsql_running.go

Lines 596 to 598 in 2ca7454

// TODO(radu): this should go through the flow scheduler.
flow.Run(ctx, func() {})
in pkg/sql.(*DistSQLPlanner).Run

cockroach/pkg/sql/distsql_running.go

Lines 1444 to 1446 in 2ca7454

recv.expectedRowsRead = int64(physPlan.TotalEstimatedScannedRows)
runCleanup := dsp.Run(ctx, planCtx, txn, physPlan, recv, evalCtx, nil /* finishedSetupFn */)
return func() {
in pkg/sql.(*DistSQLPlanner).PlanAndRun

cockroach/pkg/sql/conn_executor_exec.go

Lines 1470 to 1472 in 2ca7454

// the planner whether or not to plan remote table readers.
cleanup := ex.server.cfg.DistSQLPlanner.PlanAndRun(
ctx, evalCtx, planCtx, planner.txn, planner.curPlan.main, recv,
in pkg/sql.(*connExecutor).execWithDistSQLEngine

cockroach/pkg/sql/conn_executor_exec.go

Lines 1144 to 1146 in 2ca7454

ex.sessionTracing.TraceExecStart(ctx, "distributed")
stats, err := ex.execWithDistSQLEngine(
ctx, planner, stmt.AST.StatementReturnType(), res, distribute, progAtomic,
in pkg/sql.(*connExecutor).dispatchToExecutionEngine

cockroach/pkg/sql/conn_executor_exec.go

Lines 685 to 687 in 2ca7454

if err := ex.dispatchToExecutionEngine(stmtCtx, p, res); err != nil {
stmtThresholdSpan.Finish()
in pkg/sql.(*connExecutor).execStmtInOpenState

cockroach/pkg/sql/conn_executor_exec.go

Lines 142 to 144 in 2ca7454

} else {
ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, prepared, pinfo, res, canAutoCommit)
}
in pkg/sql.(*connExecutor).execStmt

cockroach/pkg/sql/conn_executor_exec.go

Lines 230 to 232 in 2ca7454

}
ev, payload, err = ex.execStmt(ctx, portal.Stmt.Statement, portal.Stmt, pinfo, stmtRes, canAutoCommit)
// Portal suspension is supported via a "side" state machine
in pkg/sql.(*connExecutor).execPortal

cockroach/pkg/sql/conn_executor.go

Lines 1951 to 1953 in 2ca7454

canAutoCommit := ex.implicitTxn() && tcmd.FollowedBySync
ev, payload, err = ex.execPortal(ctx, portal, portalName, stmtRes, pinfo, canAutoCommit)
return err
in pkg/sql.(*connExecutor).execCmd.func2

cockroach/pkg/sql/conn_executor.go

Lines 1953 to 1955 in 2ca7454

return err
}()
// Note: we write to ex.statsCollector.phaseTimes, instead of ex.phaseTimes,
in pkg/sql.(*connExecutor).execCmd

cockroach/pkg/sql/conn_executor.go

Lines 1799 to 1801 in 2ca7454

var err error
if err = ex.execCmd(); err != nil {
if errors.IsAny(err, io.EOF, errDrainingComplete) {
in pkg/sql.(*connExecutor).run

pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn.func1 at line 746
GOROOT/src/runtime/panic.go in runtime.gopanic at line 1038
GOROOT/src/runtime/panic.go in runtime.panicmem at line 221
GOROOT/src/runtime/signal_unix.go in runtime.sigpanic at line 735
pkg/util/tracing/context.go in pkg/util/tracing.SpanFromContext at line 25
pkg/util/log/trace.go in pkg/util/log.getSpanOrEventLog at line 91
pkg/util/log/trace.go in pkg/util/log.vEventf at line 206
pkg/util/log/trace.go in pkg/util/log.VEventf at line 232
pkg/sql/rowexec/zigzagjoiner.go in pkg/sql/rowexec.(*zigzagJoiner).close at line 523
pkg/sql/rowexec/zigzagjoiner.go in pkg/sql/rowexec.(*zigzagJoiner).ConsumerClosed at line 995
pkg/sql/execinfra/processorsbase.go in pkg/sql/execinfra.(*ProcessorBaseNoHelper).InternalCloseEx at line 946
pkg/sql/execinfra/processorsbase.go in pkg/sql/execinfra.(*ProcessorBase).InternalCloseEx at line 924
pkg/sql/execinfra/processorsbase.go in pkg/sql/execinfra.(*ProcessorBase).InternalClose at line 915
pkg/sql/rowexec/joinreader.go in pkg/sql/rowexec.(*joinReader).close at line 1057
pkg/sql/rowexec/joinreader.go in pkg/sql/rowexec.(*joinReader).ConsumerClosed at line 1053
pkg/sql/execinfra/processorsbase.go in pkg/sql/execinfra.(*ProcessorBaseNoHelper).InternalCloseEx at line 946
pkg/sql/execinfra/processorsbase.go in pkg/sql/execinfra.(*ProcessorBaseNoHelper).InternalClose at line 934
pkg/sql/colexec/columnarizer.go in pkg/sql/colexec.(*Columnarizer).Close at line 296
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).close at line 339
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).Run.func1 at line 269
pkg/sql/colflow/flow_coordinator.go in pkg/sql/colflow.(*BatchFlowCoordinator).Run at line 283
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlow).Run at line 260
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).Run at line 597
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRun at line 1445
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithDistSQLEngine at line 1471
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1145
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 686
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 143
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execPortal at line 231
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func2 at line 1952
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 1954
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 1800
Tag Value
Cockroach Release v22.1.3
Cockroach SHA: 2ca7454
Platform linux amd64
Distribution CCL
Environment v22.1.3
Command start-single-node
Go Version ``
# of CPUs
# of Goroutines

Jira issue: CRDB-17939

Metadata

Metadata

Assignees

Labels

C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.O-sentryOriginated from an in-the-wild panic report.T-sql-queriesSQL Queries Team

Type

No type

Projects

SQL Queries

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions