Skip to content

kvserver: nil pointer exception in TestDecommission #83402

@stevendanna

Description

@stevendanna

Describe the problem

Seen here in CI: https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_UnitTests/5575109?showRootCauses=false&expandBuildChangesSection=true&expandBuildProblemsSection=true&expandBuildTestsSection=true

I haven't been able to reproduce this via stress, but this test is rather resource intensive so it takes a while to stress.

=== RUN   TestDecommission
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestDecommission459668680
    test_log_scope.go:80: use -show-logs to present logs inline
*
* INFO: Running test with the default test tenant. If you are only seeing a test case failure when this message appears, there may be a problem with your test case running within tenants.
*
    panic.go:1038: -- test log scope end --
ERROR: a panic has occurred!
Details cannot be printed yet because we are still unwinding.
Hopefully the test harness prints the panic below, otherwise check the test logs.
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestDecommission459668680
    panic.go:1038: runtime error: invalid memory address or nil pointer dereference
        goroutine 473340 [running]:
        runtime/debug.Stack()
          /usr/local/go/src/runtime/debug/stack.go:24 +0x65
        github.com/cockroachdb/cockroach/pkg/util/leaktest.AfterTest.func1()
          /go/src/github.com/cockroachdb/cockroach/pkg/util/leaktest/leaktest.go:110 +0x166
        panic({0x4032700, 0x8225a50})
          /usr/local/go/src/runtime/panic.go:1038 +0x215
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).maybeRejectUnsafeProposalLocked(0xc0023b52b8, {0x5bbbeb8, 0xc00b058150}, {0x0, 0x0}, 0xc004960240)
          /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_proposal_buf.go:690 +0x317
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).FlushLockedWithRaftGroup(0xc0023b52b8, {0x5bbbeb8, 0xc00b058150}, {0x0, 0x0})
          /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_proposal_buf.go:428 +0x3d6
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).FlushLockedWithoutProposing(0xc003614090, {0x5bbbeb8, 0xc00b058150})
          /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/replica_proposal_buf.go:921 +0x2e
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaft.func2.1(0xc0023b4c00)
          /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:711 +0x74
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*storeReplicaVisitor).Visit(0xc00d8a2f30, 0xc007e93610)
          /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store.go:416 +0x1a8
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).VisitReplicas(0xc0066e2000, 0x0, {0x0, 0x0, 0xc007e93628})
          /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store.go:2633 +0x9e
        github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRaft.func2()
          /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/store_raft.go:709 +0x56
        github.com/cockroachdb/cockroach/pkg/util/stop.CloserFn.Close(0xc00f8bccd8)
          /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 +0x1a
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0xc00514e090, {0x5bbbe48, 0xc000074140})
          /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:558 +0x262
        github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).stopServerLocked(0xc003310000, 0x3)
          /go/src/github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:205 +0x4e
        github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).stopServers(0xc003310000, {0x5bbbe48, 0xc000074140})
          /go/src/github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:141 +0x2bc
        github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start.func2()
          /go/src/github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:438 +0x2b
        github.com/cockroachdb/cockroach/pkg/util/stop.CloserFn.Close(0xc0049fa240)
          /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 +0x1a
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0xc0049fa240, {0x5bbbe48, 0xc000074128})
          /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:558 +0x262
        github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestDecommission(0xc002ca8000)
          /go/src/github.com/cockroachdb/cockroach/pkg/kv/kvserver/client_raft_test.go:3231 +0x785
        testing.tRunner(0xc002ca8000, 0x4896da8)
          /usr/local/go/src/testing/testing.go:1259 +0x102
        created by testing.(*T).Run
          /usr/local/go/src/testing/testing.go:1306 +0x35a
--- FAIL: TestDecommission (32.09s)

Jira issue: CRDB-17053

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-bugCode not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions