Skip to content

stability: panic on RangeLookup dispatched to correct range #6000

@mberhault

Description

@mberhault

build sha: c382dd7
Beta cluster with photos + block_writer load. Restarted from scratch earlier today.

~1h after a restart, encountered the following:

panic: RangeLookup dispatched to correct range, but no matching RangeDescriptor was found: /Meta2/Table/55/1/8752590996753796817/"88e08dd7-d1b6-4aeb-ad46-1880b351e731"/6611/0

1: running [Created by grpc.(*Server).serveStreams.func1 @ server.go:324]
              panic.go:464            panic(#1, #18)
    storage   store.go:1557           (*Store).Send.func1(0xc833bd1258, 0xc833bd12c8, 0x14449b721ca69e59, 0, 0xc833bd12c0, #5)
              panic.go:426            panic(#1, #18)
    storage   replica_command.go:909  (*Replica).RangeLookup(#12, #19, #9, #22, 0, #4, 0x4, 0x1, 0, 0, ...)
    storage   replica_command.go:129  (*Replica).executeCmd(#12, #20, #17, 0, 0, 0, #19, #9, 0, #22, ...)
    storage   replica.go:1784         (*Replica).executeBatch(#12, #20, #17, 0, 0, #19, #9, 0, #22, 0, ...)
    storage   replica.go:1071         (*Replica).addReadOnlyCmd(#12, #20, #17, #22, 0, #4, 0x4, 0x1, 0, 0, ...)
    storage   replica.go:765          (*Replica).Send(#12, #20, #17, #22, 0, #4, 0x4, 0x1, 0, 0, ...)
    storage   store.go:1639           (*Store).Send(#5, #20, #17, #22, 0, #4, 0x4, 0x1, 0, 0, ...)
    storage   stores.go:177           (*Stores).Send(#6, #20, 0xc82bb06c30, 0, 0, #4, 0x4, 0x1, 0, 0, ...)
    server    node.go:729             (*Node).Batch.func2()
    stop      stopper.go:166          (*Stopper).RunTask(#8, 0xc833bd1870, 0)
    server    node.go:741             (*Node).Batch(#7, #20, #16, 0xc8271b2150, 0, 0, 0)
    roachpb   api.pb.go:1306          _Internal_Batch_Handler(#2, #7, #20, #16, 0xc833082a00, 0, 0, 0, 0)
    grpc      server.go:497           (*Server).processUnaryRPC(#11, #21, #14, #15, #10, #3, 0, 0, 0)
    grpc      server.go:646           (*Server).handleStream(#11, #21, #14, #15, 0)
    grpc      server.go:323           (*Server).serveStreams.func1.1(#13, #11, #21, #14, #15)

Full log (from ec2-52-91-234-29.compute-1.amazonaws.com):
node4.log.parse.txt

Metadata

Metadata

Assignees

No one assigned

    Labels

    S-1-stabilitySevere stability issues that can be fixed by upgrading, but usually don’t resolve by restarting

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions