Skip to content

Pantry crashed when processing a region with mismatched information #6353

@askfongjojo

Description

@askfongjojo

This issue was seen on rack2 after it was updated to omicron commit 4a6be3a40f6c9f876ff53308532f6c37f030b13f. Here are the log lines at the time pantry crashed:

angela@jeeves /staff/core/omicron-pantry-crash $ tail -15 oxide-crucible-pantry\:default.log.1723665490 | looker
19:58:02.159Z INFO crucible-pantry (datafile): client task is exiting
     = downstairs
    client = 2
    session_id = 4b49aac8-6f89-466b-8df4-ce378ea59121
19:58:09.762Z INFO crucible-pantry (datafile): connecting to [fd00:1122:3344:10a::7]:19010
     = downstairs
    client = 0
    session_id = ba4bf6b6-8a2b-4d5e-b861-8ce61b8119a7
19:58:09.762Z INFO crucible-pantry (datafile): ds_connection connected from Ok([fd00:1122:3344:105::4]:41371)
     = downstairs
    client = 0
    session_id = ba4bf6b6-8a2b-4d5e-b861-8ce61b8119a7
19:58:09.762Z INFO crucible-pantry (datafile): ds_transition from New to WaitActive
     = downstairs
    client = 0
    session_id = ba4bf6b6-8a2b-4d5e-b861-8ce61b8119a7
19:58:09.762Z INFO crucible-pantry (datafile): [0] Transition from New to WaitActive
     = downstairs
    client = 0
    session_id = ba4bf6b6-8a2b-4d5e-b861-8ce61b8119a7
19:58:09.763Z INFO crucible-pantry (datafile): downstairs client at Some([fd00:1122:3344:10a::7]:19010) has region UUID 6bf93010-3719-41bf-bc51-a03cd4442b89
     = downstairs
    client = 0
    session_id = ba4bf6b6-8a2b-4d5e-b861-8ce61b8119a7
19:58:09.763Z WARN crucible-pantry (datafile): Replace e03d2ffc-792f-4b2f-b1df-7a477b9ea19b with 6bf93010-3719-41bf-bc51-a03cd4442b89 before active
     = downstairs
    client = 0
    session_id = ba4bf6b6-8a2b-4d5e-b861-8ce61b8119a7
thread 'tokio-runtime-worker' panicked at upstairs/src/client.rs:1981:25:
[0] New downstairs region info mismatch: Received(RegionDefinition { block_size: 512, extent_size: Block { value: 131072, shift: 9 }, extent_count: 16, uuid: 79160ed2-b164-444f-853a-3d83242b79d9, encrypted: true, database_read_version: 1, database_write_version: 1 }) vs. RegionDefinition { block_size: 512, extent_size: Block { value: 131072, shift: 9 }, extent_count: 160, uuid: 6bf93010-3719-41bf-bc51-a03cd4442b89, encrypted: true, database_read_version: 1, database_write_version: 1 }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[ Aug 14 19:58:10 Stopping because all processes in service exited. ]
[ Aug 14 19:58:10 Executing start method ("/opt/oxide/pantry/bin/crucible-pantry run -l [fd00:1122:3344:105::4]:17000"). ]
note: configured to log to "/dev/stdout"
19:58:10.415Z INFO crucible-pantry (dropshot): listening
    local_addr = [fd00:1122:3344:105::4]:17000
19:58:10.428Z INFO crucible-pantry: listen IP: [fd00:1122:3344:105::4]:17000

The core file is available in catacomb under /staff/dock/rack2/mupdate-20240815/cores.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions