angela@jeeves /staff/core/omicron-pantry-crash $ tail -15 oxide-crucible-pantry\:default.log.1723665490 | looker
19:58:02.159Z INFO crucible-pantry (datafile): client task is exiting
= downstairs
client = 2
session_id = 4b49aac8-6f89-466b-8df4-ce378ea59121
19:58:09.762Z INFO crucible-pantry (datafile): connecting to [fd00:1122:3344:10a::7]:19010
= downstairs
client = 0
session_id = ba4bf6b6-8a2b-4d5e-b861-8ce61b8119a7
19:58:09.762Z INFO crucible-pantry (datafile): ds_connection connected from Ok([fd00:1122:3344:105::4]:41371)
= downstairs
client = 0
session_id = ba4bf6b6-8a2b-4d5e-b861-8ce61b8119a7
19:58:09.762Z INFO crucible-pantry (datafile): ds_transition from New to WaitActive
= downstairs
client = 0
session_id = ba4bf6b6-8a2b-4d5e-b861-8ce61b8119a7
19:58:09.762Z INFO crucible-pantry (datafile): [0] Transition from New to WaitActive
= downstairs
client = 0
session_id = ba4bf6b6-8a2b-4d5e-b861-8ce61b8119a7
19:58:09.763Z INFO crucible-pantry (datafile): downstairs client at Some([fd00:1122:3344:10a::7]:19010) has region UUID 6bf93010-3719-41bf-bc51-a03cd4442b89
= downstairs
client = 0
session_id = ba4bf6b6-8a2b-4d5e-b861-8ce61b8119a7
19:58:09.763Z WARN crucible-pantry (datafile): Replace e03d2ffc-792f-4b2f-b1df-7a477b9ea19b with 6bf93010-3719-41bf-bc51-a03cd4442b89 before active
= downstairs
client = 0
session_id = ba4bf6b6-8a2b-4d5e-b861-8ce61b8119a7
thread 'tokio-runtime-worker' panicked at upstairs/src/client.rs:1981:25:
[0] New downstairs region info mismatch: Received(RegionDefinition { block_size: 512, extent_size: Block { value: 131072, shift: 9 }, extent_count: 16, uuid: 79160ed2-b164-444f-853a-3d83242b79d9, encrypted: true, database_read_version: 1, database_write_version: 1 }) vs. RegionDefinition { block_size: 512, extent_size: Block { value: 131072, shift: 9 }, extent_count: 160, uuid: 6bf93010-3719-41bf-bc51-a03cd4442b89, encrypted: true, database_read_version: 1, database_write_version: 1 }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[ Aug 14 19:58:10 Stopping because all processes in service exited. ]
[ Aug 14 19:58:10 Executing start method ("/opt/oxide/pantry/bin/crucible-pantry run -l [fd00:1122:3344:105::4]:17000"). ]
note: configured to log to "/dev/stdout"
19:58:10.415Z INFO crucible-pantry (dropshot): listening
local_addr = [fd00:1122:3344:105::4]:17000
19:58:10.428Z INFO crucible-pantry: listen IP: [fd00:1122:3344:105::4]:17000
This issue was seen on rack2 after it was updated to omicron commit
4a6be3a40f6c9f876ff53308532f6c37f030b13f. Here are the log lines at the time pantry crashed:The core file is available in catacomb under
/staff/dock/rack2/mupdate-20240815/cores.