This test failed on a CI run on #10534:
https://github.com/oxidecomputer/omicron/pull/10534/checks?check_run_id=79380945518
Log showing the specific test failure:
https://buildomat.eng.oxide.computer/wg/0/details/01KT7G8DCZ660QZZ4JWBYT8F5M/vKy6chpHoq098zym3392IT9aQ8BAK409vTvnqGsFV29od6cB/01KT7G8V70HH9A9X8P8HPFBDZC#S8832
Excerpt from the log showing the failure:
TRY 1 FAIL [ 12.356s] (─────────) omicron-nexus::test_all integration_tests::updates::test_update_status
stdout ───
running 1 test
test integration_tests::updates::test_update_status ... FAILED
failures:
failures:
integration_tests::updates::test_update_status
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 562 filtered out; finished in 11.80s
stderr ───
log file: /var/tmp/omicron_tmp/test_all-9c1cbd77b0b87f6b-test_update_status.22413.0.log
note: configured to log to "/var/tmp/omicron_tmp/test_all-9c1cbd77b0b87f6b-test_update_status.22413.0.log"
DB URL: postgresql://root@[::1]:38178/omicron?sslmode=disable
DB address: [::1]:38178
log file: /var/tmp/omicron_tmp/test_all-9c1cbd77b0b87f6b-test_update_status.22413.2.log
note: configured to log to "/var/tmp/omicron_tmp/test_all-9c1cbd77b0b87f6b-test_update_status.22413.2.log"
thread 'integration_tests::updates::test_update_status' (2) panicked at nexus/test-utils/src/resource_helpers.rs:132:13:
failed to make "GET" request to /v1/system/update/status: expected status code 200 OK, found 500 Internal Server Error
stack backtrace:
0: __rustc::rust_begin_unwind
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/std/src/panicking.rs:689:5
1: core::panicking::panic_fmt
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/panicking.rs:80:14
2: nexus_test_utils::resource_helpers::object_get::<nexus_types_versions::v2026_05_20_00::update::UpdateStatus>::{closure#0}::{closure#0}
at ./test-utils/src/resource_helpers.rs:132:13
3: <core::result::Result<nexus_test_utils::http_testing::TestResponse, anyhow::Error>>::unwrap_or_else::<nexus_test_utils::resource_helpers::object_get<nexus_types_versions::v2026_05_20_00::update::UpdateStatus>::{closure#0}::{closure#0}>
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/result.rs:1622:23
4: {async_fn#0}<nexus_types_versions::v2026_05_20_00::update::UpdateStatus>
at ./test-utils/src/resource_helpers.rs:131:10
5: {async_block#0}
at ./tests/integration_tests/updates.rs:752:56
6: poll<&mut dyn core::future::future::Future<Output=core::result::Result<(), anyhow::Error>>>
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/future/future.rs:133:9
7: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=core::result::Result<(), anyhow::Error>>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/park.rs:284:71
8: with_budget<core::task::poll::Poll<core::result::Result<(), anyhow::Error>>, tokio::runtime::park::{impl#4}::block_on::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=core::result::Result<(), anyhow::Error>>>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/task/coop/mod.rs:167:5
9: budget<core::task::poll::Poll<core::result::Result<(), anyhow::Error>>, tokio::runtime::park::{impl#4}::block_on::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=core::result::Result<(), anyhow::Error>>>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/task/coop/mod.rs:133:5
10: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=core::result::Result<(), anyhow::Error>>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/park.rs:284:31
11: <tokio::runtime::context::blocking::BlockingRegionGuard>::block_on::<core::pin::Pin<&mut dyn core::future::future::Future<Output = core::result::Result<(), anyhow::Error>>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/context/blocking.rs:66:14
12: <tokio::runtime::scheduler::multi_thread::MultiThread>::block_on::<core::pin::Pin<&mut dyn core::future::future::Future<Output = core::result::Result<(), anyhow::Error>>>>::{closure#0}
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/multi_thread/mod.rs:92:22
13: tokio::runtime::context::runtime::enter_runtime::<<tokio::runtime::scheduler::multi_thread::MultiThread>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = core::result::Result<(), anyhow::Error>>>>::{closure#0}, core::result::Result<(), anyhow::Error>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/context/runtime.rs:65:16
14: <tokio::runtime::scheduler::multi_thread::MultiThread>::block_on::<core::pin::Pin<&mut dyn core::future::future::Future<Output = core::result::Result<(), anyhow::Error>>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/multi_thread/mod.rs:91:9
15: <tokio::runtime::runtime::Runtime>::block_on_inner::<core::pin::Pin<&mut dyn core::future::future::Future<Output = core::result::Result<(), anyhow::Error>>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/runtime.rs:373:50
16: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=core::result::Result<(), anyhow::Error>>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/runtime.rs:345:18
17: test_update_status
at ./tests/integration_tests/updates.rs:833:7
18: test_all::integration_tests::updates::test_update_status::{closure#0}
at ./tests/integration_tests/updates.rs:737:34
19: <test_all::integration_tests::updates::test_update_status::{closure#0} as core::ops::function::FnOnce<()>>::call_once
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/ops/function.rs:250:5
20: core::ops::function::FnOnce::call_once
at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
WARN: temporary directory leaked: "/var/tmp/omicron_tmp/.tmpp7F64A"
If you would like to access the database for debugging, run the following:
# Run the database
cargo xtask db-dev run --no-populate --store-dir "/var/tmp/omicron_tmp/.tmpp7F64A/data"
# Access the database. Note the port may change if you run multiple databases.
cockroach sql --host=localhost:32221 --insecure
WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 22466) and a temporary directory leaked, /var/tmp/omicron_tmp/test_all-9c1cbd77b0b87f6b-test_update_status.22413.1-clickhouse-P5Fi8u)
failed to clean up ClickHouse data dir:
- /var/tmp/omicron_tmp/test_all-9c1cbd77b0b87f6b-test_update_status.22413.1-clickhouse-P5Fi8u: File exists (os error 17)
WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
WARN: dendrite temporary directory leaked: /var/tmp/omicron_tmp/.tmpUugoPi
WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
WARN: mgd temporary directory leaked: /var/tmp/omicron_tmp/.tmp4rb2Uz
This one is kind of strange. Looking at the logs from the failing test itself, it seems like there are errors connecting to the database. The first one is here:
2026-06-03T20:13:59.002Z WARN test_update_status (ServerContext): Failed is_valid check; connection failed ping
error = no connection to the server\n
Looking at the CRDB log, things definitely seem unhappy, but I'm not familiar with what normal behavior looks like. I do see that the server says it was terminated, so this could have been interrupted somehow. I definitely canceled an earlier CI run, but maybe that didn't do what I thought it did.
This test failed on a CI run on #10534:
https://github.com/oxidecomputer/omicron/pull/10534/checks?check_run_id=79380945518
Log showing the specific test failure:
https://buildomat.eng.oxide.computer/wg/0/details/01KT7G8DCZ660QZZ4JWBYT8F5M/vKy6chpHoq098zym3392IT9aQ8BAK409vTvnqGsFV29od6cB/01KT7G8V70HH9A9X8P8HPFBDZC#S8832
Excerpt from the log showing the failure:
This one is kind of strange. Looking at the logs from the failing test itself, it seems like there are errors connecting to the database. The first one is here:
Looking at the CRDB log, things definitely seem unhappy, but I'm not familiar with what normal behavior looks like. I do see that the server says it was terminated, so this could have been interrupted somehow. I definitely canceled an earlier CI run, but maybe that didn't do what I thought it did.