Skip to content

test failed in CI: integration_tests::updates::test_update_status #10546

@bnaecker

Description

@bnaecker

This test failed on a CI run on #10534:

https://github.com/oxidecomputer/omicron/pull/10534/checks?check_run_id=79380945518

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01KT7G8DCZ660QZZ4JWBYT8F5M/vKy6chpHoq098zym3392IT9aQ8BAK409vTvnqGsFV29od6cB/01KT7G8V70HH9A9X8P8HPFBDZC#S8832

Excerpt from the log showing the failure:

TRY 1 FAIL [  12.356s] (─────────) omicron-nexus::test_all integration_tests::updates::test_update_status
stdout ───

  running 1 test
  test integration_tests::updates::test_update_status ... FAILED

  failures:

  failures:
      integration_tests::updates::test_update_status

  test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 562 filtered out; finished in 11.80s
  
stderr ───
  log file: /var/tmp/omicron_tmp/test_all-9c1cbd77b0b87f6b-test_update_status.22413.0.log
  note: configured to log to "/var/tmp/omicron_tmp/test_all-9c1cbd77b0b87f6b-test_update_status.22413.0.log"
  DB URL: postgresql://root@[::1]:38178/omicron?sslmode=disable
  DB address: [::1]:38178
  log file: /var/tmp/omicron_tmp/test_all-9c1cbd77b0b87f6b-test_update_status.22413.2.log
  note: configured to log to "/var/tmp/omicron_tmp/test_all-9c1cbd77b0b87f6b-test_update_status.22413.2.log"

  thread 'integration_tests::updates::test_update_status' (2) panicked at nexus/test-utils/src/resource_helpers.rs:132:13:
  failed to make "GET" request to /v1/system/update/status: expected status code 200 OK, found 500 Internal Server Error
  stack backtrace:
     0: __rustc::rust_begin_unwind
               at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/std/src/panicking.rs:689:5
     1: core::panicking::panic_fmt
               at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/panicking.rs:80:14
     2: nexus_test_utils::resource_helpers::object_get::<nexus_types_versions::v2026_05_20_00::update::UpdateStatus>::{closure#0}::{closure#0}
               at ./test-utils/src/resource_helpers.rs:132:13
     3: <core::result::Result<nexus_test_utils::http_testing::TestResponse, anyhow::Error>>::unwrap_or_else::<nexus_test_utils::resource_helpers::object_get<nexus_types_versions::v2026_05_20_00::update::UpdateStatus>::{closure#0}::{closure#0}>
               at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/result.rs:1622:23
     4: {async_fn#0}<nexus_types_versions::v2026_05_20_00::update::UpdateStatus>
               at ./test-utils/src/resource_helpers.rs:131:10
     5: {async_block#0}
               at ./tests/integration_tests/updates.rs:752:56
     6: poll<&mut dyn core::future::future::Future<Output=core::result::Result<(), anyhow::Error>>>
               at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/future/future.rs:133:9
     7: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=core::result::Result<(), anyhow::Error>>>>
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/park.rs:284:71
     8: with_budget<core::task::poll::Poll<core::result::Result<(), anyhow::Error>>, tokio::runtime::park::{impl#4}::block_on::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=core::result::Result<(), anyhow::Error>>>>>
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/task/coop/mod.rs:167:5
     9: budget<core::task::poll::Poll<core::result::Result<(), anyhow::Error>>, tokio::runtime::park::{impl#4}::block_on::{closure_env#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=core::result::Result<(), anyhow::Error>>>>>
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/task/coop/mod.rs:133:5
    10: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=core::result::Result<(), anyhow::Error>>>>
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/park.rs:284:31
    11: <tokio::runtime::context::blocking::BlockingRegionGuard>::block_on::<core::pin::Pin<&mut dyn core::future::future::Future<Output = core::result::Result<(), anyhow::Error>>>>
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/context/blocking.rs:66:14
    12: <tokio::runtime::scheduler::multi_thread::MultiThread>::block_on::<core::pin::Pin<&mut dyn core::future::future::Future<Output = core::result::Result<(), anyhow::Error>>>>::{closure#0}
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/multi_thread/mod.rs:92:22
    13: tokio::runtime::context::runtime::enter_runtime::<<tokio::runtime::scheduler::multi_thread::MultiThread>::block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output = core::result::Result<(), anyhow::Error>>>>::{closure#0}, core::result::Result<(), anyhow::Error>>
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/context/runtime.rs:65:16
    14: <tokio::runtime::scheduler::multi_thread::MultiThread>::block_on::<core::pin::Pin<&mut dyn core::future::future::Future<Output = core::result::Result<(), anyhow::Error>>>>
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/scheduler/multi_thread/mod.rs:91:9
    15: <tokio::runtime::runtime::Runtime>::block_on_inner::<core::pin::Pin<&mut dyn core::future::future::Future<Output = core::result::Result<(), anyhow::Error>>>>
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/runtime.rs:373:50
    16: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=core::result::Result<(), anyhow::Error>>>>
               at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.52.1/src/runtime/runtime.rs:345:18
    17: test_update_status
               at ./tests/integration_tests/updates.rs:833:7
    18: test_all::integration_tests::updates::test_update_status::{closure#0}
               at ./tests/integration_tests/updates.rs:737:34
    19: <test_all::integration_tests::updates::test_update_status::{closure#0} as core::ops::function::FnOnce<()>>::call_once
               at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/ops/function.rs:250:5
    20: core::ops::function::FnOnce::call_once
               at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/ops/function.rs:250:5
  note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
  WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
  WARN: temporary directory leaked: "/var/tmp/omicron_tmp/.tmpp7F64A"
  	If you would like to access the database for debugging, run the following:

  	# Run the database
  	cargo xtask db-dev run --no-populate --store-dir "/var/tmp/omicron_tmp/.tmpp7F64A/data"
  	# Access the database. Note the port may change if you run multiple databases.
  	cockroach sql --host=localhost:32221 --insecure
  WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 22466) and a temporary directory leaked, /var/tmp/omicron_tmp/test_all-9c1cbd77b0b87f6b-test_update_status.22413.1-clickhouse-P5Fi8u)
  failed to clean up ClickHouse data dir:
  - /var/tmp/omicron_tmp/test_all-9c1cbd77b0b87f6b-test_update_status.22413.1-clickhouse-P5Fi8u: File exists (os error 17)
  WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
  WARN: dendrite temporary directory leaked: /var/tmp/omicron_tmp/.tmpUugoPi
  WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
  WARN: mgd temporary directory leaked: /var/tmp/omicron_tmp/.tmp4rb2Uz

This one is kind of strange. Looking at the logs from the failing test itself, it seems like there are errors connecting to the database. The first one is here:

2026-06-03T20:13:59.002Z	WARN	test_update_status (ServerContext): Failed is_valid check; connection failed ping
    error = no connection to the server\n

Looking at the CRDB log, things definitely seem unhappy, but I'm not familiar with what normal behavior looks like. I do see that the server says it was terminated, so this could have been interrupted somehow. I definitely canceled an earlier CI run, but maybe that didn't do what I thought it did.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Test FlakeTests that work. Wait, no. Actually yes. Hang on. Something is broken.

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions