Skip to content

test failed in CI: oximeter_collector::agent::tests::test_self_stat_error_counter #7255

@bnaecker

Description

@bnaecker

This test failed on a CI run on #7253 :

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01JF12TXM9VAR639ZT9NC0EV0M/8kts96alUpAV5HlmPMQHUvs5ttxnTkXtPqkdb32jXJVcZ6ME/01JF12WXHS2JX5704Y2Q5074MP#S6919

Excerpt from the log showing the failure:

2024-12-13T23:53:22.038Z	--- STDERR:              oximeter-collector agent::tests::test_self_stat_error_counter ---
2024-12-13T23:53:22.039Z	log file: /var/tmp/omicron_tmp/oximeter_collector-5094f8def4546e47-test_self_stat_error_counter.232625.0.log
2024-12-13T23:53:22.039Z	note: configured to log to "/var/tmp/omicron_tmp/oximeter_collector-5094f8def4546e47-test_self_stat_error_counter.232625.0.log"
2024-12-13T23:53:22.039Z	thread 'agent::tests::test_self_stat_error_counter' panicked at oximeter/collector/src/agent.rs:895:9:
2024-12-13T23:53:22.039Z	assertion `left == right` failed: number of collections reported by the collection task differs from the number reported by the always-ded producer server itself
2024-12-13T23:53:22.039Z	  left: 4
2024-12-13T23:53:22.039Z	 right: 5
2024-12-13T23:53:22.039Z	stack backtrace:
2024-12-13T23:53:22.039Z	   0: rust_begin_unwind
2024-12-13T23:53:22.039Z	             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:665:5
2024-12-13T23:53:22.039Z	   1: core::panicking::panic_fmt
2024-12-13T23:53:22.039Z	             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panicking.rs:74:14
2024-12-13T23:53:22.039Z	   2: core::panicking::assert_failed_inner
2024-12-13T23:53:22.039Z	             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panicking.rs:405:23
2024-12-13T23:53:22.039Z	   3: core::panicking::assert_failed
2024-12-13T23:53:22.039Z	             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panicking.rs:365:5
2024-12-13T23:53:22.039Z	   4: {async_block#0}
2024-12-13T23:53:22.039Z	             at ./src/agent.rs:895:9
2024-12-13T23:53:22.039Z	   5: poll<&mut dyn core::future::future::Future<Output=()>>
2024-12-13T23:53:22.039Z	             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/future/future.rs:123:9
2024-12-13T23:53:22.040Z	   6: poll<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
2024-12-13T23:53:22.040Z	             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/future/future.rs:123:9
2024-12-13T23:53:22.040Z	   7: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
2024-12-13T23:53:22.040Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:696:57
2024-12-13T23:53:22.040Z	   8: with_budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
2024-12-13T23:53:22.040Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:107:5
2024-12-13T23:53:22.040Z	   9: budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
2024-12-13T23:53:22.040Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/coop.rs:73:5
2024-12-13T23:53:22.040Z	  10: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
2024-12-13T23:53:22.040Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:696:25
2024-12-13T23:53:22.040Z	  11: tokio::runtime::scheduler::current_thread::Context::enter
2024-12-13T23:53:22.040Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:423:19
2024-12-13T23:53:22.040Z	  12: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
2024-12-13T23:53:22.040Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:695:36
2024-12-13T23:53:22.040Z	  13: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
2024-12-13T23:53:22.040Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:774:68
2024-12-13T23:53:22.040Z	  14: tokio::runtime::context::scoped::Scoped<T>::set
2024-12-13T23:53:22.040Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/scoped.rs:40:9
2024-12-13T23:53:22.040Z	  15: tokio::runtime::context::set_scheduler::{{closure}}
2024-12-13T23:53:22.040Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context.rs:180:26
2024-12-13T23:53:22.040Z	  16: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#8}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
2024-12-13T23:53:22.040Z	             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/local.rs:283:12
2024-12-13T23:53:22.040Z	  17: std::thread::local::LocalKey<T>::with
2024-12-13T23:53:22.040Z	             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/thread/local.rs:260:9
2024-12-13T23:53:22.041Z	  18: tokio::runtime::context::set_scheduler
2024-12-13T23:53:22.041Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context.rs:180:9
2024-12-13T23:53:22.041Z	  19: tokio::runtime::scheduler::current_thread::CoreGuard::enter
2024-12-13T23:53:22.041Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:774:27
2024-12-13T23:53:22.041Z	  20: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
2024-12-13T23:53:22.041Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:683:19
2024-12-13T23:53:22.041Z	  21: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
2024-12-13T23:53:22.041Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:191:28
2024-12-13T23:53:22.041Z	  22: tokio::runtime::context::runtime::enter_runtime
2024-12-13T23:53:22.041Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16
2024-12-13T23:53:22.041Z	  23: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
2024-12-13T23:53:22.041Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/scheduler/current_thread/mod.rs:179:9
2024-12-13T23:53:22.041Z	  24: tokio::runtime::runtime::Runtime::block_on_inner
2024-12-13T23:53:22.041Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs:361:47
2024-12-13T23:53:22.042Z	  25: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
2024-12-13T23:53:22.042Z	             at /home/build/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.40.0/src/runtime/runtime.rs:335:13
2024-12-13T23:53:22.042Z	  26: test_self_stat_error_counter
2024-12-13T23:53:22.042Z	             at ./src/agent.rs:903:9
2024-12-13T23:53:22.042Z	  27: oximeter_collector::agent::tests::test_self_stat_error_counter::{{closure}}
2024-12-13T23:53:22.042Z	             at ./src/agent.rs:825:44
2024-12-13T23:53:22.042Z	  28: core::ops::function::FnOnce::call_once
2024-12-13T23:53:22.042Z	             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/ops/function.rs:250:5
2024-12-13T23:53:22.042Z	  29: core::ops::function::FnOnce::call_once
2024-12-13T23:53:22.042Z	             at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/ops/function.rs:250:5
2024-12-13T23:53:22.042Z	note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Metadata

Metadata

Assignees

Labels

Test FlakeTests that work. Wait, no. Actually yes. Hang on. Something is broken.

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions