Skip to content

test failed in CI: reconfigurator-cli::test-scripts script::cmds-target-release.txt #8956

@bnaecker

Description

@bnaecker

This test failed on a CI run on #8951:

https://github.com/oxidecomputer/omicron/pull/8951/checks?check_run_id=49200214425

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01K3VA1T7NVFJ799R1A6G5TSYM/NI7ZQiSFd6NJxvzcIDDvpCqPMwqIipf7zqKNmfbVtU1SQcSw/01K3VA2XDKZ6PFSVGA99V3A5CV#S8962

Excerpt from the log showing the failure:

-c953-4807-9e84-9e750a79f68e based on parent blueprint 7f6b7297-c2bc-4f67-b3c0-c8e555ebbdc4
9342	2025-08-29T17:25:20.538Z	     planning report for blueprint 59630e63-c953-4807-9e84-9e750a79f68e:
9343	2025-08-29T17:25:20.538Z	     chicken switches:
9344	2025-08-29T17:25:20.538Z	    @@ -5883,15 +5834,12 @@
9345	2025-08-29T17:25:20.538Z	     INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: 2b8f0cb3-0295-4b3c-bc58-4fe88b57112c, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
9346	2025-08-29T17:25:20.538Z	     INFO performed noop zone image source checks on sled, sled_id: 98e6b7c2-2efa-41ca-b20a-0a4d61102fe6, num_total: 8, num_already_artifact: 7, num_eligible: 0, num_ineligible: 1
9347	2025-08-29T17:25:20.538Z	     INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: 98e6b7c2-2efa-41ca-b20a-0a4d61102fe6, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
9348	2025-08-29T17:25:20.538Z	     INFO performed noop zone image source checks on sled, sled_id: d81c6a84-79b8-4958-ae41-ea46c9b19763, num_total: 8, num_already_artifact: 4, num_eligible: 0, num_ineligible: 4
9349	2025-08-29T17:25:20.538Z	     INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: d81c6a84-79b8-4958-ae41-ea46c9b19763, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
9350	2025-08-29T17:25:20.538Z	    -WARN cannot configure host OS update for board (active phase 1 slot doesn't match boot disk; is the sled already being updated?), serial_number: serial0, part_number: model0, active_phase_1_slot: B, boot_disk: A
9351	2025-08-29T17:25:20.538Z	     INFO skipping board for MGS-driven update, serial_number: serial0, part_number: model0
9352	2025-08-29T17:25:20.538Z	    -WARN cannot configure host OS update for board (active phase 1 slot doesn't match boot disk; is the sled already being updated?), serial_number: serial1, part_number: model1, active_phase_1_slot: B, boot_disk: A
9353	2025-08-29T17:25:20.538Z	     INFO skipping board for MGS-driven update, serial_number: serial1, part_number: model1
9354	2025-08-29T17:25:20.538Z	    -WARN cannot configure host OS update for board (active phase 1 slot doesn't match boot disk; is the sled already being updated?), serial_number: serial2, part_number: model2, active_phase_1_slot: B, boot_disk: A
9355	2025-08-29T17:25:20.538Z	     INFO skipping board for MGS-driven update, serial_number: serial2, part_number: model2
9356	2025-08-29T17:25:20.539Z	     INFO ran out of boards for MGS-driven update
9357	2025-08-29T17:25:20.539Z	     generated blueprint e93650dc-b5ba-4ec7-8550-9171c1ada194 based on parent blueprint 59630e63-c953-4807-9e84-9e750a79f68e
9358	2025-08-29T17:25:20.539Z	     planning report for blueprint e93650dc-b5ba-4ec7-8550-9171c1ada194:
9359	2025-08-29T17:25:20.539Z	     chicken switches:
9360	2025-08-29T17:25:20.539Z	    @@ -6017,15 +5965,12 @@
9361	2025-08-29T17:25:20.539Z	     INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: 2b8f0cb3-0295-4b3c-bc58-4fe88b57112c, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
9362	2025-08-29T17:25:20.539Z	     INFO performed noop zone image source checks on sled, sled_id: 98e6b7c2-2efa-41ca-b20a-0a4d61102fe6, num_total: 8, num_already_artifact: 7, num_eligible: 0, num_ineligible: 1
9363	2025-08-29T17:25:20.539Z	     INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: 98e6b7c2-2efa-41ca-b20a-0a4d61102fe6, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
9364	2025-08-29T17:25:20.539Z	     INFO performed noop zone image source checks on sled, sled_id: d81c6a84-79b8-4958-ae41-ea46c9b19763, num_total: 7, num_already_artifact: 4, num_eligible: 0, num_ineligible: 3
9365	2025-08-29T17:25:20.539Z	     INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: d81c6a84-79b8-4958-ae41-ea46c9b19763, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
9366	2025-08-29T17:25:20.539Z	    -WARN cannot configure host OS update for board (active phase 1 slot doesn't match boot disk; is the sled already being updated?), serial_number: serial0, part_number: model0, active_phase_1_slot: B, boot_disk: A
9367	2025-08-29T17:25:20.539Z	     INFO skipping board for MGS-driven update, serial_number: serial0, part_number: model0
9368	2025-08-29T17:25:20.539Z	    -WARN cannot configure host OS update for board (active phase 1 slot doesn't match boot disk; is the sled already being updated?), serial_number: serial1, part_number: model1, active_phase_1_slot: B, boot_disk: A
9369	2025-08-29T17:25:20.539Z	     INFO skipping board for MGS-driven update, serial_number: serial1, part_number: model1
9370	2025-08-29T17:25:20.539Z	    -WARN cannot configure host OS update for board (active phase 1 slot doesn't match boot disk; is the sled already being updated?), serial_number: serial2, part_number: model2, active_phase_1_slot: B, boot_disk: A
9371	2025-08-29T17:25:20.539Z	     INFO skipping board for MGS-driven update, serial_number: serial2, part_number: model2
9372	2025-08-29T17:25:20.539Z	     INFO ran out of boards for MGS-driven update
9373	2025-08-29T17:25:20.539Z	     INFO some zones not yet up-to-date, sled_id: d81c6a84-79b8-4958-ae41-ea46c9b19763, zones_currently_updating: [ZoneCurrentlyUpdating { zone_id: d5fd048a-8786-42d3-938e-820eae95d7f4 (service), zone_kind: InternalNtp, reason: MissingInInventory { bp_image_source: Artifact { version: Available { version: ArtifactVersion("1.0.0") }, hash: ArtifactHash("67593d686ed04a1709f93972b71f4ebc148a9362120f65d239943e814a9a7439") } } }]
9374	2025-08-29T17:25:20.539Z	     generated blueprint 90650737-8142-47a6-9a48-a10efc487e57 based on parent blueprint e93650dc-b5ba-4ec7-8550-9171c1ada194
9375	2025-08-29T17:25:20.539Z	     planning report for blueprint 90650737-8142-47a6-9a48-a10efc487e57:
9376	2025-08-29T17:25:20.539Z	    @@ -6153,15 +6098,12 @@
9377	2025-08-29T17:25:20.539Z	     INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: 2b8f0cb3-0295-4b3c-bc58-4fe88b57112c, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
9378	2025-08-29T17:25:20.539Z	     INFO performed noop zone image source checks on sled, sled_id: 98e6b7c2-2efa-41ca-b20a-0a4d61102fe6, num_total: 8, num_already_artifact: 7, num_eligible: 0, num_ineligible: 1
9379	2025-08-29T17:25:20.539Z	     INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: 98e6b7c2-2efa-41ca-b20a-0a4d61102fe6, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
9380	2025-08-29T17:25:20.539Z	     INFO performed noop zone image source checks on sled, sled_id: d81c6a84-79b8-4958-ae41-ea46c9b19763, num_total: 8, num_already_artifact: 5, num_eligible: 0, num_ineligible: 3
9381	2025-08-29T17:25:20.539Z	     INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: d81c6a84-79b8-4958-ae41-ea46c9b19763, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
9382	2025-08-29T17:25:20.539Z	    -WARN cannot configure host OS update for board (active phase 1 slot doesn't match boot disk; is the sled already being updated?), serial_number: serial0, part_number: model0, active_phase_1_slot: B, boot_disk: A
9383	2025-08-29T17:25:20.539Z	     INFO skipping board for MGS-driven update, serial_number: serial0, part_number: model0
9384	2025-08-29T17:25:20.539Z	    -WARN cannot configure host OS update for board (active phase 1 slot doesn't match boot disk; is the sled already being updated?), serial_number: serial1, part_number: model1, active_phase_1_slot: B, boot_disk: A
9385	2025-08-29T17:25:20.539Z	     INFO skipping board for MGS-driven update, serial_number: serial1, part_number: model1
9386	2025-08-29T17:25:20.539Z	    -WARN cannot configure host OS update for board (active phase 1 slot doesn't match boot disk; is the sled already being updated?), serial_number: serial2, part_number: model2, active_phase_1_slot: B, boot_disk: A
9387	2025-08-29T17:25:20.539Z	     INFO skipping board for MGS-driven update, serial_number: serial2, part_number: model2
9388	2025-08-29T17:25:20.539Z	     INFO ran out of boards for MGS-driven update
9389	2025-08-29T17:25:20.539Z	     generated blueprint 2182613d-dc9f-41eb-9c6a-d33801849caa based on parent blueprint 90650737-8142-47a6-9a48-a10efc487e57
9390	2025-08-29T17:25:20.539Z	     planning report for blueprint 2182613d-dc9f-41eb-9c6a-d33801849caa:
9391	2025-08-29T17:25:20.539Z	     chicken switches:
9392	2025-08-29T17:25:20.539Z	    @@ -6280,15 +6222,12 @@
9393	2025-08-29T17:25:20.539Z	     INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: 2b8f0cb3-0295-4b3c-bc58-4fe88b57112c, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
9394	2025-08-29T17:25:20.540Z	     INFO performed noop zone image source checks on sled, sled_id: 98e6b7c2-2efa-41ca-b20a-0a4d61102fe6, num_total: 8, num_already_artifact: 7, num_eligible: 0, num_ineligible: 1
9395	2025-08-29T17:25:20.540Z	     INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: 98e6b7c2-2efa-41ca-b20a-0a4d61102fe6, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
9396	2025-08-29T17:25:20.540Z	     INFO performed noop zone image source checks on sled, sled_id: d81c6a84-79b8-4958-ae41-ea46c9b19763, num_total: 8, num_already_artifact: 6, num_eligible: 0, num_ineligible: 2
9397	2025-08-29T17:25:20.540Z	     INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: d81c6a84-79b8-4958-ae41-ea46c9b19763, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
9398	2025-08-29T17:25:20.540Z	    -WARN cannot configure host OS update for board (active phase 1 slot doesn't match boot disk; is the sled already being updated?), serial_number: serial0, part_number: model0, active_phase_1_slot: B, boot_disk: A
9399	2025-08-29T17:25:20.540Z	     INFO skipping board for MGS-driven update, serial_number: serial0, part_number: model0
9400	2025-08-29T17:25:20.540Z	    -WARN cannot configure host OS update for board (active phase 1 slot doesn't match boot disk; is the sled already being updated?), serial_number: serial1, part_number: model1, active_phase_1_slot: B, boot_disk: A
9401	2025-08-29T17:25:20.540Z	     INFO skipping board for MGS-driven update, serial_number: serial1, part_number: model1
9402	2025-08-29T17:25:20.540Z	    -WARN cannot configure host OS update for board (active phase 1 slot doesn't match boot disk; is the sled already being updated?), serial_number: serial2, part_number: model2, active_phase_1_slot: B, boot_disk: A
9403	2025-08-29T17:25:20.540Z	     INFO skipping board for MGS-driven update, serial_number: serial2, part_number: model2
9404	2025-08-29T17:25:20.540Z	     INFO ran out of boards for MGS-driven update
9405	2025-08-29T17:25:20.540Z	     generated blueprint e8b088a8-7da0-480b-a2dc-75ffef068ece based on parent blueprint 2182613d-dc9f-41eb-9c6a-d33801849caa
9406	2025-08-29T17:25:20.541Z	     planning report for blueprint e8b088a8-7da0-480b-a2dc-75ffef068ece:
9407	2025-08-29T17:25:20.541Z	     chicken switches:
9408	2025-08-29T17:25:20.541Z	    @@ -6425,15 +6364,12 @@
9409	2025-08-29T17:25:20.541Z	     INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: 2b8f0cb3-0295-4b3c-bc58-4fe88b57112c, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
9410	2025-08-29T17:25:20.541Z	     INFO performed noop zone image source checks on sled, sled_id: 98e6b7c2-2efa-41ca-b20a-0a4d61102fe6, num_total: 8, num_already_artifact: 7, num_eligible: 0, num_ineligible: 1
9411	2025-08-29T17:25:20.541Z	     INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: 98e6b7c2-2efa-41ca-b20a-0a4d61102fe6, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
9412	2025-08-29T17:25:20.541Z	     INFO performed noop zone image source checks on sled, sled_id: d81c6a84-79b8-4958-ae41-ea46c9b19763, num_total: 7, num_already_artifact: 6, num_eligible: 0, num_ineligible: 1
9413	2025-08-29T17:25:20.541Z	     INFO BootPartitionDetails inventory hash not found in TUF repo, ignoring for noop checks, sled_id: d81c6a84-79b8-4958-ae41-ea46c9b19763, slot: a, expected_hash: 0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a
9414	2025-08-29T17:25:20.541Z	    -WARN cannot configure host OS update for board (active phase 1 slot doesn't match boot disk; is the sled already being updated?), serial_number: serial0, part_number: model0, active_phase_1_slot: B, boot_disk: A
9415	2025-08-29T17:25:20.541Z	     INFO skipping board for MGS-driven update, serial_number: serial0, part_number: model0
9416	2025-08-29T17:25:20.545Z	    -WARN cannot configure host OS update for board (active phase 1 slot doesn't match boot disk; is the sled already being updated?), serial_number: serial1, part_number: model1, active_phase_1_slot: B, boot_disk: A
9417	2025-08-29T17:25:20.545Z	     INFO skipping board for MGS-driven update, serial_number: serial1, part_number: model1
9418	2025-08-29T17:25:20.545Z	    -WARN cannot configure host OS update for board (active phase 1 slot doesn't match boot disk; is the sled already being updated?), serial_number: serial2, part_number: model2, active_phase_1_slot: B, boot_disk: A
9419	2025-08-29T17:25:20.545Z	     INFO skipping board for MGS-driven update, serial_number: serial2, part_number: model2
9420	2025-08-29T17:25:20.545Z	     INFO ran out of boards for MGS-driven update
9421	2025-08-29T17:25:20.545Z	     generated blueprint 810ea95a-4730-43dd-867e-1984aeb9d873 based on parent blueprint e8b088a8-7da0-480b-a2dc-75ffef068ece
9422	2025-08-29T17:25:20.545Z	     planning report for blueprint 810ea95a-4730-43dd-867e-1984aeb9d873:
9423	2025-08-29T17:25:20.545Z	     chicken switches:
9424	2025-08-29T17:25:20.545Z	
9425	2025-08-29T17:25:20.545Z	    test script::cmds-target-release.txt ... FAILED
9426	2025-08-29T17:25:20.546Z	
9427	2025-08-29T17:25:20.546Z	    failures:
9428	2025-08-29T17:25:20.546Z	
9429	2025-08-29T17:25:20.546Z	    ---- script::cmds-target-release.txt ----
9430	2025-08-29T17:25:20.546Z	    test panicked: assertion failed: string doesn't match the contents of file: "tests/output/cmds-target-release-stdout" see diffset above
9431	2025-08-29T17:25:20.546Z	                    set EXPECTORATE=overwrite if these changes are intentional
9432	2025-08-29T17:25:20.546Z	
9433	2025-08-29T17:25:20.546Z	
9434	2025-08-29T17:25:20.546Z	    failures:
9435	2025-08-29T17:25:20.546Z	        script::cmds-target-release.txt
9436	2025-08-29T17:25:20.546Z	
9437	2025-08-29T17:25:20.546Z	    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.38s

This change was isolated to the IP Pool APIs in Omicron, so I'm pretty sure this is an unrelated flake.

Metadata

Metadata

Assignees

Labels

Test FlakeTests that work. Wait, no. Actually yes. Hang on. Something is broken.

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions