Ceph QA

rocky 10 runs:
https://pulpito.ceph.com/yaarit-2026-01-23_22:08:51-rados-wip-rocky10-branch-of-the-day-2026-01-23-1769128778-distro-default-trial/

scheduled with:

$ ./virtualenv/bin/teuthology-suite -s rados --priority 11 -m trial --suite-repo https://git.ceph.com/ceph-ci.git --ceph-repo https://git.ceph.com/ceph-ci.git -c wip-rocky10-branch-of-the-day-2026-01-23-1769128778 --force-priority --newest 10 --filter rocky --subset 111/120000 --suite-branch rocky101 $(pwd)/cephadm-rocky10.yaml 

Please note that teuthology.git needs to be up to date and bootstrap needs to be rerun before running the command above.

new tracker that opened - all of them are infra and not related to any changes that wip-rocky10-branch-of-the-day-2026-01-23-1769128778 includes.

15696 - new tracker - Mixed image Ubuntu and Rocky10 https://tracker.ceph.com/issues/74552
[15699, 15701, 15702, 15705, 15706, ] https://tracker.ceph.com/issues/74553 - Rocky10 test failed with SELinuxError: SELinux denials found on ubuntu@trial124.front.sepia.ceph.com

Unrelated issues:
[15697, 15698, 15700, 15703, 15704, 15707, 15709, 15711] - https://tracker.ceph.com/issues/71816 - Failed to write to /dev/nvme-fabrics: Invalid argument

Adding more Rocky tests with command:
teuthology-suite -s rados --priority 11 -m trial --suite-repo https://git.ceph.com/ceph-ci.git --ceph-repo https://git.ceph.com/ceph-ci.git -c wip-rocky10-branch-of-the-day-2026-01-23-1769128778 --force-priority --newest 10 --filter "rpm_latest,rocky_10" --subset 111/120000 --suite-branch rocky101 $(pwd)/cephadm-rocky10.yaml

rpm_latest was missing from the previous run.

analyze of https://pulpito.ceph.com/nmordech-2026-01-25_11:10:14-rados-wip-rocky10-branch-of-the-day-2026-01-23-1769128778-distro-default-trial/

['17059'] - new issue: https://tracker.ceph.com/issues/74564 - prometheus is not available (could be part of infra issues)
['17041', '17085', '17023', '17086', '17005'] - https://tracker.ceph.com/issues/73822 - Rocky10 - rados/verify - valgrind error: MismatchedFree operator delete[](void*, unsigned long, std::align_val_t) RocksDBStore::close() RocksDBStore::~RocksDBStore()
['16960', '16988', '17014'] - https://tracker.ceph.com/issues/74553 - Rocky10 test failed with SELinuxError: SELinux denials found on ubuntu@trial124.front.sepia.ceph.com
['16960', '16988', '17014'] - https://tracker.ceph.com/issues/74553 - Rocky10 test failed with SELinuxError: SELinux denials found on ubuntu@trial124.front.sepia.ceph.com
['17039', '17037'] - https://tracker.ceph.com/issues/74565 - reimage timeout
['17038', '17052'] - Mixed image Ubuntu and Rocky10 https://tracker.ceph.com/issues/74552
['17076'] - https://tracker.ceph.com/issues/74563 - ansible error: 'Failed to download packages: No URLs in mirrorlist'
16974 - https://tracker.ceph.com/issues/74568 - g++ is missing

Unrelated issues:
['16964', '16985', '17048', '17081', '16995', '17077', '17024', '17029'] - https://tracker.ceph.com/issues/71816 - Failed to write to /dev/nvme-fabrics: Invalid argument
[17026] - https://tracker.ceph.com/issues/73630 - objectstore/test_fuse.sh: line 126: fusermount: command not found
17079 - https://tracker.ceph.com/issues/71506 - Unhandled exception from module 'dashboard' while running on mgr.y: Timeout('Port 8443 not free on ::.')
['16984', '17075'] - https://tracker.ceph.com/issues/74149 - Prometheus module fails when trying to load security configuration JSON
['17012'] - https://tracker.ceph.com/issues/74332 - ceph-kvstore-tool segfault in workunits/cephtool/test_kvstore_tool.sh
['17018', '17053'] https://tracker.ceph.com/issues/72945 - Data digests are inconsistent during scrubbing

analyze of: https://pulpito.ceph.com/lflores-2026-01-23_19:07:45-rados-wip-rocky10-branch-of-the-day-2026-01-23-1769128778-distro-default-trial/

['15353', '15333', '15307'] - https://tracker.ceph.com/issues/74565 - reimage timeout

Unrelated issues:
['15260', '15313', '15348', '15359', '15406', '15345', '15271', '15277', '15389', '15299', '15371', '15294'] - https://tracker.ceph.com/issues/71506 - Unhandled exception from module 'dashboard' while running on mgr.y: Timeout('Port 8443 not free on ::.')
['15318', '15309', '15402', '15399'] - https://tracker.ceph.com/issues/74332 - ceph-kvstore-tool segfault in workunits/cephtool/test_kvstore_tool.sh
['15295', '15385'] -
['15308', '15382', '15281', '15357', '15330', '15257', '15410'] https://tracker.ceph.com/issues/68337 - OSD experiencing slow operations in BlueStore (BLUESTORE_SLOW_OP_ALERT) in cluster log
['15369', '15315', '15324', '15267', '15354', '15320', '15415', '15403'] - https://tracker.ceph.com/issues/72945 - Data digests are inconsistent during scrubbing
['15419'] - https://tracker.ceph.com/issues/65719 - debian-17.2.6 jammy repository does not have a Release file
['15326', '15304', '15291', '15250', '15370', '15381'] - https://tracker.ceph.com/issues/74518 - rados/perf: pools reached nearfull (POOL_NEARFULL)
['15280'] - https://tracker.ceph.com/issues/59335 - Found coredumps on smithi related to sqlite3 - (cephsqlite)
['15253', '15340'] - https://tracker.ceph.com/issues/68668 - rados/dashboard: mkfs.xfs: cannot open /dev/vg_nvme/lv_2: Device or resource busy
['15247'] - https://tracker.ceph.com/issues/74524 - TEST_backfill_pool_priority: "The primary PG X.X didn't become the in progress item on remote"
['15368'] - https://tracker.ceph.com/issues/20344 - make check fails with Error EIO: load dlopen(build/lib/libec_FAKE.so): build/lib/libec_FAKE.so: cannot open shared object file: No such file or directory
['15276'] - https://tracker.ceph.com/issues/68833 - TEST_just_deep_scrubs fails from "key is query_last_duration: negation:1 # expected: 0 # in actual: 0"
['15317', '15383'] - https://tracker.ceph.com/issues/71005 - qa/tasks/backfill_toofull.py line 155 - assert backfillfull < 0.9 - for test with compression

analyze of https://pulpito.ceph.com/nmordech-2026-01-26_18:32:32-rados:cephadm-wip-rocky10-branch-of-the-day-2026-01-23-1769128778-distro-default-trial/

New issue found - but not releated to the PRs:
https://tracker.ceph.com/issues/74577 - test_iscsi_setup.sh - No such path /iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/hosts/iqn.1994-05.com.redhat:client1

Rocky10 issues still occur:
['18789', '18791', '18792', '18796'] - https://tracker.ceph.com/issues/73823 - orch/cephadm: nvme-loop task fails on rocky 10

Unrelated issues:
['18793'] - https://tracker.ceph.com/issues/74523 - cephadm: task/test_set_mon_crush_locations fails because mon.a's crush location is not in datacenter a
['18795'] - https://tracker.ceph.com/issues/67555 - failed to fetch repository task/test_cephadm_repos
['18794'] - https://tracker.ceph.com/issues/74519 - cephadm/osds: Error: statfs /etc/ceph/ceph.client.admin.keyring: no such file or directory

Analyze: https://pulpito.ceph.com/nmordech-2026-01-26_18:28:29-rados-wip-rocky10-branch-of-the-day-2026-01-23-1769128778-distro-default-trial/

Unrelated issues:
['18755', '18779'] - https://tracker.ceph.com/issues/71506 - Unhandled exception from module 'dashboard' while running on mgr.y: Timeout('Port 8443 not free on ::.')
['18702', '18786', '18717', '18785', '18742'] - https://tracker.ceph.com/issues/73822 - Rocky10 - rados/verify - valgrind error: MismatchedFree operator delete[](void*, unsigned long, std::align_val_t) RocksDBStore::close() RocksDBStore::~RocksDBStore()
['18708', '18773'] - https://tracker.ceph.com/issues/74332 - ceph-kvstore-tool segfault in workunits/cephtool/test_kvstore_tool.sh
['18664', '18778', '18747', '18685', '18726', '18780', '18718'] - https://tracker.ceph.com/issues/71816 - Failed to write to /dev/nvme-fabrics: Invalid argument
['18736', '18781', '18748', '18681', '18712'] - https://tracker.ceph.com/issues/72945 - Data digests are inconsistent during scrubbing
['18689', '18709', '18730'] - https://tracker.ceph.com/issues/74553 - Rocky10 test failed with SELinuxError: SELinux denials found on ubuntu@trial124.front.sepia.ceph.com
['18723'] - https://tracker.ceph.com/issues/73630 - objectstore/test_fuse.sh: line 126: fusermount: command not found
['18775'] - https://tracker.ceph.com/issues/74517 - drop.ceph.com is unreachable
['18684'] - https://tracker.ceph.com/issues/74149 - Prometheus module fails when trying to load security configuration JSON
['18662'] - https://tracker.ceph.com/issues/74524 - TEST_backfill_pool_priority: "The primary PG X.X didn't become the in progress item on remote"
['18766'] - https://tracker.ceph.com/issues/74519 - cephadm/osds: Error: statfs /etc/ceph/ceph.client.admin.keyring: no such file or directory

I mostly agree with Nitzan's review, although I added a few more issues that I think are related:

Issues related to the Rocky10 Changes:
1. https://tracker.ceph.com/issues/74543 - AttributeError in dashboard module - (mgr)
2. https://tracker.ceph.com/issues/74564 - Rocky10 - prometheus not active - (mgr)
3. https://tracker.ceph.com/issues/73823 - orch/cephadm: nvme-loop task fails on rocky 10 - (Orchestrator)
4. https://tracker.ceph.com/issues/74604 - Rocky10 - MismatchedFree delete coming from ceph-osd-classic code - (RADOS)
5. https://tracker.ceph.com/issues/74553 - Rocky10 test failed with SELinuxError: SELinux denials found on ubuntu@trial124.front.sepia.ceph.com - (Ceph)
6. https://tracker.ceph.com/issues/74605 - Rocky10 - ERROR: test_sql_autocommit1 (tasks.mgr.test_devicehealth.TestDeviceHealth.test_sql_autocommit1) - (mgr)
7. https://tracker.ceph.com/issues/74606 - Rocky10 - Failed to download metadata for repo ''baseos'': Cannot prepare internal mirrorlist - (Infrastructure)
8. https://tracker.ceph.com/issues/74568 - Rocky10 - g++ missing - (RADOS)
9. https://tracker.ceph.com/issues/74608 - Rocky10 - teuthology fails to download a rocky10 package via mirrorlist - (teuthology)
10. https://tracker.ceph.com/issues/74609 - Rocky10 - rados/upgrade: failure to install reef rocky10 packages - (RADOS)

Non-rocky related:
1. https://tracker.ceph.com/issues/71506 - Unhandled exception from module 'dashboard' while running on mgr.y: Timeout('Port 8443 not free on ::.') - (Orchestrator)
2. https://tracker.ceph.com/issues/66603 - rados/cephadm/smoke: CEPHADM_AGENT_DOWN: 2 Cephadm Agent(s) are not reporting. Hosts may be offline - (Orchestrator)
3. https://tracker.ceph.com/issues/74332 - ceph-kvstore-tool segfault in workunits/cephtool/test_kvstore_tool.sh - (RADOS)
4. https://tracker.ceph.com/issues/70669 - ERROR: test_list_enabled_module: cephfs resource temporarily unavailable - (Dashboard)
5. https://tracker.ceph.com/issues/74501 - "IOPS is not within the threshold limit" errors - (RADOS)
6. https://tracker.ceph.com/issues/72945 - Data digests are inconsistent during scrubbing - (RADOS)
7. https://tracker.ceph.com/issues/74178 - qa/standalone/mon/availability.sh: line 64: Syntax error due to missing availability status - (Ceph)
8. https://tracker.ceph.com/issues/74519 - cephadm/osds: Error: statfs /etc/ceph/ceph.client.admin.keyring: no such file or directory - (Orchestrator)
9. https://tracker.ceph.com/issues/65719 - debian-17.2.6 jammy repository does not have a Release file - (Orchestrator)
10. https://tracker.ceph.com/issues/74523 - cephadm: task/test_set_mon_crush_locations fails because mon.a's crush location is not in datacenter a - (Orchestrator)
11. https://tracker.ceph.com/issues/68668 - rados/dashboard: mkfs.xfs: cannot open /dev/vg_nvme/lv_2: Device or resource busy - (Orchestrator)
12. https://tracker.ceph.com/issues/59335 - Found coredumps on smithi related to sqlite3 - (cephsqlite)
13. https://tracker.ceph.com/issues/74524 - TEST_backfill_pool_priority: "The primary PG X.X didn't become the in progress item on remote" - (RADOS)
14. https://tracker.ceph.com/issues/74004 - qa/standalone/ceph-helpers: ceph pg query hangs indefinitely - (RADOS)
15. https://tracker.ceph.com/issues/74603 - qa/standalone/scrub/osd-scrub-test.sh: TEST_just_deep_scrubs scrub duration check fails - (RADOS)
16. https://tracker.ceph.com/issues/71005 - qa/tasks/backfill_toofull.py line 155 - assert backfillfull < 0.9 - for test with compression - (RADOS)
17. https://tracker.ceph.com/issues/73630 - objectstore/test_fuse.sh: line 126: fusermount: command not found - (CephFS)
18. https://tracker.ceph.com/issues/74149 - Prometheus module fails when trying to load security configuration JSON - (mgr)
19. https://tracker.ceph.com/issues/74517 - drop.ceph.com is unreachable - (Infrastructure)
20. https://tracker.ceph.com/issues/74607 - do_messenger_dump_basics_test: Error ENOENT: problem getting command descriptions from mds.a - (CephFS)
21. https://tracker.ceph.com/issues/72945 - Data digests are inconsistent during scrubbing - (RADOS)
22. https://tracker.ceph.com/issues/74525 - TEST_bluestore_expand fails on "free space added" check - (bluestore)

I have updated the ticket with the rocky10-related issues.

Sam + Nitzan suspected lab issues with https://tracker.ceph.com/issues/74543 (AttributeError in dashboard module), but I'm still not convinced this is not coming from the Python changes we introduced.

I added some analysis to the ticket.

analyze of: https://pulpito.ceph.com/nmordech-2026-01-28_16:21:31-rados:cephadm-wip-rocky10-branch-of-the-day-2026-01-23-1769128778-distro-default-trial/

1. https://tracker.ceph.com/issues/74641 - Port 7150 not free on .. <--new , lets related it to the latest changes like Laura suggested

Non-rocky related:
1. https://tracker.ceph.com/issues/65719 - debian-17.2.6 jammy repository does not have a Release file - (Orchestrator)
2. https://tracker.ceph.com/issues/74501 - "IOPS is not within the threshold limit" errors - (RADOS)
3. https://tracker.ceph.com/issues/71506 - Unhandled exception from module 'dashboard' while running on mgr.y: Timeout('Port 8443 not free on ::.') - (Orchestrator)
4. https://tracker.ceph.com/issues/71816 - Failed to write to /dev/nvme-fabrics: Invalid argument
5. https://tracker.ceph.com/issues/74577 - test_iscsi_setup.sh - No such path /iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/hosts/iqn.1994-05.com.redhat:client1
6. https://tracker.ceph.com/issues/74523 - cephadm: task/test_set_mon_crush_locations fails because mon.a's crush location is not in datacenter a - (Orchestrator)

analyze of: https://pulpito.ceph.com/nmordech-2026-01-28_16:11:20-rados-wip-rocky10-branch-of-the-day-2026-01-23-1769128778-distro-default-trial/

Rocky10:
1. https://tracker.ceph.com/issues/74553 - Rocky10 test failed with SELinuxError: SELinux denials found on ubuntu@trial124.front.sepia.ceph.com
2. https://tracker.ceph.com/issues/74565 - reimage timeout
3. https://tracker.ceph.com/issues/74643 - cherrypy.process.wspbus.ChannelFailures: TypeError('certfile should be a valid filesystem path') <-- new occur
4. https://tracker.ceph.com/issues/74605 - Rocky10 - ERROR: test_sql_autocommit1 (tasks.mgr.test_devicehealth.TestDeviceHealth.test_sql_autocommit1) - (mgr)

Non-rocky related:
1. https://tracker.ceph.com/issues/74501 - "IOPS is not within the threshold limit" errors - (RADOS)
2. https://tracker.ceph.com/issues/73630 - objectstore/test_fuse.sh: line 126: fusermount: command not found
3. https://tracker.ceph.com/issues/74607 - do_messenger_dump_basics_test: Error ENOENT: problem getting command descriptions from mds.a - (CephFS)
4. https://tracker.ceph.com/issues/74074 - ceph-dencoder - failed with 20.3 corpus repo (RADOS)
5. https://tracker.ceph.com/issues/73822 - Rocky10 - rados/verify - valgrind error: MismatchedFree operator delete[](void*, unsigned long, std::align_val_t) RocksDBStore::close() RocksDBStore::~RocksDBStore()
6. https://tracker.ceph.com/issues/73249 - osd/MissingLoc.h: FAILED ceph_assert(0 == "unexpected need for missing item")
7. https://tracker.ceph.com/issues/74332 - ceph-kvstore-tool segfault in workunits/cephtool/test_kvstore_tool.sh
8. https://tracker.ceph.com/issues/72945 - Data digests are inconsistent during scrubbing - (RADOS)
9. https://tracker.ceph.com/issues/71816 - Failed to write to /dev/nvme-fabrics: Invalid argument
10. https://tracker.ceph.com/issues/71506 - Unhandled exception from module 'dashboard' while running on mgr.y: Timeout('Port 8443 not free on ::.') - (Orchestrator)
11. https://tracker.ceph.com/issues/74577 - test_iscsi_setup.sh - No such path /iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/hosts/iqn.1994-05.com.redhat:client1
12. https://tracker.ceph.com/issues/71005 - qa/tasks/backfill_toofull.py line 155 - assert backfillfull < 0.9 - for test with compression - (RADOS)
13. https://tracker.ceph.com/issues/74525 - TEST_bluestore_expand fails on "free space added" check - (bluestore)
14. https://tracker.ceph.com/issues/74149 - Prometheus module fails when trying to load security configuration JSON - (mgr)
15. https://tracker.ceph.com/issues/64435 - Test fails from "timed out waiting; will kill: <Greenlet"...
16. https://tracker.ceph.com/issues/68668 - rados/dashboard: mkfs.xfs: cannot open /dev/vg_nvme/lv_2: Device or resource busy
17. https://tracker.ceph.com/issues/65719 - debian-17.2.6 jammy repository does not have a Release file - (Orchestrator)

@Nitzan Mordechai your latest review looks comprehensive.

I am currently trying to knock out https://tracker.ceph.com/issues/74609.

I scheduled some initial tests at https://tracker.ceph.com/issues/74609#note-4.

analyze https://pulpito.ceph.com/nmordech-2026-01-29_15:32:36-rados:mgr-wip-rocky10-branch-of-the-day-2026-01-23-1769128778-distro-default-trial/

Rocky10:
1. https://tracker.ceph.com/issues/74605 - Rocky10 - ERROR: test_sql_autocommit1 (tasks.mgr.test_devicehealth.TestDeviceHealth.test_sql_autocommit1) - (mgr)

Non-rocky related:
1. https://tracker.ceph.com/issues/69954 - test_selftest_command_spam: ConnectionRefusedError: [Errno 111] Connection refused
2. https://tracker.ceph.com/issues/74149 - Prometheus module fails when trying to load security configuration JSON