Ceph QA

The two failures in the test run are tracked here:

1. https://tracker.ceph.com/issues/62235 - Assert failure: test_ceph_osd_pool_create_utf8 - (RADOS)
2. https://tracker.ceph.com/issues/73750 - rados/basic: Segmentation fault during neorados tests - (RADOS)

1 is a known issue, but it hasn't been seen since Pacific, so we should probably analyze it further to see if it's somehow rocky 10 - specific.

2 is a new issue. At a glance, I don't see if/how it's obviously related to rocky 10, but some coredumps were generated, so I'll try to get some more debug info so we can tell for sure.

The first error actually occurred on a test that used ubuntu 22.04 packages, so that can't possibly be related to rocky 10. I think the second issue is the one to focus on, as it used rocky 10 packages.

The rocky test failure repeated 3x (see same pulpito link above).

I scheduled the same type of test on main so it runs on centos and ubuntu packages to verify whether the issue is exclusive to rocky:
http://pulpito.ceph.com/lflores-2025-11-10_16:32:42-rados:basic-main-distro-default-smithi/

Laura Flores wrote in #note-7:

The rocky test failure repeated 3x (see same pulpito link above).

I scheduled the same type of test on main so it runs on centos and ubuntu packages to verify whether the issue is exclusive to rocky:
http://pulpito.ceph.com/lflores-2025-11-10_16:32:42-rados:basic-main-distro-default-smithi/

Tests from main came back clean, which indicates that this is specific to rocky10.

Latest updates are in https://tracker.ceph.com/issues/73750#note-5.

Adam Emmerson is looking into the rocky10 issue and seeing if it's possible to reproduce locally.

Laura Flores wrote in #note-9:

Adam Emmerson is looking into the rocky10 issue and seeing if it's possible to reproduce locally.

Status is still the same as of now.

Latest update: https://tracker.ceph.com/issues/73750#note-11

Adam may have found a local reproducer and is working on it.

Ran some tests to verify the fix on the latest rocky10 branch:
https://pulpito.ceph.com/lflores-2025-12-01_21:19:40-rados:basic-wip-rocky10-branch-of-the-day-2025-11-26-1764177936-distro-default-smithi/

./teuthology/virtualenv/bin/teuthology-suite -v -m smithi -c wip-rocky10-branch-of-the-day-2025-11-26-1764177936 -r yaarit-2025-11-06_20:06:52-rados:basic-wip-rocky10-branch-of-the-day-2025-11-05-1762369819-distro-default-smithi -p 60 -N 3 --filter-all "rados_api_tests" 

We are also testing this on plain main (details in description).

@Laura Flores fyi https://github.com/ceph/ceph/pull/66244 is still in draft mode and there are many related failures in the rados suite as a result.

The plain main run is approved: https://tracker.ceph.com/projects/rados/wiki/MAIN#httpstrackercephcomissues73749

If the test results that were scheduled by @Nitzan Mordechai on the latest rocky10 of the day branch look good, then we can consider the PR approved on all angles: https://tracker.ceph.com/issues/74070

@Laura Flores, there are 3 different core dumps on ceph-mgr that we need to check which of the PRs are causing them, the list is on https://tracker.ceph.com/issues/74070 and i'm still looking on them.

Nitzan Mordechai wrote in #note-24:

@Laura Flores, there are 3 different core dumps on ceph-mgr that we need to check which of the PRs are causing them, the list is on https://tracker.ceph.com/issues/74070 and i'm still looking on them.

ACK