build flavors: Cleanup "crimson", introduce "debug" by Matan-B · Pull Request #2497 · ceph/ceph-build

Matan-B · 2025-11-23T12:57:18Z

With ceph/ceph#66229 merged,
Crimson is now included (though not used) by default in our RPM builds.
This means the existing default flavor can also be used for Crimson testing
by selecting Crimson as the default OSD package.

Notes:

The previous workaround related to DWITH_STATIC_LIBSTDCXX is no longer
relevant for Crimson (it was tied to older compiler issues).
The crimson-only branch name selection is also cleaned up,
as centos9-only can now be used instead.
This change breaks Crimson Tentacle CI builds:
The packaging update that includes Crimson in RPM builds was not backported to Tentacle.
Tentacle builds would still require a dedicated flavor that enables WITH_CRIMSON
However, since Crimson changes have not been backported to Tentacle (since the first RCA),
there is no strong reason to keep building and testing the same Crimson HEAD.
So we can use this opprtuinity to stop nightly Crimson/Tentacle builds and tests.

See last Crimson tentacle run, (which is not expected to change):
https://pulpito.ceph.com/teuthology-2025-11-22_22:56:11-crimson-rados-tentacle-distro-crimson-debug-smithi/

Currently, the only flavor used for testing is "default". While this ensures that the tested flavor matches the released flavor, it can also be a limitation. Introducing a debug flavor would allow us to test branches that require additional or more thorough validation. The main difference is that built-in assertions would be compiled in. The reasons for this change are: a) The Crimson suite uses a crimson-debug flavor for project PR gating. The next commit will clean up the Crimson flavor entirely, and the new debug flavor introduced here could be used as its replacement. b) Good practice: having an additional build with debug enabled is useful when retesting or performing extra checks. Initially, the new debug flavor will only apply to centos9 builds. If it proves valuable, we can expand support to other distros. Note: The current way to schedule debug builds is by using a *-debug branch name. Having a dedicated flavor seems more straightforward. Signed-off-by: Matan Breizman <mbreizma@redhat.com>

tchaikov · 2025-11-25T13:24:33Z

will take a look later this week.

Matan-B · 2025-11-25T17:00:37Z

Marking as dnm for now, to get the crimson-rados suite ready for the changes here prior to this merged

Matan-B · 2025-12-16T10:59:55Z

changeset:

Added a temporary way to schedule *-crimson-tentacle builds.

Next steps (ceph/ceph#66648)

Update crimson-rados suite to use the new debug flavor by default
- Make sure we are able to pass --flavor default even if the above is set.
Update crontab to stop using crimson-debug flavor

With ceph/ceph-build#2497 merged we no loger build Tentacle+Crimson regularly. As Crimson no longer backport changes into Tentacle, there's no reason to keep testing it. Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Use the new Debug flavor introduced in ceph/ceph-build#2497. This should provide more nightly test coverage. Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Prior to ceph/ceph-build#2497, Crimson was tested using the crimson-debug flavor. To preserve the same testing strategy, use the "debug" flavor for crimson-rados testing runs. Note: `--flavor default` should override this setting to allow testing with non-debug flavors. Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Matan-B · 2025-12-16T11:17:02Z

This PR is ready to be merged afaict.
Note, Crimson nightly runs are using the default flavor for quite some time now and all seem sane. There's no reason to keep the additional flavor.

There's no real way to test the changes here I think so we can fix issues once this is merged.
@athanatos, @tchaikov - Is there anything I've missed?

tchaikov

left a few comments.

ceph-dev-new-setup/build/build

ceph-perf-pull-requests/config/definitions/ceph-perf-pull-requests.yml

With ceph/ceph#66229 merged, Crimson is now included (though not used) by default in our RPM builds. This means the existing default flavor can also be used for Crimson testing by selecting Crimson as the default OSD package. Notes: * The previous workaround related to DWITH_STATIC_LIBSTDCXX is no longer relevant for Crimson (it was tied to older compiler issues). * The crimson-only branch name selection is also cleaned up, as centos9-only can now be used instead. * This change breaks Crimson Tentacle CI builds: The packaging update that includes Crimson in RPM builds was not backported to Tentacle. Tentacle builds would still require a dedicated flavor that enables WITH_CRIMSON However, since Crimson changes have not been backported to Tentacle (since the first RCA), there is no strong reason to keep building and testing the same Crimson HEAD. So we can use this opprtuinity to stop nightly Crimson/Tentacle builds and tests. See last Crimson tentacle run, (which is not expected to change): https://pulpito.ceph.com/teuthology-2025-11-22_22:56:11-crimson-rados-tentacle-distro-crimson-debug-smithi/ Signed-off-by: Matan Breizman <mbreizma@redhat.com>

See the following comment: ``` Tentacle is the last release that needs dedicated Crimson builds, Later releases are able to use Crimson with the default build. As the "Crimson flavor" is no longer available, we need a *temporary* way to be able build Crimson for tentacle. Note: This could be removed once Crimson we have Umbrella release builds. ``` Signed-off-by: Matan Breizman <mbreizma@redhat.com>

batrick · 2025-12-18T15:54:11Z

ceph-dev-new-setup/build/build

+# Later releases are able to use Crimson with the default build.
+# As the "Crimson flavor" is no longer available, we need a *temporary* way
+# to be able build Crimson for tentacle.
+# Note: This could be removed once Crimson we have Umbrella release builds.


We will still need to be able to build Tentacle until upgrade testing from T to V is no longer necessary (probably when V is EOL).

This section is only true for Crimson. We only build the last stable release (and main) as there's no upgrade testing/support yet. Starting U we will probably start working on upgradability. We still break on-disk data structures from time to time as we are still flexible with this - the plan is to stop with this approach starting U.
This is why we should be able to remove this bit once U is out as Crimson will no longer be supported in T at all. I'll mention this in this comment to avoid confusion.

batrick · 2025-12-18T15:56:01Z

ceph-dev-new-setup/build/build

+# As the "Crimson flavor" is no longer available, we need a *temporary* way
+# to be able build Crimson for tentacle.
+# Note: This could be removed once Crimson we have Umbrella release builds.
+if [[ "$BRANCH" == *-crimson-tentacle ]]; then


Let's have -<suffix> only be -debug otherwise you can't have both of these?

Alternatively: why not just turn these switches on unconditionally for tentacle?

Let's have - only be -debug otherwise you can't have both of these?

I guess the -debug suffix could be removed with the new "debug" flavor introduced. @batrick, WDYT?

I'll try moving this logic to git trailers instead of branch name suffix

Alternatively: why not just turn these switches on unconditionally for tentacle?

Unless I misunderstand the suggestion, this way every Tentacle build will be with Crimson enabled. We only want to be able to enable Crimson with tentacle iff specifically needed for some reason.

Let's have - only be -debug otherwise you can't have both of these?

I guess the -debug suffix could be removed with the new "debug" flavor introduced. @batrick, WDYT?

Ah, I missed that. That sounds even better!

I'll try moving this logic to git trailers instead of branch name suffix

That would be great. I've tried to make it easier to build branches with these new git trailers here:

https://github.com/ceph/ceph/blob/2e87714b94a9e16c764ef6f97de50aecf1b0c41e/src/script/ptl-tool.py#L604-L605

Please give it a try! Maybe you can submit a change for the new git trailer?

Alternatively: why not just turn these switches on unconditionally for tentacle?

Unless I misunderstand the suggestion, this way every Tentacle build will be with Crimson enabled. We only want to be able to enable Crimson with tentacle iff specifically needed for some reason.

Got it, nevermind.

@batrick Are these suggestions 'nice to haves' or must haves? I think CentOS9 builds are blocked by this PR not being merged.

nice to haves

djgalloway · 2025-12-19T13:36:25Z

@djgalloway hi David, the failure is not related to this change. and i don't think it's transient:

Building remotely on [ec2-44-223-66-26.compute-1.amazonaws.com](https://jenkins.ceph.com/computer/ec2-44-223-66-26.compute-1.amazonaws.com/) (buster jammy ec2 arm64 huge installed-os-jammy xenial) in workspace /home/jenkins-build/build/workspace/ceph-build-pull-requests
...
/tmp/jenkins7947793624570378365.sh: line 1891: warning: setlocale: LC_ALL: cannot change locale (en_us.UTF-8): No such file or directory
...
+ for reqs in $WORKSPACE/ansible/requirements/*
+ ./.venv/bin/ansible-galaxy install -r /home/jenkins-build/build/workspace/ceph-build-pull-requests/ansible/requirements/sensu-requirements.yml -p /home/jenkins-build/build/workspace/ceph-build-pull-requests/ansible/roles --force
ERROR: Ansible could not initialize the preferred locale: unsupported locale setting
Build step 'Execute shell' marked build as failure

see https://jenkins.ceph.com/job/ceph-build-pull-requests/2909/consoleFull#19485512547dc3600c-28d8-42e2-9653-291a0c275fdf

probably we need to generate this locale in ceph-cm-ansible:role/testnode ?

It would go in here. I'll have to take the builders offline to fix. #2512

djgalloway · 2025-12-19T15:48:47Z

@djgalloway hi David, the failure is not related to this change. and i don't think it's transient:
Building remotely on [ec2-44-223-66-26.compute-1.amazonaws.com](https://jenkins.ceph.com/computer/ec2-44-223-66-26.compute-1.amazonaws.com/) (buster jammy ec2 arm64 huge installed-os-jammy xenial) in workspace /home/jenkins-build/build/workspace/ceph-build-pull-requests
...
/tmp/jenkins7947793624570378365.sh: line 1891: warning: setlocale: LC_ALL: cannot change locale (en_us.UTF-8): No such file or directory
...
+ for reqs in $WORKSPACE/ansible/requirements/*
+ ./.venv/bin/ansible-galaxy install -r /home/jenkins-build/build/workspace/ceph-build-pull-requests/ansible/requirements/sensu-requirements.yml -p /home/jenkins-build/build/workspace/ceph-build-pull-requests/ansible/roles --force
ERROR: Ansible could not initialize the preferred locale: unsupported locale setting
Build step 'Execute shell' marked build as failure
see https://jenkins.ceph.com/job/ceph-build-pull-requests/2909/consoleFull#19485512547dc3600c-28d8-42e2-9653-291a0c275fdf
probably we need to generate this locale in ceph-cm-ansible:role/testnode ?
It would go in here. I'll have to take the builders offline to fix. #2512

Done

djgalloway · 2025-12-19T15:52:02Z

I am merging this as-is. @batrick's requested changes are for a job that largely is no longer used anyway.

Matan-B · 2025-12-21T08:37:43Z

Thanks for merging, I'll follow up with the suggestions above.

edit: pushed #2515 for the cleanup part. As per the *-crimson-tentacle, using git trailers is not an option since we are not allowing to pass CMake options (future possible new feature?) . I've kept the existing behavior as we are likely to remove it soon anyway (first U RC).

the newly introduced "debug" flavor from ceph#2497 can be used insead. Signed-off-by: Matan Breizman <mbreizma@redhat.com>

The newly introduced "debug" flavor from ceph#2497 can be used instead. Add "DWITH_CEPH_DEBUG_MUTEX" to preserve existing *-debug behavior. Signed-off-by: Matan Breizman <mbreizma@redhat.com>

ceph#2497 introduced a debug flavor. This seems to cause conflicts with the image being pushed to quay as one of the flavors might override the other. One possible solution to this would be to name the debug images differently. However, since the current behavior might cause unexpected testing results. Skip building containers for debug builds at all. Signed-off-by: Matan Breizman <mbreizma@redhat.com>

We no longer have crimson-flavors see ceph/ceph-build#2497 Signed-off-by: Matan Breizman <mbreizma@redhat.com>

ceph/ceph-build#2497 introduced a debug flavor. This seems to cause conflicts with the image being pushed to quay as one of the flavors might override the other. Tag debug build containers explicitly. Alternative solution would be to skip debug containers all together. However. these might be useful for development purposes. Note, prune-quay might also need to be updated once this is merged. Signed-off-by: Matan Breizman <mbreizma@redhat.com>