cephadm, ceph-volume: deploy crimson OSDs using cephadm by shraddhaag · Pull Request #66811 · ceph/ceph

shraddhaag · 2026-01-06T12:58:04Z

This commit enables us to deploy both classic and crimson type OSDs using cephadm. To enable the same, a new feature, osd_type is added to DriverGroupSpec. The default value for the same is classic, but can also be set to crimson. When this value is read by cephadm, the entrypoint is changed from /usr/bin/ceph-osd to /usr/bin/ceph-osd-crimson.

Fixes: https://tracker.ceph.com/issues/74081

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins test classic perf Jenkins Job | Jenkins Job Definition
jenkins test crimson perf Jenkins Job | Jenkins Job Definition
jenkins test signed Jenkins Job | Jenkins Job Definition
jenkins test make check Jenkins Job | Jenkins Job Definition
jenkins test make check arm64 Jenkins Job | Jenkins Job Definition
jenkins test submodules Jenkins Job | Jenkins Job Definition
jenkins test dashboard Jenkins Job | Jenkins Job Definition
jenkins test dashboard cephadm Jenkins Job | Jenkins Job Definition
jenkins test api Jenkins Job | Jenkins Job Definition
jenkins test docs ReadTheDocs | Github Workflow Definition
jenkins test ceph-volume all Jenkins Jobs | Jenkins Jobs Definition
jenkins test windows Jenkins Job | Jenkins Job Definition
jenkins test rook e2e Jenkins Job | Jenkins Job Definition

You must only issue one Jenkins command per-comment. Jenkins does not understand
comments with more than one command.

src/python-common/ceph/deployment/drive_group.py

src/cephadm/cephadmlib/daemons/ceph.py

phlogistonjohn

typo in commit message subject: add osd_type to orchestratory

I also generally recommend prefixing the subject with a directoy/topic like nearly all the existing ceph commits do eg. cephadm: add osd_type to orchestrator

src/python-common/ceph/deployment/drive_group.py

Matan-B

Once ready, can you please update the Crimson user docs with how a Crimson OSD could be added (command example)?

src/ceph-volume/ceph_volume/devices/lvm/activate.py

guits · 2026-01-30T11:05:00Z

jenkins ceph-volume unit tests

guits · 2026-01-30T14:13:38Z

jenkins ceph-volume unit tests

guits · 2026-01-30T15:21:02Z

@shraddhaag should this test be updated ?

ceph/src/pybind/mgr/cephadm/tests/test_cephadm.py

Lines 1177 to 1184 in af3ff2b

    
           _run_cephadm.assert_any_call( 
        
               'test', 'osd', 'ceph-volume', 
        
               ['--config-json', '-', '--', 'lvm', 'batch', 
        
                   '--no-auto', '/dev/sdb', '--db-devices', '/dev/sdc', 
        
                   '--wal-devices', '/dev/sdd', '--objectstore', 'bluestore', '--yes', '--no-systemd'], 
        
               env_vars=['CEPH_VOLUME_OSDSPEC_AFFINITY=noncollocated'], 
        
               error_ok=True, stdin='{"config": "", "keyring": ""}', 
        
           )

(is it missing the new --osd-type flag ?)

This commit enables us to deploy both classic and crimson type OSDs using cephadm. To enable the same, a new feature, osd_type is added to DriverGroupSpec. The default value for the same is classic, but can also be set to crimson. When this value is read by cephadm, the entrypoint is changed from /usr/bin/ceph-osd to /usr/bin/ceph-osd-crimson. Fixes: https://tracker.ceph.com/issues/74081 Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>

Prior to this commit, ceph-volume was using hardcoded OSD binary to issue commands (eg - to perform mkfs, etc). This commit enables ceph-volume to start supporting crimson OSDs. A new argument, --osd-type is introduced with the default value classic. When this parameter is set to 'crimson', ceph-osd-crimson binary will be used to execute OSD commands. Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>

Matan-B

lgtm!
I think that we have an ack from cephadm side (@adk3798).
Missing an ack from ceph-vol (@guits) to merge this.

Matan-B · 2026-02-02T13:26:51Z

src/pybind/mgr/cephadm/tests/test_cephadm.py

        [
            # no preview and only one disk, prepare is used due the hack that is in place.
-            (['/dev/sda'], False, ["lvm batch --no-auto /dev/sda --objectstore bluestore --yes --no-systemd"]),
+            (['/dev/sda'], False, ["lvm batch --no-auto /dev/sda --objectstore bluestore --osd-type classic --yes --no-systemd"]),


Can we keep classic as the default to avoid passing it in the test files?

I've added classic as the default in both cephadm and ceph-volume.

https://github.com/ceph/ceph/pull/66811/changes#diff-6b4e0db0cd759015d29f327cd90efa79030025319c317c449efc06841c6e120bR95

https://github.com/ceph/ceph/pull/66811/changes#diff-809b1c074086d65e7f527ec569e94a1ebba49a7e2774b15a3987793f78456444R295

The reason we have to supply the commands this way is because DriveGroupSpec injects these arguments. Also, in future when we have to make crimson as the default, we won't have to make much changes, only change the default value.

shraddhaag · 2026-02-03T07:51:28Z

jenkins test rook e2e

shraddhaag · 2026-02-03T07:51:47Z

jenkins test make check

shraddhaag · 2026-02-04T08:01:41Z

Some context about why ceph-volume changes were needed:

While testing out these changes 0fdc81b, OSDs were getting killed in the boot process itself. Looking at the debug logs for OSD boot process (post #67090), the final error looked something like this:

DEBUG 2026-01-28 15:14:36,301 [shard 0:main] osd - OSD::start: open metadata collection
DEBUG 2026-01-28 15:14:36,301 [shard 0:main] osd - OSD::open_meta_coll: opening metadata collection
DEBUG 2026-01-28 15:14:36,301 [shard 0:main] alienstore - open_collection
INFO  2026-01-28 15:14:36,301 [shard 0:main] prioritycache - prioritycache tune_memory target: 4294967296 mapped: 6004736 unmapped: 688128 heap: 6692864 old mem: 134217728 new mem: 2841625343
DEBUG 2026-01-28 15:14:36,301 [shard 0:main] osd - OSD::open_meta_coll: registering metadata collection
DEBUG 2026-01-28 15:14:36,302 [shard 0:main] osd - OSD::start: loading superblock
DEBUG 2026-01-28 15:14:36,302 [shard 0:main] osd - OSDMeta::load_superblock:
DEBUG 2026-01-28 15:14:36,302 [shard 0:main] alienstore - read
ERROR 2026-01-28 15:14:36,302 [shard 0:main] osd - /ceph/rpmbuild/BUILD/ceph-20.3.0-4805-g06aa012d/src/crimson/common/errorator.h:1319 : In function 'crimson::ct_error::assert_all::operator()<const crimson::unthrowable_wrapper<const std::error_code&, ((const std::error_code&)(& crimson::ec<2>))>&>(const crimson::unthrowable_wrapper<const std::error_code&, ((const std::error_code&)(& crimson::ec<2>))>&)::<lambda(auto:119&&)> [with auto:119 = const std::error_code&]', abort() 
 open_meta_coll error: No such file or directory

ERROR 2026-01-28 15:14:36,302 [shard 0:main] osd - Aborting Got SIGABRT on shard 0 - Stopping all shards
ERROR 2026-01-28 15:14:37,591 [shard 0:main] osd - Got SIGABRT on shard 0

ie., crimson OSD was failing when trying to open the superblock. The error being No such file or directory. So I looked in the logs above to see if the superblock was created or not:

2026-01-28T15:08:59.138+0000 7f2628a6a780 10 write_superblock sb(0577bd6c-fc5b-11f0-a2d1-525400b1a44c osd.0 401f34c7-c712-4e99-a948-d47fba81a834 e0 maps [] lci=[0,0] tlb=0)

From the above log, while the superblock is indeed getting created, this log is not coming from crimson OSD, but instead from classic OSD. (ref: https://github.com/shraddhaag/ceph/blob/main/src/osd/OSD.cc#L2134).

The problem being, the first command to do mkfs was being issued by the classic OSD binary and the OSD boot was being done by crimson OSD. In cephadm, mkfs is done by ceph-volume. Looking at the code, the OSD binary used was hardcoded ().

To fix this, a new flag is introduced in ceph-volume --osd type, with default being set to classic. When crimson value is provided, the crimson binary is chosen to issue the mkfs command.

shraddhaag · 2026-02-04T08:02:31Z

The PR has been tested with KCLI. Both a classic and a crimson cluster came up successfully with HEALTH_OK.

Docs to follow in a follow-up PR.

shraddhaag · 2026-02-04T08:09:42Z

jenkins ceph-volume unit tests

Dismissing the review for merging this PR. Will add the default value in a followup PR.

github-actions · 2026-02-04T11:23:38Z

This is an automated message by src/script/redmine-upkeep.py.

I have resolved the following tracker ticket due to the merge of this PR:

https://tracker.ceph.com/issues/74081

No backports are pending for the ticket. If this is incorrect, please update the tracker
ticket and reset to Pending Backport state.

Update Log: https://github.com/ceph/ceph/actions/runs/21669492167

This commit adds support for deploying crimson OSDs using cephadm with the method raw. Support for lvm crimson OSD was added previously in: ceph#66811. Fixes: https://tracker.ceph.com/issues/74960 Signed-off-by: Shraddha Agrawal <shraddha.agrawal000@gmail.com>

github-actions bot added ceph-volume cephadm pybind labels Jan 6, 2026

adk3798 approved these changes Jan 8, 2026

View reviewed changes

src/python-common/ceph/deployment/drive_group.py Outdated Show resolved Hide resolved

adk3798 reviewed Jan 8, 2026

View reviewed changes

src/python-common/ceph/deployment/drive_group.py Outdated Show resolved Hide resolved

src/cephadm/cephadmlib/daemons/ceph.py Show resolved Hide resolved

shraddhaag force-pushed the wip-shraddhaag-cephadm-add-osd-type branch 4 times, most recently from 168cf2a to 8a66d63 Compare January 12, 2026 06:50

Matan-B mentioned this pull request Jan 12, 2026

qa/config/crimson_qa_overrides: cephadm to use crimson binary #66458

Closed

14 tasks

Matan-B added this to Crimson Jan 12, 2026

Matan-B moved this to In Progress in Crimson Jan 12, 2026

phlogistonjohn previously requested changes Jan 12, 2026

View reviewed changes

src/python-common/ceph/deployment/drive_group.py Show resolved Hide resolved

github-actions bot added the orchestrator label Jan 19, 2026

shraddhaag force-pushed the wip-shraddhaag-cephadm-add-osd-type branch from 1888d7a to 9c784b5 Compare January 19, 2026 14:48

Matan-B reviewed Jan 19, 2026

View reviewed changes

This was referenced Jan 20, 2026

cmake,qa,crimson: misc ASAN usage fixes #66994

Open

Revert "common/options: fix heap-use-after-free by using never-destro… #67020

Merged

shraddhaag mentioned this pull request Jan 28, 2026

crimson/osd: add verbose DEBUG logs for OSD startup #67090

Merged

14 tasks

shraddhaag force-pushed the wip-shraddhaag-cephadm-add-osd-type branch from 9c784b5 to 818b453 Compare January 30, 2026 10:30

guits requested changes Jan 30, 2026

View reviewed changes

src/ceph-volume/ceph_volume/devices/lvm/activate.py Show resolved Hide resolved

shraddhaag force-pushed the wip-shraddhaag-cephadm-add-osd-type branch 2 times, most recently from afa11f0 to 9302a65 Compare January 30, 2026 13:48

shraddhaag force-pushed the wip-shraddhaag-cephadm-add-osd-type branch 4 times, most recently from 151cf17 to 1521dcb Compare February 2, 2026 07:55

shraddhaag force-pushed the wip-shraddhaag-cephadm-add-osd-type branch from 1521dcb to 346a396 Compare February 2, 2026 12:05

shraddhaag added 2 commits February 2, 2026 17:43

shraddhaag force-pushed the wip-shraddhaag-cephadm-add-osd-type branch from 346a396 to 585992a Compare February 2, 2026 12:15

Matan-B approved these changes Feb 2, 2026

View reviewed changes

shraddhaag marked this pull request as ready for review February 4, 2026 08:08

shraddhaag requested a review from a team as a code owner February 4, 2026 08:08

shraddhaag requested a review from guits February 4, 2026 08:09

shraddhaag changed the title ~~WIP: deploy crimson OSDs using cephadm~~ cephadm, ceph-volume: deploy crimson OSDs using cephadm Feb 4, 2026

guits approved these changes Feb 4, 2026

View reviewed changes

shraddhaag requested a review from phlogistonjohn February 4, 2026 09:46

shraddhaag merged commit 030f947 into ceph:main Feb 4, 2026
21 of 25 checks passed

shraddhaag moved this from In Progress to Merged (Main) in Crimson Feb 4, 2026

This was referenced Feb 10, 2026

cephadm, ceph-volume: add tests for crimson OSD support #67290

Merged

doc: add instructions for deploying crimson with cephadm #67220

Merged

cephadm: add support for seastore #67274

Merged

shraddhaag mentioned this pull request Feb 17, 2026

cephdm: add support for raw bluestore + crimson OSD deployment #67374

Merged

14 tasks

Conversation

shraddhaag commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution Guidelines

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

phlogistonjohn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Matan-B left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

guits commented Jan 30, 2026

Uh oh!

guits commented Jan 30, 2026

Uh oh!

guits commented Jan 30, 2026

Uh oh!

Matan-B left a comment

Choose a reason for hiding this comment

Uh oh!

Matan-B Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

shraddhaag Feb 4, 2026

Choose a reason for hiding this comment

Uh oh!

shraddhaag commented Feb 3, 2026

Uh oh!

shraddhaag commented Feb 3, 2026

Uh oh!

shraddhaag commented Feb 4, 2026

Uh oh!

shraddhaag commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shraddhaag commented Feb 4, 2026

Uh oh!

Uh oh!

github-actions bot commented Feb 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

shraddhaag commented Jan 6, 2026 •

edited

Loading

shraddhaag commented Feb 4, 2026 •

edited

Loading