Skip to content

crimson: Add objectstore tool with partial functionality#63604

Merged
Matan-B merged 4 commits intoceph:mainfrom
guojidan:kv-tool
Jul 30, 2025
Merged

crimson: Add objectstore tool with partial functionality#63604
Matan-B merged 4 commits intoceph:mainfrom
guojidan:kv-tool

Conversation

@guojidan
Copy link
Contributor

@guojidan guojidan commented May 30, 2025

Contribution Guidelines

  • To sign and title your commits, please refer to Submitting Patches to Ceph.

  • If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

  • When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

  • Tracker (select at least one)
    • References tracker ticket
    • Very recent bug; references commit where it was introduced
    • New feature (ticket optional)
    • Doc update (no ticket needed)
    • Code cleanup (no ticket needed)
  • Component impact
    • Affects Dashboard, opened tracker ticket
    • Affects Orchestrator, opened tracker ticket
    • No impact that needs to be tracked
  • Documentation (select at least one)
    • Updates relevant documentation
    • No doc update is appropriate
  • Tests (select at least one)
Show available Jenkins commands

closed: https://tracker.ceph.com/issues/71504

PR Description:

This PR introduces a new crimson-objectstore-tool that provides essential debugging and inspection capabilities for Crimson object stores.

Features:

  • Store inspection: List PGs, objects, and omap entries
  • OMAP operations: Get/set/list omap key-value pairs

Key Components:

  • crimson_objectstore_tool.cc - Main tool implementation with CLI interface
  • objectstore_tool.{cc,h} - Core functionality wrapper around FuturizedStore
  • CMake integration for build system

Usage Examples:

it's fully compatible with ceph-objectstore-tool

@guojidan guojidan requested a review from a team as a code owner May 30, 2025 09:06
@guojidan guojidan marked this pull request as draft May 30, 2025 09:35
@Matan-B Matan-B added this to Crimson Jun 3, 2025
@Matan-B Matan-B moved this to Awaits review in Crimson Jun 3, 2025
@Matan-B Matan-B self-requested a review June 3, 2025 09:14
@guojidan guojidan force-pushed the kv-tool branch 2 times, most recently from 4b86652 to 4621266 Compare June 5, 2025 07:26
@github-actions github-actions bot added the tests label Jun 5, 2025
@guojidan guojidan marked this pull request as ready for review June 5, 2025 07:28
@guojidan
Copy link
Contributor Author

guojidan commented Jun 5, 2025

jenkins test make check

@guojidan
Copy link
Contributor Author

guojidan commented Jun 5, 2025

jenkins test make check arm64

@cyx1231st cyx1231st self-requested a review June 9, 2025 08:37
Copy link
Member

@cyx1231st cyx1231st left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry about the late review. I'm not very familiar with the background of objectstore tool yet.

It will be helpful to understand the specific scenarios this objectstore tool is designed for as well as the related Ceph infrastructure.

Copy link
Contributor

@Matan-B Matan-B left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry about the late review. I'm not very familiar with the background of objectstore tool yet.

It will be helpful to understand the specific scenarios this objectstore tool is designed for as well as the related Ceph infrastructure.

See existing obejctstore tool used in classic: https://docs.ceph.com/en/latest/man/8/ceph-objectstore-tool/
This offline tool allows exposing and manipulating the underlying rados objects. This tool would be essential to mitigate issue with deployed clusters or for debugging/reproducing issues for development (e.g listing the objects to spot leaks, setting specific metadata to reproduce scenarios or exposing dedicated behavior to mitigate issues like "reset-last-complete"). This should largely help with identifying issues at the objectstore level in the future (and possibly now).


@guojidan, Thank you for working on this one! After @cyx1231st comments are addressed, would you like to consider switching some of the intorduced code to coroutines? This would make the tool easier to maintain in future iterations of it.

@Matan-B Matan-B moved this from Awaits review to In Progress in Crimson Jun 12, 2025
@guojidan
Copy link
Contributor Author

guojidan commented Jun 13, 2025

WIP

@guojidan guojidan force-pushed the kv-tool branch 2 times, most recently from 79bf069 to 975533c Compare June 17, 2025 03:34
@guojidan
Copy link
Contributor Author

  1. use coroutines whenever possible, somewhere can not use coroutine because have .get() code.
  2. Automatically obtain the shard-id without requiring user input.
  3. Except for outputting the data after executing the command, all logging should be done using seastar::logger.
  4. fix some format error.

@guojidan guojidan force-pushed the kv-tool branch 2 times, most recently from f7474cf to c99aff5 Compare June 17, 2025 05:47
@Matan-B Matan-B moved this from In Progress to Awaits review in Crimson Jun 17, 2025
Copy link
Member

@cyx1231st cyx1231st left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good from SeaStore perspective!

Copy link
Contributor

@Matan-B Matan-B left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewing qa/.. side first, left few comments.

Comment on lines +8 to +12
- install:
ceph:
flavor: crimson
- ceph:
flavor: crimson
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We no longer specify flavors in qa. See #63833

Copy link
Contributor

@Matan-B Matan-B Jun 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The structure of the new added test suite should look something like this:
Matan-B@4cccea5
Feel free to cherry-pick or edit according to the structure there.

db = {}

# Setup local and remote data
LOCALDIR = tempfile.mkdtemp("cos")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does "cos" stand for?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😆 I don't know. I'm just imitating ceph_objectstore_tool.py to write, so I'll keep consistent with ceph_objectstore_tool.py and use cod.

@@ -0,0 +1,536 @@
"""
crimson_objectstore_tool - Test of crimson-objectstore-tool utility
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we reuse the existing qa/tasks/ceph_objectstore_tool.py? IIUC, the main difference is the tool binary name which could be defined based on a new supported parameter? Something like:

- objectstore_tool:
    crimson_objectstore_tool: true #use crimson-objectstore-tool
    objects: 5
    pgnum: 8

and a corresponding:

CRIMSON = config.get('crimson_objectstore_tool', false)

Any unsupported features could be also skipped by introducing a SKIP_IF_CRIMSON method. We use similar approach with all /src/test/.. unittests.

@Matan-B
Copy link
Contributor

Matan-B commented Jul 16, 2025

rebasing on top of #62530 which changed omap behavior and require changes

@Matan-B
Copy link
Contributor

Matan-B commented Jul 17, 2025

The tool is finally available in the rpm package:

2025-07-17T07:41:38.033 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --op list
2025-07-17T07:41:38.162 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to get seastar arguments: Could not read seastore superblock: Decoder at 'static void 

Matan-B added 2 commits July 23, 2025 13:16
fixes: https://tracker.ceph.com/issues/71504

Signed-off-by: junxiang Mu <1948535941@qq.com>
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Include the tool when packaging for deb/rpm.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
@Matan-B
Copy link
Contributor

Matan-B commented Jul 24, 2025

@Matan-B Matan-B removed the DNM label Jul 24, 2025
@Matan-B Matan-B moved this from In Progress to Awaits review in Crimson Jul 24, 2025
@Matan-B Matan-B requested review from a team, amathuria, athanatos and cyx1231st July 24, 2025 15:11
@Matan-B
Copy link
Contributor

Matan-B commented Jul 28, 2025

jenkins retest this please

@amathuria
Copy link
Contributor

jenkins test make check arm64

2 similar comments
@Matan-B
Copy link
Contributor

Matan-B commented Jul 28, 2025

jenkins test make check arm64

@Matan-B
Copy link
Contributor

Matan-B commented Jul 29, 2025

jenkins test make check arm64

@Matan-B Matan-B moved this from Awaits review to Tested in Crimson Jul 29, 2025
@Matan-B
Copy link
Contributor

Matan-B commented Jul 29, 2025

jenkins test make check

Matan-B added 2 commits July 29, 2025 13:42
fixes: https://tracker.ceph.com/issues/71504

Signed-off-by: junxiang Mu <1948535941@qq.com>
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
@Matan-B
Copy link
Contributor

Matan-B commented Jul 29, 2025

@Matan-B Matan-B merged commit c892f3b into ceph:main Jul 30, 2025
13 checks passed
@Matan-B Matan-B moved this from Tested to Merged (Main) in Crimson Jul 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Merged (Main)

Development

Successfully merging this pull request may close these issues.

5 participants