Skip to content

os/bluestore: Debug code to make reshard fail faster#62817

Merged
aclamk merged 2 commits intoceph:mainfrom
aclamk:aclamk-bs-reshard-problem-replicate
Jun 6, 2025
Merged

os/bluestore: Debug code to make reshard fail faster#62817
aclamk merged 2 commits intoceph:mainfrom
aclamk:aclamk-bs-reshard-problem-replicate

Conversation

@aclamk
Copy link
Contributor

@aclamk aclamk commented Apr 15, 2025

Catch reshard on producing an invalid result on encode.
This makes it much easier to catch errors.

This PR is not intended to be merged. It is a helper to verify #62816.

Replication steps:
1.
Deploy 5-6 OSD vstart cluster
2.

./bin/ceph osd erasure-code-profile set myecprofile k=2 m=1
./bin/ceph osd pool create ecpool 16 16 erasure myecprofile
./bin/ceph osd pool set ecpool allow_ec_overwrites true
fio --ioengine=rados --pool=ecpool --touch_objects=0 --clientname=admin --conf=ceph.conf --time_based --runtime=1h --iodepth=64 --numjobs=8 --group_reporting=1 --rw=randwrite --file_service_type=pareto:0.20:0 --bssplit=4k/16:8k/10:12k/9:16k/8:20k/7:24k/7:28k/6:32k/6:36k/5:40k/5:44k/4:48k/4:52k/4:56k/3:60k/3:64k/3 --size=50000M --nrfiles=12500 '--filename_format=stress_obj.$jobnum.$filenum' --name=test-ec-esb

Contribution Guidelines

  • To sign and title your commits, please refer to Submitting Patches to Ceph.

  • If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

  • When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

  • Tracker (select at least one)
    • References tracker ticket
    • Very recent bug; references commit where it was introduced
    • New feature (ticket optional)
    • Doc update (no ticket needed)
    • Code cleanup (no ticket needed)
  • Component impact
    • Affects Dashboard, opened tracker ticket
    • Affects Orchestrator, opened tracker ticket
    • No impact that needs to be tracked
  • Documentation (select at least one)
    • Updates relevant documentation
    • No doc update is appropriate
  • Tests (select at least one)
Show available Jenkins commands

@aclamk aclamk added the DNM label Apr 15, 2025
@aclamk aclamk requested a review from a team as a code owner April 15, 2025 09:08
Jayaprakash-ibm added a commit to Jayaprakash-ibm/ceph that referenced this pull request Apr 16, 2025
Adds a test to reproduce the !ito->is_valid() assertion in BlueStore
with bluestore_elastic_shared_blobs=true on a 2+1 EC pool using a
FIO randwrite workload (512 concurrent ops, 50G, 12,500 objects).
The test deploys a 6-OSD cluster and runs FIO for 1 hour via workunit,
failing if an OSD crashes. Validates the bug in PR ceph#62817.

Signed-off-by: Jaya Prakash <jayaprakash@ibm.com>
@aclamk aclamk force-pushed the aclamk-bs-reshard-problem-replicate branch from 57836ec to 59f1a77 Compare May 28, 2025 14:15
@aclamk aclamk added aclamk-testing-nauvoo bluestore testing and removed DNM labels May 28, 2025
@ronen-fr
Copy link
Contributor

Is it still a DNM?

@aclamk aclamk requested a review from ifed01 June 3, 2025 09:39
@aclamk
Copy link
Contributor Author

aclamk commented Jun 3, 2025

jenkins test make check

Copy link
Contributor

@ifed01 ifed01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, please don't forget to set default value to false for that new config param.

aclamk added 2 commits June 5, 2025 08:06
Catch reshard on producing invalid result on encode.
This makes it much easier to catch error.
Intented for teuthology testing.

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>
@aclamk aclamk force-pushed the aclamk-bs-reshard-problem-replicate branch from 831e4a0 to 3ce9d03 Compare June 5, 2025 08:12
@github-actions
Copy link

github-actions bot commented Jun 5, 2025

Config Diff Tool Output

+ added: bluestore_debug_extent_map_encode_check

The above configuration changes are found in the PR. Please update the relevant release documentation if necessary.

@aclamk
Copy link
Contributor Author

aclamk commented Jun 5, 2025

jenkins test make check

@aclamk
Copy link
Contributor Author

aclamk commented Jun 5, 2025

jenkins test windows

@aclamk
Copy link
Contributor Author

aclamk commented Jun 6, 2025

jenkins test make check

@aclamk
Copy link
Contributor Author

aclamk commented Jun 6, 2025

Verified by https://tracker.ceph.com/issues/71503

@aclamk aclamk merged commit 11d6006 into ceph:main Jun 6, 2025
12 of 13 checks passed
@ljflores
Copy link
Member

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants