Skip to content

qa/suites/crimson-rados/thrash: enable chance_down#56511

Merged
Matan-B merged 1 commit intoceph:mainfrom
Matan-B:wip-matanb-crimson-thrash-osd
Apr 17, 2024
Merged

qa/suites/crimson-rados/thrash: enable chance_down#56511
Matan-B merged 1 commit intoceph:mainfrom
Matan-B:wip-matanb-crimson-thrash-osd

Conversation

@Matan-B
Copy link
Copy Markdown
Contributor

@Matan-B Matan-B commented Mar 27, 2024 •
edited by athanatos
Loading 

As the thrash tests were introduced, some options were disabled
until the tests are stabilized.

Re-enable chance_down option (default is 0.4) to detect bugs on restart.

Since it will probably take few iterations before thrash and recovery tests ('default.yaml')
will pass successfully, add anoter 'simple.yaml' which should remain stable.

Fixes: https://tracker.ceph.com/issues/65130

Contribution Guidelines

  • To sign and title your commits, please refer to Submitting Patches to Ceph.

  • If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

  • When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

  • Tracker (select at least one)
    • References tracker ticket
    • Very recent bug; references commit where it was introduced
    • New feature (ticket optional)
    • Doc update (no ticket needed)
    • Code cleanup (no ticket needed)
  • Component impact
    • Affects Dashboard, opened tracker ticket
    • Affects Orchestrator, opened tracker ticket
    • No impact that needs to be tracked
  • Documentation (select at least one)
    • Updates relevant documentation
    • No doc update is appropriate
  • Tests (select at least one)
Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard cephadm
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox
  • jenkins test windows
  • jenkins test rook e2e

@Matan-B Matan-B requested review from a team and athanatos March 27, 2024 08:44
@Matan-B
Copy link
Copy Markdown
Contributor Author

Matan-B commented Mar 27, 2024 •
edited
Loading

@athanatos, testing with #56376 to verify that we catch the error now.

thrash tests with snaps enabled with this PR:
https://pulpito.ceph.com/matan-2024-03-27_08:49:03-crimson-rados-wip-sjust-crimson-testing-2024-03-20-distro-crimson-smithi/

Edit: OSD logs are not available yet but teuthology log looks promising:

2024-03-27T09:23:28.758 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [1, 2, 3, 0]

2024-03-27T09:23:33.824 INFO:tasks.thrashosds.thrasher:Reviving osd 1
2024-03-27T09:23:33.824 INFO:tasks.ceph.osd.1:Restarting daemon

2024-03-27T09:23:43.176 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s

@Matan-B
Copy link
Copy Markdown
Contributor Author

Matan-B commented Mar 27, 2024 •
edited
Loading

We have other issues that involve restarting that require handling before merging, tested this PR on main as well:
https://pulpito.ceph.com/matan-2024-03-27_13:02:57-crimson-rados-main-distro-crimson-smithi/

Hopefully that's caused by the scrub tracked issue

@athanatos
Copy link
Copy Markdown
Contributor

athanatos commented Mar 27, 2024

@Matan-B Ah, that makes sense. If you have time, it would be helpful to do a first pass of that run and file bugs for the defects. We can split up the work on fixing them.

@Matan-B
Copy link
Copy Markdown
Contributor Author

Matan-B commented Mar 28, 2024

@Matan-B Ah, that makes sense. If you have time, it would be helpful to do a first pass of that run and file bugs for the defects. We can split up the work on fixing them.

@athanatos,

https://tracker.ceph.com/issues/65203
https://tracker.ceph.com/issues/65201
https://tracker.ceph.com/issues/65200

@Matan-B
Copy link
Copy Markdown
Contributor Author

Matan-B commented Apr 1, 2024 •
edited
Loading

https://tracker.ceph.com/issues/65203
https://tracker.ceph.com/issues/65201
https://tracker.ceph.com/issues/65200

Should be addressed by #56611 and #56610, testing
https://shaman.ceph.com/builds/ceph/wip-matanb-crimson-testing-thrash-options

https://pulpito.ceph.com/matan-2024-04-01_13:06:33-crimson-rados-wip-matanb-crimson-testing-thrash-options-distro-crimson-smithi/

@Matan-B
Copy link
Copy Markdown
Contributor Author

Matan-B commented Apr 1, 2024

@athanatos, the previous trackers seem to be ok, new one popped out:

https://tracker.ceph.com/issues/65247

@Matan-B
Copy link
Copy Markdown
Contributor Author

Matan-B commented Apr 11, 2024

Updating relevant PRs:
#56611
#56610
#56832
#56806

CC: @xxhdx1985126, @athanatos

@Matan-B
qa/suites/crimson-rados/thrash: enable chance_down
3727517 
As the thrash tests were introduced, some options were disabled
until the tests are stabilized.

Re-enable chance_down option (default is 0.4) to detect bugs on restart.

Since it will probably take few iterations before thrash and recovery tests ('default.yaml')
will pass successfully, add anoter 'simple.yaml' which should remain stable.

Fixes: https://tracker.ceph.com/issues/65130

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
@Matan-B Matan-B force-pushed the wip-matanb-crimson-thrash-osd branch from 6db0718 to 3727517 Compare April 16, 2024 08:24
@Matan-B Matan-B requested a review from xxhdx1985126 April 16, 2024 08:28
@Matan-B
Copy link
Copy Markdown
Contributor Author

Matan-B commented Apr 16, 2024

I'll merge this based on:
https://pulpito.ceph.com/matan-2024-04-16_14:35:17-crimson-rados-main-distro-crimson-smithi/

@xuxuehan, we can start to test+merge the 'crimson-replicated-recovery' PRs based on the non-recovery thrash tests (as we did before) until both are stable.

@Matan-B
Copy link
Copy Markdown
Contributor Author

Matan-B commented Apr 16, 2024

jenkins test api

@Matan-B
Copy link
Copy Markdown
Contributor Author

Matan-B commented Apr 16, 2024

jenkins test make check

@athanatos
Copy link
Copy Markdown
Contributor

athanatos commented Apr 16, 2024

jenkins test api

@Matan-B Matan-B merged commit 469fae1 into ceph:main Apr 17, 2024
@Matan-B Matan-B mentioned this pull request May 16, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@athanatos athanatos athanatos approved these changes

@xxhdx1985126 xxhdx1985126 xxhdx1985126 approved these changes

Assignees

No one assigned

Labels

crimson crimson-replicated-recovery

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Matan-B @athanatos @xxhdx1985126