Skip to content

osd/OSD: rewrite track_pools_and_pg_num_changes logic#54988

Merged
yuriw merged 5 commits intoceph:mainfrom
Matan-B:wip-matanb-mapgap-fix
Jul 18, 2024
Merged

osd/OSD: rewrite track_pools_and_pg_num_changes logic#54988
yuriw merged 5 commits intoceph:mainfrom
Matan-B:wip-matanb-mapgap-fix

Conversation

@Matan-B
Copy link
Contributor

@Matan-B Matan-B commented Dec 21, 2023

  • track_pools_and_pg_num_changes() is moved into a separate method

  • track_pools_and_pg_num_changes() inner logic is moved into a separate method

  • Map gap case is now properly handled, last_map should start with the newest map we have.
    Fixes: https://tracker.ceph.com/issues/63881

    Example fix usage after letting osd.2 hit a map gap while creating 2 pools and changing pg_num in the meanwhile:

    2023-12-21T14:23:22.252+0000 7faea797f700 10 osd.2 23 track_pools_and_pg_num_changes can't get previous map 122 first start of this osd after a map gap
    2023-12-21T14:23:22.252+0000 7faea797f700 10 osd.2 23 track_pools_and_pg_num_changes recording pool 2 pg_num 1 -> 3
    2023-12-21T14:23:22.252+0000 7faea797f700 10 osd.2 23 track_pools_and_pg_num_changes recording new pool 3 pg_num 1
    2023-12-21T14:23:22.252+0000 7faea797f700 10 osd.2 23 track_pools_and_pg_num_changes recording new pool 4 pg_num 1
    

Contribution Guidelines

  • To sign and title your commits, please refer to Submitting Patches to Ceph.

  • If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

  • When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

  • Tracker (select at least one)
    • References tracker ticket
    • Very recent bug; references commit where it was introduced
    • New feature (ticket optional)
    • Doc update (no ticket needed)
    • Code cleanup (no ticket needed)
  • Component impact
    • Affects Dashboard, opened tracker ticket
    • Affects Orchestrator, opened tracker ticket
    • No impact that needs to be tracked
  • Documentation (select at least one)
    • Updates relevant documentation
    • No doc update is appropriate
  • Tests (select at least one)
Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard cephadm
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox
  • jenkins test windows
  • jenkins test rook e2e

@Matan-B Matan-B requested a review from athanatos December 21, 2023 14:25
@Matan-B Matan-B changed the title [WIP] osd/OSD: fix inaccurate pg_num_history on mapgaps [WIP] osd/OSD: rewrite track_pools_and_pg_num_changes logic Dec 27, 2023
@Matan-B Matan-B force-pushed the wip-matanb-mapgap-fix branch from 9004e95 to a320579 Compare December 27, 2023 14:50
@athanatos
Copy link
Contributor

I think it would be helpful to add a comment to pg_num_history listing how it's used.

@github-actions
Copy link

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

@github-actions github-actions bot added the stale label Mar 22, 2024
@Matan-B Matan-B force-pushed the wip-matanb-mapgap-fix branch from a320579 to 04380c6 Compare March 25, 2024 13:32
@Matan-B Matan-B force-pushed the wip-matanb-mapgap-fix branch from 04380c6 to e36a60f Compare March 25, 2024 13:33
@Matan-B Matan-B changed the title [WIP] osd/OSD: rewrite track_pools_and_pg_num_changes logic osd/OSD: rewrite track_pools_and_pg_num_changes logic Mar 25, 2024
@Matan-B Matan-B removed the stale label Mar 25, 2024
@Matan-B Matan-B marked this pull request as ready for review March 25, 2024 13:45
@Matan-B Matan-B requested a review from a team as a code owner March 25, 2024 13:45
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
@Matan-B Matan-B force-pushed the wip-matanb-mapgap-fix branch from e36a60f to d0baf4f Compare March 26, 2024 14:24
Copy link
Contributor

@athanatos athanatos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure it's actually ok for pg_num_history to have inaccurate epochs associated with the pg creation, removal, and pg num change events. A comment explaining why this is safe would help.

Refactor, no change in behavior

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
@Matan-B Matan-B force-pushed the wip-matanb-mapgap-fix branch 2 times, most recently from f033797 to ddb1657 Compare March 27, 2024 11:20
@Matan-B Matan-B requested a review from athanatos March 27, 2024 11:22
@Matan-B Matan-B force-pushed the wip-matanb-mapgap-fix branch from ddb1657 to 8ebf51e Compare March 27, 2024 12:47
@Matan-B
Copy link
Contributor Author

Matan-B commented Mar 27, 2024

@athanatos, probably worth mentioning that I suspect the mapgap issue can cause random errors much later on such as https://tracker.ceph.com/issues/57628, where the OSD would start and load the pgs as exited before the gap (even if those no longer exist). I can't verify the specific tracker since the logs are out of date.

@Matan-B Matan-B force-pushed the wip-matanb-mapgap-fix branch from 8ebf51e to 20ae911 Compare March 31, 2024 12:03
* track_pools_and_pg_num_changes call is moved before the superblock updates to
  know if we have any OSDMaps *before* trimming them (if possible).
  This is used to identify map gaps.

* track_pools_and_pg_num_changes inner loop is moved into
  _track_pools_and_pg_num_changes.

* lastmap now starts with the newest_map we have, even on mapgap.
  Previously, on mapgaps, we would (falsly) assume that this is the first start
  of this OSD and would ignore the state before the gap. This case
  is fully described in the tracker below:

  Fixes: https://tracker.ceph.com/issues/63881

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
@Matan-B Matan-B force-pushed the wip-matanb-mapgap-fix branch from 20ae911 to f283458 Compare March 31, 2024 12:26
Avoid p,q usage

Signed-off-by: Matan Breizman <mbreizma@redhat.com>
@Matan-B Matan-B force-pushed the wip-matanb-mapgap-fix branch from f283458 to 78d7668 Compare March 31, 2024 12:33
@Matan-B Matan-B requested a review from athanatos March 31, 2024 12:35
@Matan-B Matan-B removed the crimson label Mar 31, 2024
@Matan-B Matan-B requested a review from NitzanMordhai April 4, 2024 08:44
@athanatos
Copy link
Contributor

jenkins test api

@yuriw
Copy link
Contributor

yuriw commented May 3, 2024

This PR is under test in https://tracker.ceph.com/issues/65796.

@github-actions
Copy link

github-actions bot commented Jul 2, 2024

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

@github-actions github-actions bot added the stale label Jul 2, 2024
@Matan-B Matan-B removed the stale label Jul 3, 2024
@kamoltat
Copy link
Member

@yuriw yuriw merged commit 25fb8aa into ceph:main Jul 18, 2024
NitzanMordhai pushed a commit to NitzanMordhai/ceph that referenced this pull request Aug 1, 2024
osd/OSD: rewrite track_pools_and_pg_num_changes logic

Reviewed-by: Samuel Just <sjust@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants