Skip to content

osd: optimize PG removal (part1)#37314

Merged
tchaikov merged 6 commits intoceph:masterfrom
ifed01:wip-ifed-faster-rm-p1
Dec 5, 2020
Merged

osd: optimize PG removal (part1)#37314
tchaikov merged 6 commits intoceph:masterfrom
ifed01:wip-ifed-faster-rm-p1

Conversation

@ifed01
Copy link
Contributor

@ifed01 ifed01 commented Sep 22, 2020 •
edited
Loading

This is the first set of patches intended to improve user experience while doing pool/PG removal.
Includes previous position reuse for collection listing, trimming removed object from bluestore cache (which speeds-up final collection reaping) plus some means to gather statistics on the removal process.

Fixes: https://tracker.ceph.com/issues/47044

Signed-off-by: Igor Fedotov ifedotov@suse.com

Checklist

  • References tracker ticket
  • Updates documentation if necessary
  • Includes tests for new functionality or reproducer for bug
Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox

@tchaikov

This comment has been minimized.

Sign in to view
@ifed01 ifed01 force-pushed the wip-ifed-faster-rm-p1 branch from 1560978 to e813f64 Compare September 23, 2020 11:08
@ifed01 ifed01 requested a review from a team as a code owner September 23, 2020 11:08
@ifed01 ifed01 force-pushed the wip-ifed-faster-rm-p1 branch from e813f64 to 172cf71 Compare September 23, 2020 12:33
@ifed01
Copy link
Contributor Author

ifed01 commented Sep 23, 2020

this change breaks crimson build, see

@tchaikov - fixed, thanks!

neha-ojha
neha-ojha reviewed Sep 25, 2020
View reviewed changes
Copy link
Member

@neha-ojha neha-ojha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ifed01 could you please attach your performance test results to this PR?

@tchaikov
Copy link
Contributor

tchaikov commented Sep 27, 2020

@ifed01
Copy link
Contributor Author

ifed01 commented Sep 29, 2020

@neha-ojha - here it is
https://docs.google.com/spreadsheets/d/1ngQA-x7Qpk0HARlkfZIOVFW8TGuAYhmhoW4cUdgmYh8/edit?usp=sharing

@ifed01 ifed01 force-pushed the wip-ifed-faster-rm-p1 branch from 172cf71 to 66317cd Compare September 29, 2020 18:04
@ifed01
Copy link
Contributor Author

ifed01 commented Sep 29, 2020

Here is another set of numbers for PG removal;
https://docs.google.com/spreadsheets/d/17V2mXUDEMAFVmSC67o1rQtrWNnAm_itBgzdxh1vRJMY/edit?usp=sharing

Columns B-E at "Overall" sheet cover both original performance and the changes from this PR.

@ifed01
Copy link
Contributor Author

ifed01 commented Sep 30, 2020

jenkins test classic perf

@ifed01 ifed01 mentioned this pull request Sep 30, 2020
Closed
3 tasks
@ifed01
Copy link
Contributor Author

ifed01 commented Sep 30, 2020

jenkins test classic perf

@ifed01 ifed01 force-pushed the wip-ifed-faster-rm-p1 branch from b4b4d3c to 66317cd Compare September 30, 2020 18:15
@ifed01
Copy link
Contributor Author

ifed01 commented Oct 1, 2020

jenkins test make check please

@ifed01
Copy link
Contributor Author

ifed01 commented Oct 1, 2020

jenkins test make check

1 similar comment
@ifed01
Copy link
Contributor Author

ifed01 commented Oct 1, 2020

jenkins test make check

@palza
Copy link

palza commented Oct 18, 2020

hi team
is there any update on completion of review and merging pr?
the most important for me is 30bfcf8

@tchaikov tchaikov self-requested a review October 21, 2020 03:08
neha-ojha
neha-ojha reviewed Oct 21, 2020
View reviewed changes
Copy link
Member

@neha-ojha neha-ojha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks good to me overall, @aclamk do you mind taking a look at the bluestore changes?
I think we should merge this PR first and keep the other improvements in #37496.

@tchaikov
Copy link
Contributor

tchaikov commented Oct 22, 2020

@ifed01 ifed01 force-pushed the wip-ifed-faster-rm-p1 branch from 66317cd to be18c4f Compare November 16, 2020 18:36
@ifed01 ifed01 force-pushed the wip-ifed-faster-rm-p1 branch from be18c4f to 9e78994 Compare November 16, 2020 18:47
@ifed01
Copy link
Contributor Author

ifed01 commented Nov 16, 2020

@aclamk - I've resolved most of your comments but trim-on-remove logics simplification. The latter seems not working to me, see my relevant comment...
So please give this PR another try. Thanks!

@tchaikov
Copy link
Contributor

tchaikov commented Nov 18, 2020

jenkins test make check

@ifed01 ifed01 force-pushed the wip-ifed-faster-rm-p1 branch from aaf780d to 9707e0a Compare November 19, 2020 18:51
@tchaikov tchaikov mentioned this pull request Dec 4, 2020
Closed
3 tasks
tchaikov
tchaikov reviewed Dec 4, 2020
View reviewed changes
tchaikov
tchaikov reviewed Dec 4, 2020
View reviewed changes
ifed01 added 6 commits December 5, 2020 00:09
@ifed01
osd/pg: use next when calling collection_list for pg removal
7f04700 
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
@ifed01
os/bluestore: perf counters for omap_get_keys[_get_values]
8684061 
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
@ifed01
os/bluestore: track kv utilization in kv_sync_thread
0504b95 
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
@ifed01
os/bluestore: introduce removal perf counter
cc9d584 
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
@ifed01
osd: adjust osd_delete_sleep_ssd[_hybrid]
6e09f0c 
1) Do not differentiate all-flash and hybrid(ssd+hdd) deployments since
they both bound to SSD drive backing DB volume.
2) Drop osd_delete_sleep_hybrid from 2s to 1s
3) Raise osd_delete_sleep_ssd to 1s since 0 is a way too low - having
it that low tend to cause a significant negative imact on OSD's regular
operation.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
@ifed01
doc/osd-config-ref: adjust default value for osd_delete_sleep_hybrid
ba9b547 
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
@ifed01 ifed01 force-pushed the wip-ifed-faster-rm-p1 branch from 9707e0a to ba9b547 Compare December 4, 2020 21:12
@github-actions github-actions bot added the common label Dec 4, 2020
@tchaikov
Copy link
Contributor

tchaikov commented Dec 5, 2020

@tchaikov tchaikov merged commit 4ae8abb into ceph:master Dec 5, 2020
@tchaikov
Copy link
Contributor

tchaikov commented Dec 5, 2020

@ifed01 shall we backport this change to, at least, octopus?

@dvanders
Copy link
Contributor

dvanders commented Dec 5, 2020

N too please. (We can take a stab at that if help is needed).

@ifed01 ifed01 deleted the wip-ifed-faster-rm-p1 branch December 7, 2020 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dvanders dvanders dvanders left review comments

@neha-ojha neha-ojha neha-ojha left review comments

@tchaikov tchaikov tchaikov approved these changes

@aclamk aclamk aclamk approved these changes

@markhpc markhpc Awaiting requested review from markhpc

@jdurgin jdurgin Awaiting requested review from jdurgin

Assignees

No one assigned

Labels

bluestore common core crimson documentation needs-qa performance wip-kefu-testing

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@ifed01 @tchaikov @palza @neha-ojha @dvanders @aclamk