os/bluestore: v.2 framework for more intelligent DB space usage by ifed01 · Pull Request #29687 · ceph/ceph

ifed01 · 2019-08-15T12:34:37Z

This is an evolution and simplification of #28960. It neither uses different naming model to learn sst file level nor requires some hints from RocksDB.
Instead it relies on existing wal/db/db.slow paths to track space usage per WAL/DB/SLOW 'levels' and WAL/DB/SLOW devices. And attempts to reuse extra DB space for slow "level" rather than put it unconditionally to main device. Depending on the current and previous utilization...
Similarly we can reuse WAL space for DB "level" if needed (not implemented).

Internal/RocksDB stats dump (Note 44GB of "slow" data put on DB volume for free:

bluestore(/home/if/ceph/build/dev/osd0)  bluefs bdev sizes: bluefs bdev sizes: bluefs bdev sizes:
0 : device size 0x140000000 : own 0x[1000\~13ffff000] = 0x13ffff000 : using 0x1f2ff000(499 MiB)
1 : device size 0x1900000000 : own 0x[2000\~18ffffe000] = 0x18ffffe000 : using 0x11dd8fe000(71 GiB)
2 : device size 0x1900000000 : own 0x[100000\~18efe00000,18f0000000\~4100000,18f4800000\~4100000,18f9000000\~4100000,18fd800000\~2800000] = 0x18fe900000 : using 0x18fb800000(100 GiB)

RocksDBBlueFSVolumeSelector: wal_total:5100273664, db_total:102005473280, slow_total:102005473280, db_avail:44291850240 usage matrix:
LEVEL, WAL, DB, SLOW, ****, ****, REAL
WAL 522190848,1048576,0,0,0,513338069
DB 0,31607226368,0,0,0,31461968188
SLOW 0,*44343230464*,107298684928,0,0,150927338859
UNSORTED 0,0,0,0,0,0
TOTALS 522190848,75951505408,107298684928,0,0,0
MAXIMUMS:
538968064,1048576,0,0,0,530301834
0,37870370816,0,0,0,37697560316
0,44343230464,107298684928,0,0,150927338859
0,0,0,0,0,0
538968064,79353085952,107298684928,0,0,0
db_statistics {
    "rocksdb_compaction_statistics": "",
    "": "",
    "": "** Compaction Stats [default] **",
    "": "Level    Files   Size     
    "": "----------------------------------------------------------------------------------------------------------------------------------------------------------------------------",
    "": "  L0      1/0   202.98 MB   
    "": "  L1      7/1   395.22 MB   
    "": "  L2     63/5    3.77 GB   
    "": "  L3    435/1   24.57 GB 
    "": "  L4   2246/0   140.56 GB 
    "": " Sum   2752/7   169.48 GB

Another sample - 32GB of L4 data (originally targeted to slow device) at DB volume:

bluestore(/home/if/ceph/build/dev/osd0)  bluefs bdev sizes: bluefs bdev sizes: bluefs bdev sizes:
0 : device size 0x140000000 : own 0x[1000\~13ffff000] = 0x13ffff000 : using 0x147ff000(328 MiB)
1 : device size 0x1e00000000 : own 0x[2000\~18ffffe000] = 0x18ffffe000 : using 0x7d58fe000(31 GiB)
2 : device size 0x1900000000 : own ... = 0x1675400000 : using 0x109100000(4.1 GiB)

RocksDBBlueFSVolumeSelector: wal_total:5100273664, db_total:122406567936, slow_total:102005473280, db_avail:64692944896 usage matrix:
LEVEL, WAL, DB, SLOW, ****, ****, REAL
WAL 342884352,1048576,0,0,0,338348352
DB 0,370147328,0,0,0,353936829
SLOW 0,*32027705344*,3994025984,0,0,35833722268
UNSORTED 0,0,0,0,0,0
TOTALS 342884352,32398901248,3994025984,0,0,0
MAXIMUMS:
522190848,1048576,0,0,0,513350357
0,33274462208,0,0,0,33122793939
0,64760053760,107298684928,0,0,168613750477
0,0,0,0,0,0
522190848,77617692672,107298684928,0,0,0
db_statistics {
    "rocksdb_compaction_statistics": "",
    "": "",
    "": "** Compaction Stats [default] **",
    "": "Level    Files   Size
    "": "--------------------
    "": "  L0      0/0    0.00 KB   
    "": "  L1      0/0    0.00 KB       
    "": "  L2      0/0    0.00 KB
    "": "  L3     21/0   336.46 MB   
    "": "  L4    556/0   33.37 GB   
    "": " Sum    577/0   33.70 GB

Signed-off-by: Igor Fedotov ifedotov@suse.com

Checklist

References tracker ticket
Updates documentation if necessary
Includes tests for new functionality or reproducer for bug

Show available Jenkins commands

jenkins retest this please
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test docs
jenkins render docs

liewegas

I like this. It seems like even if we had the explicit hinting, we'd need the 2d map for usage tracking to know where to put things--these are mostly orthogonal parts of the problem.

Can we add some basic tests the verify that we end up putting some of the higher-level data on db?

liewegas · 2019-08-23T15:09:04Z

src/os/bluestore/BlueFS.cc

    if (prefer_bdev != dev_target && devs_source.count(prefer_bdev)) {
      prefer_bdev = dev_target_new;
-    }
+    }*/


remove this?

liewegas · 2019-08-23T15:09:13Z

src/os/bluestore/BlueFS.cc

    if (prefer_bdev != dev_target && devs_source.count(prefer_bdev)) {
      prefer_bdev = dev_target_new;
-    }
+    }*/


liewegas · 2019-08-23T15:44:36Z

src/kv/RocksDBStore.cc

  return 0;
 }

+int RocksDBStore::ParseOptionsFromStringStatic(


instead of duplicating the function above, can we call the static version from the non-static, or just use the static one exclusively?

liewegas · 2019-08-23T15:45:02Z

src/os/bluestore/BlueFS.cc

-    block_all(MAX_BDEV)
+  bdev(MAX_BDEV),
+  ioc(MAX_BDEV),
+  block_all(MAX_BDEV)


this should stay aligned with cct(cct)?

liewegas · 2019-08-23T15:48:54Z

src/common/options.cc

+    Option("bluestore_volume_selection_reserved_factor", Option::TYPE_FLOAT, Option::LEVEL_ADVANCED)
+      .set_flag(Option::FLAG_STARTUP)
+      .set_default(2.0)
+      .set_description("Max DB level size multiplier. Togather they determine amount of space to preserve from usage due to 'use some extra' policy"),


s/togather/together/

Do I understand correctly that this means that 2x of the estimated DB level(s) size will be reserved, and then what is left over after that will be used for subsequent levels?

Right. And additional cap is maximum amount of data previously observed at DB for DB/WAL levels. So the actual reserved size is max of both.

Are there situations where this reservation might actually force a level to slow storage because there is no longer space on a faster device?

@anthonyeleven - theoretically yes. The 2x multiplier is a bit speculative and doesn't have any strong rationales behind. So in the worst scenario some lower level data can go to slow device while higher level one are at fast device. Don't think this is highly likely though...

liewegas · 2019-08-23T15:49:51Z

src/os/bluestore/BlueFS.cc

 	  << dendl;

+  // set volume selector
+  if (vselector == nullptr) {


wouldn't it be simpler/safer to reset(nullptr) in umount and at end of mkfs, and assert vselector is null before you instantiate it here (and below in mount())?

liewegas · 2019-08-23T15:50:20Z

src/os/bluestore/BlueFS.h

  virtual uint8_t select_prefer_bdev(void* hint) = 0;
  virtual void get_paths(const std::string& base, paths& res) const = 0;
  virtual void dump(CephContext* cct) = 0;
+  virtual void restart() = 0;


then you wouldn't need restart()...

ifed01 · 2019-08-27T10:14:39Z

I like this. It seems like even if we had the explicit hinting, we'd need the 2d map for usage tracking to know where to put things--these are mostly orthogonal parts of the problem.
Correct

Can we add some basic tests the verify that we end up putting some of the higher-level data on db?

Will do.

ifed01 · 2019-08-30T16:43:20Z

@liewegas - done, please review.

ifed01 · 2019-09-27T10:02:06Z

jenkins retest this please

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

…r BlueFS. It allows excessive space usage for higher DB levels. Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

This is an alternative (and straightforward) way to specify barred space not allowed for 'use some extra' BlueFS volume selector policy. Disabled by default since it should depend on RocksDB settings and actual volume size. Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

ifed01 · 2019-11-26T21:35:05Z

@liewegas - ping? any barriers for this to go?

liewegas

yay!

tchaikov · 2019-12-05T07:49:34Z

http://pulpito.ceph.com/kchai-2019-12-02_17:51:44-rados-wip-kefu-testing-2019-12-02-2333-distro-basic-mira/

failures tracked by

i suspect that https://tracker.ceph.com/issues/43131 was caused by this PR.

@ifed01 could you help take a look?

liewegas · 2019-12-08T21:23:43Z

opened https://tracker.ceph.com/issues/43189 for the hung pgs.. it occurs on master too, so not related to this PR. I think this can merge!

* refs/pull/29687/head: test/store_test: test coverage for anti-spillover framework. os/bluestore: introduce bluestore_volume_selection_reserved cfg option. os/bluestore: beautify RocksDBBlueFSVolumeSelector::dump output os/bluestore: introduce 'bluestore bluefs stats' admin socket command os/bluestore: implement more intelligent DB volume space managment for BlueFS. os/kv: add static method to parse RocksDB options os/bluestore: introduce volume selector abstraction to BlueFS Reviewed-by: Sage Weil <sage@redhat.com>

It turned out that we did't enable it when introduced this feature in ceph#29687. Fixes: https://tracker.ceph.com/issues/47053 Signed-off-by: Igor Fedotov <ifedotov@suse.com>

It turned out that we did't enable it when introduced this feature in ceph#29687. Fixes: https://tracker.ceph.com/issues/47053 Signed-off-by: Igor Fedotov <ifedotov@suse.com> (cherry picked from commit 41823db)

It turned out that we did't enable it when introduced this feature in ceph#29687. Fixes: https://tracker.ceph.com/issues/47053 Signed-off-by: Igor Fedotov <ifedotov@suse.com>

It turned out that we did't enable it when introduced this feature in ceph#29687. Fixes: https://tracker.ceph.com/issues/47053 Signed-off-by: Igor Fedotov <ifedotov@suse.com> (cherry picked from commit 41823db) (cherry picked from commit 6d37c6a) Resolves: rhbz#1877910

It turned out that we did't enable it when introduced this feature in ceph#29687. Fixes: https://tracker.ceph.com/issues/47053 Signed-off-by: Igor Fedotov <ifedotov@suse.com> (cherry picked from commit 41823db)

markhpc added bluestore core performance labels Aug 15, 2019

ifed01 force-pushed the wip-ifed-flex-bluefs-size-simplified branch from f237499 to 31ea145 Compare August 15, 2019 14:08

ifed01 requested review from liewegas and markhpc August 15, 2019 14:09

ifed01 force-pushed the wip-ifed-flex-bluefs-size-simplified branch 2 times, most recently from 7f22ce9 to 69e47d9 Compare August 16, 2019 14:42

ifed01 mentioned this pull request Aug 16, 2019

[RFC][WIP]os/bluestore: framework for more intelligent DB space usage #28960

Closed

3 tasks

ifed01 changed the title ~~[DNM][RFC] os/bluestore: os/bluestore: v.2 framework for more intelligent DB space usage~~ os/bluestore: os/bluestore: v.2 framework for more intelligent DB space usage Aug 19, 2019

ifed01 changed the title ~~os/bluestore: os/bluestore: v.2 framework for more intelligent DB space usage~~ os/bluestore: v.2 framework for more intelligent DB space usage Aug 19, 2019

liewegas reviewed Aug 23, 2019

View reviewed changes

ifed01 force-pushed the wip-ifed-flex-bluefs-size-simplified branch 2 times, most recently from 5aad709 to f4ffbfd Compare August 30, 2019 16:32

ifed01 force-pushed the wip-ifed-flex-bluefs-size-simplified branch from f4ffbfd to 97d7972 Compare September 9, 2019 17:36

ifed01 force-pushed the wip-ifed-flex-bluefs-size-simplified branch from 97d7972 to 44765db Compare September 26, 2019 15:50

ifed01 force-pushed the wip-ifed-flex-bluefs-size-simplified branch from 44765db to c2cddbe Compare October 31, 2019 12:06

ifed01 added 7 commits November 26, 2019 21:28

os/bluestore: introduce volume selector abstraction to BlueFS

012c5c3

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/kv: add static method to parse RocksDB options

518faf4

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: implement more intelligent DB volume space managment fo…

5f72c37

…r BlueFS. It allows excessive space usage for higher DB levels. Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: introduce 'bluestore bluefs stats' admin socket command

f5568eb

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: beautify RocksDBBlueFSVolumeSelector::dump output

fc84588

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

test/store_test: test coverage for anti-spillover framework.

57c6a79

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

ifed01 force-pushed the wip-ifed-flex-bluefs-size-simplified branch from c2cddbe to 57c6a79 Compare November 26, 2019 19:51

liewegas added the needs-qa label Nov 26, 2019

liewegas approved these changes Nov 26, 2019

View reviewed changes

tchaikov added the wip-kefu-testing label Dec 2, 2019

tchaikov removed the wip-kefu-testing label Dec 5, 2019

liewegas added the wip-sage3-testing label Dec 5, 2019

liewegas merged commit 57c6a79 into ceph:master Dec 8, 2019

ifed01 deleted the wip-ifed-flex-bluefs-size-simplified branch April 1, 2020 10:52

ifed01 mentioned this pull request Aug 21, 2020

os/bluestore: enable more flexible bluefs space management by default. #36750

Merged

3 tasks

anthonyeleven mentioned this pull request Dec 18, 2020

doc/rados/configuration: clean up BlueStore config and enhance DB sizing #38621

Merged

3 tasks

Conversation

ifed01 commented Aug 15, 2019 • edited by tchaikov Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

liewegas left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anthonyeleven Oct 1, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ifed01 commented Aug 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ifed01 commented Aug 30, 2019

Uh oh!

ifed01 commented Sep 27, 2019

Uh oh!

ifed01 commented Nov 26, 2019

Uh oh!

liewegas left a comment

Choose a reason for hiding this comment

Uh oh!

tchaikov commented Dec 5, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liewegas commented Dec 8, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ifed01 commented Aug 15, 2019 •

edited by tchaikov

Loading

anthonyeleven Oct 1, 2019 •

edited

Loading

ifed01 commented Aug 27, 2019 •

edited

Loading

tchaikov commented Dec 5, 2019 •

edited

Loading