os/bluestore: improve cache_onode and cache_buffer mempool accounting by ideepika · Pull Request #40066 · ceph/ceph

ideepika · 2021-03-12T13:12:55Z

move physical extent to mempool bluestore_extent from cache_other
move bluestore_shared_blob_t from cache_other to shared_blob mempool

Signed-off-by: Deepika Upadhyay dupadhya@redhat.com

Checklist

References tracker ticket
Updates documentation if necessary
Includes tests for new functionality or reproducer for bug

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox

aclamk

Nice stuff.

src/os/bluestore/bluestore_types.h

doc/dev/mempool_accounting.rst

tchaikov · 2021-03-25T02:50:45Z

What is a mempool?
-----------------
/home/jenkins-build/build/workspace/ceph-pr-docs/doc/dev/mempool_accounting.rst:52: WARNING: Unknown directive type "see_more".

.. see_more:: https://github.com/ceph/ceph/blob/fccbdc0905e3868666fbb10803bac6b73f687cb1/src/include/mempool.h

tchaikov


/home/jenkins-build/build/workspace/ceph-pr-docs/doc/dev/mempool_accounting.rst:32: WARNING: Unexpected indentation.
looking for now-outdated files... none found
pickling environment... done

tchaikov · 2021-03-26T06:36:09Z

dismissed

ideepika · 2021-04-06T14:08:56Z

@tchaikov @neha-ojha should we merge this?

doc/dev/mempool_accounting.rst

neha-ojha · 2021-04-07T00:31:04Z

@tchaikov @neha-ojha should we merge this?

barring one nit, looks good to go with tests passing

ifed01 · 2022-09-12T09:48:12Z

jenkins test make check

ceph.spec.in

debian/control

src/os/bluestore/BlueStore.h

src/os/bluestore/BlueStore.cc

kamoltat · 2022-09-30T14:43:13Z

@ideepika please confirm if this PR caused this failure
/a/yuriw-2022-09-27_23:37:28-rados-wip-yuri2-testing-2022-09-27-1455-distro-default-smithi/7046379/

2022-09-28T07:11:23.610 INFO:teuthology.orchestra.run.smithi050.stdout:[  FAILED  ] ObjectStore/StoreTestSpecificAUSize.OnodeSizeTracking/2, where GetParam() = "bluestore"
2022-09-28T07:11:23.611 INFO:teuthology.orchestra.run.smithi050.stdout:[  FAILED  ] ObjectStore/StoreTestSpecificAUSize.BlobReuseOnOverwrite/2, where GetParam() = "bluestore"
2022-09-28T07:11:23.611 INFO:teuthology.orchestra.run.smithi050.stdout:
2022-09-28T07:11:23.611 INFO:teuthology.orchestra.run.smithi050.stdout: 2 FAILED TESTS
2022-09-28T07:11:23.619 DEBUG:teuthology.orchestra.run:got remote process result: 1
2022-09-28T07:11:23.621 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/run_tasks.py", line 103, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/run_tasks.py", line 82, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/task/exec.py", line 72, in task
    c],
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/orchestra/remote.py", line 525, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/orchestra/run.py", line 183, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi050 with status 1: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'mkdir $TESTDIR/archive/ostest && cd $TESTDIR/archive/ostest && ulimit -Sn 16384 && CEPH_ARGS="--no-log-to-stderr --log-file $TESTDIR/archive/ceph_test_objectstore.log --debug-bluestore 20" ceph_test_objectstore --gtest_filter=*/2:-*SyntheticMatrixC* --gtest_catch_exceptions=0\''
2022-09-28T07:11:23.801 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4f2e242dc2fa43f381e7e73370ef102f
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/run_tasks.py", line 103, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/run_tasks.py", line 82, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/task/exec.py", line 72, in task
    c],
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_a773b0612ada07695bef890b85208fa310b79d30/teuthology/orchestra/remote.py", line 525, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)

ideepika · 2022-09-30T14:53:51Z

@kamoltat looks related to me, i will test and work on a fix for this during the weekend, thanks for the mention!

kamoltat · 2022-09-30T18:32:06Z

@kamoltat looks related to me, i will test and work on a fix for this during the weekend, thanks for the mention!

Thank you, Deepika 😊

ideepika · 2022-10-03T17:08:03Z

sorry, have to delay on this a bit, i will try to finish by this week, thanks!

ideepika · 2022-10-10T07:04:26Z

@kamoltat fixed the failures: http://qa-proxy.ceph.com/teuthology/ideepika-2022-10-09_18:43:32-rados:objectstore-wip-mempool-buffers-distro-default-smithi/7059889/teuthology.log

ifed01 · 2022-10-10T08:57:53Z

src/test/objectstore/store_test.cc

    ASSERT_TRUE(bl_eq(expected, bl));
    ASSERT_EQ(logger->get(l_bluestore_blobs), 1u);
    ASSERT_EQ(logger->get(l_bluestore_extents), 2u);
+    ASSERT_EQ(mempool::bluestore_blob::allocated_items(), 1u);


@ideepika - do you have any ideas how this modification could fix the following issue @kamoltat shared before:
g057e8043/rpm/el8/BUILD/ceph-18.0.0-88-g057e8043/src/test/objectstore/store_test.cc:7165: Failure
2022-09-28T06:47:07.984 INFO:teuthology.orchestra.run.smithi050.stdout:Expected equality of these values:
2022-09-28T06:47:07.984 INFO:teuthology.orchestra.run.smithi050.stdout: mempool::bluestore_blob::allocated_items()
2022-09-28T06:47:07.984 INFO:teuthology.orchestra.run.smithi050.stdout: Which is: 9
2022-09-28T06:47:07.985 INFO:teuthology.orchestra.run.smithi050.stdout: 1u
2022-09-28T06:47:07.985 INFO:teuthology.orchestra.run.smithi050.stdout: Which is: 1
2022-09-28T06:47:08.498 INFO:teuthology.orchestra.run.smithi050.stdout:==> rm -r bluestore.test_temp_dir

To me this rather looks like some volatile behavior which causes mempool stats to vary. Wouldn't be more reliable to get rid of these stats instead?

I think it is due to onode_cache_shard vector which is where we are accounting for onodes in this PR
I checked as well this seem to be true:
https://github.com/ceph/ceph/pull/40066/files#diff-157b0c4b175141aba18fe37a72d49ca9175e01f69a266595d495df86b5240e86R7010
@ifed01 is there a way we can shift this accounting to some point later?

@ideepika - not sure I understand your point/question :( The above error was referring to bluestore_blob mempool. onode_cache_shard looks irrelevant in this case...
Anyway my suggestion is just not introduce these two new assertions for bluestore_blob and
bluestore_extent mempools. And hence avoid this unclear failure which we don't know root cause for yet.

sure, updated the changes

* move physical extent to mempool bluestore_extent from cache_other * move bluestore_shared_blob_t from cache_other to shared_blob mempool Signed-off-by: Deepika Upadhyay <deepika@koor.tech>

ideepika · 2022-11-08T11:16:53Z

jenkins make check arm64

ideepika · 2022-11-08T11:20:18Z

jenkins test make check arm64

ljflores · 2022-12-13T19:18:23Z

Rados suite review:

https://pulpito.ceph.com/?branch=wip-yuri-testing-2022-12-06-1204
https://pulpito.ceph.com/?branch=wip-yuri-testing-2022-12-12-1136

Failures, unrelated:
1. https://tracker.ceph.com/issues/58096
2. https://tracker.ceph.com/issues/52321
3. https://tracker.ceph.com/issues/58173
4. https://tracker.ceph.com/issues/52129
5. https://tracker.ceph.com/issues/58097
6. https://tracker.ceph.com/issues/57546
7. https://tracker.ceph.com/issues/58098
8. https://tracker.ceph.com/issues/57731
9. https://tracker.ceph.com/issues/55606
10. https://tracker.ceph.com/issues/58256
11. https://tracker.ceph.com/issues/58258

Details:
1. test_cluster_set_reset_user_config: NFS mount fails due to missing ceph directory - Ceph - Orchestrator
2. qa/tasks/rook times out: 'check osd count' reached maximum tries (90) after waiting for 900 seconds - Ceph - Orchestrator
3. api_aio_pp: failure on LibRadosAio.SimplePoolEIOFlag and LibRadosAio.PoolEIOFlag - Ceph - RADOS
4. LibRadosWatchNotify.AioWatchDelete failed - Ceph - RADOS
5. qa/workunits/post-file.sh: kex_exchange_identification: read: Connection reset by peer - Ceph - RADOS
6. rook: ensure CRDs are installed first - Ceph - Orchestrator
7. qa/workunits/rados/test_crash.sh: crashes are never posted - Ceph - RADOS
8. Problem: package container-selinux conflicts with udica < 0.2.6-1 provided by udica-0.2.4-1 - Infrastructure
9. [ERR] Unhandled exception from module ''devicehealth'' while running on mgr.y: unknown - Ceph - CephSqlite
10. ObjectStore/StoreTestSpecificAUSize.SpilloverTest/2: Expected: (logger->get(l_bluefs_slow_used_bytes)) >= (16 * 1024 * 1024), actual: 0 vs 16777216 - Ceph - Bluestore
11. rook: kubelet fails from connection refused - Ceph - Orchestrator

github-actions bot added bluestore core labels Mar 12, 2021

ideepika requested a review from aclamk March 12, 2021 13:13

aclamk approved these changes Mar 12, 2021

View reviewed changes

ifed01 reviewed Mar 12, 2021

View reviewed changes

src/os/bluestore/bluestore_types.h Outdated Show resolved Hide resolved

ideepika force-pushed the wip-mempool-buffers branch from 8b053cb to 1085a31 Compare March 19, 2021 14:34

github-actions bot added the documentation label Mar 19, 2021

ifed01 approved these changes Mar 19, 2021

View reviewed changes

doc/dev/mempool_accounting.rst Outdated Show resolved Hide resolved

neha-ojha reviewed Mar 19, 2021

View reviewed changes

doc/dev/mempool_accounting.rst Outdated Show resolved Hide resolved

neha-ojha reviewed Mar 19, 2021

View reviewed changes

doc/dev/mempool_accounting.rst Outdated Show resolved Hide resolved

ideepika force-pushed the wip-mempool-buffers branch from 27c1d56 to 15c7c87 Compare March 22, 2021 13:19

ideepika requested a review from neha-ojha March 22, 2021 13:19

ideepika added the needs-qa label Mar 22, 2021

tchaikov added the wip-kefu-testing label Mar 25, 2021

ideepika force-pushed the wip-mempool-buffers branch from 15c7c87 to b377542 Compare March 25, 2021 10:10

tchaikov previously requested changes Mar 26, 2021

View reviewed changes

tchaikov removed wip-kefu-testing needs-qa labels Mar 26, 2021

ideepika force-pushed the wip-mempool-buffers branch 3 times, most recently from f1db451 to f5ce81f Compare March 29, 2021 10:21

ideepika requested a review from tchaikov March 30, 2021 09:57

neha-ojha reviewed Apr 7, 2021

View reviewed changes

doc/dev/mempool_accounting.rst Outdated Show resolved Hide resolved

ideepika force-pushed the wip-mempool-buffers branch 2 times, most recently from 92e719f to a57b0cb Compare April 7, 2021 11:30

ifed01 reviewed Sep 12, 2022

View reviewed changes

ceph.spec.in Outdated Show resolved Hide resolved

ifed01 reviewed Sep 12, 2022

View reviewed changes

debian/control Outdated Show resolved Hide resolved

ifed01 reviewed Sep 12, 2022

View reviewed changes

src/os/bluestore/BlueStore.h Outdated Show resolved Hide resolved

ideepika force-pushed the wip-mempool-buffers branch from 0b3d9b2 to 7e987d1 Compare September 12, 2022 09:55

ifed01 reviewed Sep 12, 2022

View reviewed changes

src/os/bluestore/BlueStore.cc Outdated Show resolved Hide resolved

ideepika force-pushed the wip-mempool-buffers branch from 7e987d1 to 6c88d9a Compare September 22, 2022 12:15

ifed01 added the needs-qa label Sep 22, 2022

ljflores added the wip-yuri2-testing label Sep 27, 2022

yuriw removed the wip-yuri2-testing label Sep 30, 2022

This was referenced Oct 3, 2022

mon/OSDMonitor: Added extra check before mon.go_recovery_stretch_mode() #47340

Merged

osd: break multi condition asserts into multiple asserts #48150

Merged

mgr/telemetry: fixed log exceptions as "exception" instead of "error" #48152

Merged

ideepika force-pushed the wip-mempool-buffers branch 3 times, most recently from 3c81399 to 7d82664 Compare October 10, 2022 07:02

ideepika requested a review from ifed01 October 10, 2022 07:03

ifed01 reviewed Oct 10, 2022

View reviewed changes

os/bluestore: improve cache_onode and cache_buffer mempool accounting

0bf3519

* move physical extent to mempool bluestore_extent from cache_other * move bluestore_shared_blob_t from cache_other to shared_blob mempool Signed-off-by: Deepika Upadhyay <deepika@koor.tech>

ideepika force-pushed the wip-mempool-buffers branch from 7d82664 to 0bf3519 Compare November 1, 2022 07:03

ljflores added the wip-yuri-testing label Dec 6, 2022

yuriw merged commit 4c3212d into ceph:main Dec 15, 2022

Conversation

ideepika commented Mar 12, 2021

Checklist

Uh oh!

aclamk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tchaikov commented Mar 25, 2021

Uh oh!

tchaikov left a comment

Choose a reason for hiding this comment

Uh oh!

tchaikov commented Mar 26, 2021

Uh oh!

ideepika commented Apr 6, 2021

Uh oh!

Uh oh!

neha-ojha commented Apr 7, 2021

Uh oh!

ifed01 commented Sep 12, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kamoltat commented Sep 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ideepika commented Sep 30, 2022

Uh oh!

kamoltat commented Sep 30, 2022

Uh oh!

ideepika commented Oct 3, 2022

Uh oh!

ideepika commented Oct 10, 2022

Uh oh!

ifed01 Oct 10, 2022

Choose a reason for hiding this comment

Uh oh!

ideepika Oct 22, 2022

Choose a reason for hiding this comment

Uh oh!

ifed01 Oct 31, 2022

Choose a reason for hiding this comment

Uh oh!

ideepika Nov 1, 2022

Choose a reason for hiding this comment

Uh oh!

ideepika commented Nov 8, 2022

Uh oh!

ideepika commented Nov 8, 2022

Uh oh!

ljflores commented Dec 13, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

kamoltat commented Sep 30, 2022 •

edited

Loading