[RFC] protect against OOM killer by gollub · Pull Request #264 · ceph/ceph

gollub · 2013-05-08T14:08:31Z

This is more a RFC then a pull request. Maybe worth a blueprint - but maybe trivial enough to just get merged. A colleague of mine mentioned that now and then osd processes got killed by the OOM killer. So I can up with the idea improve the chances of osd and mon to survive on an OOM situation.

Comments and thoughts about this are welcome.

Introduce also handling for legacy OOM adjust which has different OOM disable value. Signed-off-by: Daniel Gollub <d.gollub@telekom.de>

Signed-off-by: Daniel Gollub <d.gollub@telekom.de>

Be nice and give OOM killer a chance to kill the osd. Chose the middle OOM score for now. Other services like udev or lvm go with disabling the OOM killer for them. No further reason and experience to choose lower or higher OOM score value. Signed-off-by: Daniel Gollub <d.gollub@telekom.de>

This allows to reuse the code for other daemons again. Signed-off-by: Daniel Gollub <d.gollub@telekom.de>

Signed-off-by: Daniel Gollub <d.gollub@telekom.de>

This implementation is not disabling OOM killer for the mon completelty. It just reduce that the montior get killed first on an OOM situation. Signed-off-by: Daniel Gollub <d.gollub@telekom.de>

liewegas · 2013-05-08T18:13:57Z

src/global/global_init.h

we shoud make these tunables in common/config_opts.h.. oom_* maybe?

liewegas · 2013-05-08T18:17:23Z

with config tunables, i think this is fine. it would be nice to have some docs on what adjusting the knobs actually does, though.. it's not obvious if/when/why users should do this.

dalgaaf · 2013-05-10T09:49:54Z

@gollub: I will take a look at the config part

tasks/watch_notify_same_primary: fix test for new watch/notify Reviewed-by: Samuel Just <sjust@redhat.com>

Fix broken asset link

…eph#264) Problem: While addding the new null version in suspended/unversioned bucket deleting the previous null object from bucket index only. Solution: Delete object from motr store as well as from the bucket index. Signed-off-by: Shriya Deshmukh <shriya.deshmukh@seagate.com>

seeing this run-tox-qa failure about tasks/cephfs/test_client_recovery.py: 246/285 Test ceph#264: run-tox-qa ................................***Failed 58.54 sec Requirement already satisfied: tox in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (4.6.4) Requirement already satisfied: cachetools>=5.3.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (5.3.1) Requirement already satisfied: chardet>=5.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (5.1.0) Requirement already satisfied: colorama>=0.4.6 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (0.4.6) Requirement already satisfied: filelock>=3.12.2 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (3.12.2) Requirement already satisfied: packaging>=23.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (23.1) Requirement already satisfied: platformdirs>=3.8 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (3.10.0) Requirement already satisfied: pluggy>=1.2 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (1.2.0) Requirement already satisfied: pyproject-api>=1.5.2 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (1.5.3) Requirement already satisfied: tomli>=2.0.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (2.0.1) Requirement already satisfied: virtualenv>=20.23.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (20.24.2) Requirement already satisfied: distlib<1,>=0.3.7 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from virtualenv>=20.23.1->tox) (0.3.7) flake8: install_deps /home/jenkins-build/build/workspace/ceph-pull-requests/qa> python -I -m pip install flake8 flake8: freeze /home/jenkins-build/build/workspace/ceph-pull-requests/qa> python -m pip freeze --all flake8: flake8==6.1.0,mccabe==0.7.0,pip==22.3.1,pycodestyle==2.11.0,pyflakes==3.1.0,setuptools==65.6.3,wheel==0.38.4 flake8: commands[0] /home/jenkins-build/build/workspace/ceph-pull-requests/qa> flake8 --select=F,E9 --exclude=venv,.tox ./tasks/cephfs/test_client_recovery.py:12:1: F811 redefinition of unused 'random' from line 7 flake8: exit 1 (3.72 seconds) /home/jenkins-build/build/workspace/ceph-pull-requests/qa> flake8 --select=F,E9 --exclude=venv,.tox pid=706315 flake8: FAIL ✖ in 15.42 seconds Signed-off-by: Casey Bodley <cbodley@redhat.com>

seeing this run-tox-qa failure about tasks/cephfs/test_client_recovery.py: 246/285 Test ceph#264: run-tox-qa ................................***Failed 58.54 sec Requirement already satisfied: tox in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (4.6.4) Requirement already satisfied: cachetools>=5.3.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (5.3.1) Requirement already satisfied: chardet>=5.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (5.1.0) Requirement already satisfied: colorama>=0.4.6 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (0.4.6) Requirement already satisfied: filelock>=3.12.2 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (3.12.2) Requirement already satisfied: packaging>=23.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (23.1) Requirement already satisfied: platformdirs>=3.8 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (3.10.0) Requirement already satisfied: pluggy>=1.2 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (1.2.0) Requirement already satisfied: pyproject-api>=1.5.2 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (1.5.3) Requirement already satisfied: tomli>=2.0.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (2.0.1) Requirement already satisfied: virtualenv>=20.23.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (20.24.2) Requirement already satisfied: distlib<1,>=0.3.7 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from virtualenv>=20.23.1->tox) (0.3.7) flake8: install_deps /home/jenkins-build/build/workspace/ceph-pull-requests/qa> python -I -m pip install flake8 flake8: freeze /home/jenkins-build/build/workspace/ceph-pull-requests/qa> python -m pip freeze --all flake8: flake8==6.1.0,mccabe==0.7.0,pip==22.3.1,pycodestyle==2.11.0,pyflakes==3.1.0,setuptools==65.6.3,wheel==0.38.4 flake8: commands[0] /home/jenkins-build/build/workspace/ceph-pull-requests/qa> flake8 --select=F,E9 --exclude=venv,.tox ./tasks/cephfs/test_client_recovery.py:12:1: F811 redefinition of unused 'random' from line 7 flake8: exit 1 (3.72 seconds) /home/jenkins-build/build/workspace/ceph-pull-requests/qa> flake8 --select=F,E9 --exclude=venv,.tox pid=706315 flake8: FAIL ✖ in 15.42 seconds Signed-off-by: Casey Bodley <cbodley@redhat.com> (cherry picked from commit 568a21d)

seeing this run-tox-qa failure about tasks/cephfs/test_client_recovery.py: 246/285 Test ceph#264: run-tox-qa ................................***Failed 58.54 sec Requirement already satisfied: tox in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (4.6.4) Requirement already satisfied: cachetools>=5.3.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (5.3.1) Requirement already satisfied: chardet>=5.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (5.1.0) Requirement already satisfied: colorama>=0.4.6 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (0.4.6) Requirement already satisfied: filelock>=3.12.2 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (3.12.2) Requirement already satisfied: packaging>=23.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (23.1) Requirement already satisfied: platformdirs>=3.8 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (3.10.0) Requirement already satisfied: pluggy>=1.2 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (1.2.0) Requirement already satisfied: pyproject-api>=1.5.2 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (1.5.3) Requirement already satisfied: tomli>=2.0.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (2.0.1) Requirement already satisfied: virtualenv>=20.23.1 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from tox) (20.24.2) Requirement already satisfied: distlib<1,>=0.3.7 in /home/jenkins-build/build/workspace/ceph-pull-requests/build/qa-virtualenv/lib/python3.10/site-packages (from virtualenv>=20.23.1->tox) (0.3.7) flake8: install_deps /home/jenkins-build/build/workspace/ceph-pull-requests/qa> python -I -m pip install flake8 flake8: freeze /home/jenkins-build/build/workspace/ceph-pull-requests/qa> python -m pip freeze --all flake8: flake8==6.1.0,mccabe==0.7.0,pip==22.3.1,pycodestyle==2.11.0,pyflakes==3.1.0,setuptools==65.6.3,wheel==0.38.4 flake8: commands[0] /home/jenkins-build/build/workspace/ceph-pull-requests/qa> flake8 --select=F,E9 --exclude=venv,.tox ./tasks/cephfs/test_client_recovery.py:12:1: F811 redefinition of unused 'random' from line 7 flake8: exit 1 (3.72 seconds) /home/jenkins-build/build/workspace/ceph-pull-requests/qa> flake8 --select=F,E9 --exclude=venv,.tox pid=706315 flake8: FAIL ✖ in 15.42 seconds Signed-off-by: Casey Bodley <cbodley@redhat.com>

gollub added 7 commits May 7, 2013 10:06

osd: avoid that osd get killed via OOM killer

5efcacb

Introduce also handling for legacy OOM adjust which has different OOM disable value. Signed-off-by: Daniel Gollub <d.gollub@telekom.de>

osd: do OOM adjustments only on Linux builds

5bfca76

Signed-off-by: Daniel Gollub <d.gollub@telekom.de>

osd: unify ofstream calls for OOM adjustments

e945394

Signed-off-by: Daniel Gollub <d.gollub@telekom.de>

global: move OOM adjust/protector code to global

0213f15

This allows to reuse the code for other daemons again. Signed-off-by: Daniel Gollub <d.gollub@telekom.de>

global: document the OOM killer avoidance parameters

b640052

Signed-off-by: Daniel Gollub <d.gollub@telekom.de>

mon: add OOM killer avoidance handling

3130646

This implementation is not disabling OOM killer for the mon completelty. It just reduce that the montior get killed first on an OOM situation. Signed-off-by: Daniel Gollub <d.gollub@telekom.de>

liewegas reviewed May 8, 2013
View reviewed changes

src/global/global_init.h

Copy link

Member

liewegas May 8, 2013

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we shoud make these tunables in common/config_opts.h.. oom_* maybe?

liewegas closed this Aug 17, 2013

liewegas pushed a commit that referenced this pull request Dec 14, 2016

Merge pull request #264 from ceph/wip-wn

9b7a2f1

tasks/watch_notify_same_primary: fix test for new watch/notify Reviewed-by: Samuel Just <sjust@redhat.com>

liewegas pushed a commit to liewegas/ceph that referenced this pull request Nov 5, 2021

Merge pull request ceph#264 from Thingee/fix-broken-asset

1d9e796

Fix broken asset link

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RFC] protect against OOM killer#264

[RFC] protect against OOM killer#264
gollub wants to merge 7 commits intoceph:masterfrom
gollub:avoid_oom_killer

gollub commented May 8, 2013

Uh oh!

liewegas May 8, 2013

Uh oh!

liewegas commented May 8, 2013

Uh oh!

dalgaaf commented May 10, 2013

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

gollub commented May 8, 2013

Uh oh!

liewegas May 8, 2013

Choose a reason for hiding this comment

Uh oh!

liewegas commented May 8, 2013

Uh oh!

dalgaaf commented May 10, 2013

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants