Skip to content

ceph-crash: drop privleges to run as "ceph" user, rather than root (CVE-2022-3650)#48713

Merged
dmick merged 2 commits intoceph:mainfrom
SUSE:wip-fix-CVE-2022-3650
Nov 8, 2022
Merged

ceph-crash: drop privleges to run as "ceph" user, rather than root (CVE-2022-3650)#48713
dmick merged 2 commits intoceph:mainfrom
SUSE:wip-fix-CVE-2022-3650

Conversation

@tserong
Copy link
Member

@tserong tserong commented Nov 3, 2022

This makes the ceph-crash script drop privileges and run as the "ceph" user, much the same as the other ceph daemons already do. It also incidentally fixes stderr handling.

I've tested this by applying it to SUSE's downstream Pacific branch, and running the exploit script linked off https://www.openwall.com/lists/oss-security/2022/10/25/1 before and after. Without the fix, /usr/bin/mount is replaced with evil code. With the fix applied, ceph-crash logs an appropriate error instead ("ERROR:ceph-crash:Error scraping /var/lib/ceph/crash: [Errno 13] Permission denied: '/var/lib/ceph/crash/mount' -> '/var/lib/ceph/crash/posted/mount'").

Contribution Guidelines

Checklist

  • Tracker (select at least one)
    • References tracker ticket
    • Very recent bug; references commit where it was introduced
    • New feature (ticket optional)
    • Doc update (no ticket needed)
    • Code cleanup (no ticket needed)
  • Component impact
    • Affects Dashboard, opened tracker ticket
    • Affects Orchestrator, opened tracker ticket
    • No impact that needs to be tracked
  • Documentation (select at least one)
    • Updates relevant documentation
    • No doc update is appropriate
  • Tests (select at least one)
Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard cephadm
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox
  • jenkins test windows

Popen.communicate() returns a tuple (stdout, stderr), and stderr
will be of type bytes, hence the need to decode it before checking
if it's an empty string or not.

Fixes: a77b47e
Signed-off-by: Tim Serong <tserong@suse.com>
If privileges cannot be dropped, log an error and exit.  This commit
also catches and logs exceptions when scraping the crash path, without
which ceph-crash would just exit if it encountered an error.

Fixes: CVE-2022-3650
Fixes: https://tracker.ceph.com/issues/57967
Signed-off-by: Tim Serong <tserong@suse.com>
@tserong tserong requested review from guits and leseb November 3, 2022 05:20
@tserong
Copy link
Member Author

tserong commented Nov 3, 2022

jenkins test make check

@tserong tserong requested a review from a team November 4, 2022 05:31
@neha-ojha neha-ojha requested a review from dmick November 4, 2022 17:57
Copy link
Member

@dmick dmick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine to me. Thanks for implementing this @tserong

@ljflores
Copy link
Member

ljflores commented Dec 5, 2022

@tserong @dmick before this is backported to P and Q, I have identified a possible regression in the rados suite from this PR that is currently being investigated: https://tracker.ceph.com/issues/58098

I left a request for changes on P and Q so they aren't backported until we can confirm for sure.

@tserong
Copy link
Member Author

tserong commented Dec 6, 2022

Thanks @ljflores, I've commented on the bug.

@ljflores
Copy link
Member

ljflores commented Dec 6, 2022

Thanks @tserong. I reverted the two commits on a test branch that was failing and reran the tests, which passed. So, it seems like this has caused a regression in the testing suite.

I have recorded all efforts on https://tracker.ceph.com/issues/58098 and attached the journalctl file you were interested in. Let me know if I can do anything else to help debug.

Aequitosh added a commit to Aequitosh/ceph that referenced this pull request Mar 4, 2024
A rather recent PR made ceph-crash run as "ceph" user instead of
root [0]. However, because /var/lib/ceph/crash/posted belongs to root,
ceph-crash cannot actually post any crash logs now.

This commit fixes this by recursively updating the permissions of
'/var/lib/ceph/crash', which ensures that all files and directories
used by 'ceph-crash.service' are actually owned by the user configured
for Ceph.

The previously existing loop has also been replaced by an invocation
of `find | xargs`.

[0]: ceph#48713

Fixes: https://tracker.ceph.com/issues/64548
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Aequitosh added a commit to Aequitosh/ceph that referenced this pull request Mar 4, 2024
A rather recent PR made ceph-crash run as "ceph" user instead of
root [0]. However, because /var/lib/ceph/crash/posted belongs to root,
ceph-crash cannot actually post any crash logs now.

This commit fixes this by recursively updating the permissions of
'/var/lib/ceph/crash', which ensures that all files and directories
used by 'ceph-crash.service' are actually owned by the user configured
for Ceph.

The previously existing loop has also been replaced by an invocation
of `find | xargs`.

[0]: ceph#48713

Fixes: https://tracker.ceph.com/issues/64548
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
@bluikko
Copy link
Contributor

bluikko commented Mar 20, 2024

A regression similar to #48713 (comment) exists in cephadm as listed in https://tracker.ceph.com/issues/61589 and Debian package in https://tracker.ceph.com/issues/64548 ; adding the links here for the next person trying to figure out why crash is throwing errors constantly.

ProxBot pushed a commit to proxmox/pve-manager that referenced this pull request Apr 11, 2024
Due to Ceph dropping privileges when running the 'ceph-crash' daemon
[0], it is necessary to allow the daemon to authenticate with its
cluster in a safe manner.

In order to avoid exposing sensitive keyrings or somehow escalating
its privileges again, 'ceph-crash' is therefore provided with its own
keyring in the '/etc/pve/ceph' directory. This directory, due to being
on 'pmxcfs', may be read by members of the 'www-data' group, which
'ceph-crash' is made part of [1].

Expected Configuration
----------------------

 1. A keyring file named '/etc/pve/ceph/ceph.client.crash.keyring'
    exists
 2. A section named 'client.crash' exists in '/etc/pve/ceph.conf'
 3. The 'client.crash' section has a key named 'keyring' which
    references the keyring file as '/etc/pve/ceph/$cluster.$name.keyring'
 4. The 'client.crash' section has *no* key named 'key'

New Clusters
------------

The keyring file is created and the conf file is updated after the first
monitor has been created (when calling `pveceph mon create`).

Existing Clusters
-----------------

A new helper script creates and configures the 'client.crash' keyring in
`postinst`, if:
 * Ceph is installed
 * Ceph is initialized ('/etc/pve/ceph.conf' and '/etc/pve/ceph' exist)
 * Connection to RADOS is successful

If the above conditions are met, the helper script ensures that the
existing configuration matches the expected configuration mentioned
above.

The configuration is not changed if it is already as expected.

The helper script may be called again manually if the `postinst` hook
fails. It is installed to '/usr/share/pve-manager/helpers/pve-init-ceph-crash'.

Existing `client.crash` Key
---------------------------

If a key named 'client.crash' already exists within the cluster, it is
reused and not regenerated.

[0]: ceph/ceph#48713
[1]: https://git.proxmox.com/?p=ceph.git;a=commitdiff;h=f72c698a55905d93e9a0b7b95674616547deba8a

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Friedrich Weber <f.weber@proxmox.com>
Aequitosh added a commit to Aequitosh/ceph that referenced this pull request Apr 30, 2024
A rather recent PR made ceph-crash run as "ceph" user instead of
root [0]. However, because /var/lib/ceph/crash/posted belongs to root,
ceph-crash cannot actually post any crash logs now.

This commit fixes this by recursively updating the permissions of
'/var/lib/ceph/crash', which ensures that all files and directories
used by 'ceph-crash.service' are actually owned by the user configured
for Ceph. This also accounts for existing installations.

Additionally, quote interpolated variables and use curly braces [1].

[0]: ceph#48713
[1]: https://www.shellcheck.net/wiki/SC2086

Fixes: https://tracker.ceph.com/issues/64548
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
k0ste pushed a commit to k0ste/ceph that referenced this pull request Jul 8, 2024
A rather recent PR made ceph-crash run as "ceph" user instead of
root [0]. However, because /var/lib/ceph/crash/posted belongs to root,
ceph-crash cannot actually post any crash logs now.

This commit fixes this by recursively updating the permissions of
'/var/lib/ceph/crash', which ensures that all files and directories
used by 'ceph-crash.service' are actually owned by the user configured
for Ceph. This also accounts for existing installations.

Additionally, quote interpolated variables and use curly braces [1].

[0]: ceph#48713
[1]: https://www.shellcheck.net/wiki/SC2086

Fixes: https://tracker.ceph.com/issues/64548
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
(cherry picked from commit 97a1ec4)
k0ste pushed a commit to k0ste/ceph that referenced this pull request Jul 8, 2024
A rather recent PR made ceph-crash run as "ceph" user instead of
root [0]. However, because /var/lib/ceph/crash/posted belongs to root,
ceph-crash cannot actually post any crash logs now.

This commit fixes this by recursively updating the permissions of
'/var/lib/ceph/crash', which ensures that all files and directories
used by 'ceph-crash.service' are actually owned by the user configured
for Ceph. This also accounts for existing installations.

Additionally, quote interpolated variables and use curly braces [1].

[0]: ceph#48713
[1]: https://www.shellcheck.net/wiki/SC2086

Fixes: https://tracker.ceph.com/issues/64548
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
(cherry picked from commit 97a1ec4)
k0ste pushed a commit to k0ste/ceph that referenced this pull request Jul 18, 2024
A rather recent PR made ceph-crash run as "ceph" user instead of
root [0]. However, because /var/lib/ceph/crash/posted belongs to root,
ceph-crash cannot actually post any crash logs now.

This commit fixes this by recursively updating the permissions of
'/var/lib/ceph/crash', which ensures that all files and directories
used by 'ceph-crash.service' are actually owned by the user configured
for Ceph. This also accounts for existing installations.

Additionally, quote interpolated variables and use curly braces [1].

[0]: ceph#48713
[1]: https://www.shellcheck.net/wiki/SC2086

Fixes: https://tracker.ceph.com/issues/64548
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
(cherry picked from commit 97a1ec4)
k0ste pushed a commit to k0ste/ceph that referenced this pull request Jul 24, 2024
A rather recent PR made ceph-crash run as "ceph" user instead of
root [0]. However, because /var/lib/ceph/crash/posted belongs to root,
ceph-crash cannot actually post any crash logs now.

This commit fixes this by recursively updating the permissions of
'/var/lib/ceph/crash', which ensures that all files and directories
used by 'ceph-crash.service' are actually owned by the user configured
for Ceph. This also accounts for existing installations.

Additionally, quote interpolated variables and use curly braces [1].

[0]: ceph#48713
[1]: https://www.shellcheck.net/wiki/SC2086

Fixes: https://tracker.ceph.com/issues/64548
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
(cherry picked from commit 97a1ec4)
k0ste pushed a commit to k0ste/ceph that referenced this pull request Oct 8, 2024
A rather recent PR made ceph-crash run as "ceph" user instead of
root [0]. However, because /var/lib/ceph/crash/posted belongs to root,
ceph-crash cannot actually post any crash logs now.

This commit fixes this by recursively updating the permissions of
'/var/lib/ceph/crash', which ensures that all files and directories
used by 'ceph-crash.service' are actually owned by the user configured
for Ceph. This also accounts for existing installations.

Additionally, quote interpolated variables and use curly braces [1].

[0]: ceph#48713
[1]: https://www.shellcheck.net/wiki/SC2086

Fixes: https://tracker.ceph.com/issues/64548
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
(cherry picked from commit 97a1ec4)
k0ste pushed a commit to k0ste/ceph that referenced this pull request Dec 17, 2024
A rather recent PR made ceph-crash run as "ceph" user instead of
root [0]. However, because /var/lib/ceph/crash/posted belongs to root,
ceph-crash cannot actually post any crash logs now.

This commit fixes this by recursively updating the permissions of
'/var/lib/ceph/crash', which ensures that all files and directories
used by 'ceph-crash.service' are actually owned by the user configured
for Ceph. This also accounts for existing installations.

Additionally, quote interpolated variables and use curly braces [1].

[0]: ceph#48713
[1]: https://www.shellcheck.net/wiki/SC2086

Fixes: https://tracker.ceph.com/issues/64548
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
(cherry picked from commit 97a1ec4)
k0ste pushed a commit to k0ste/ceph that referenced this pull request Jan 11, 2025
A rather recent PR made ceph-crash run as "ceph" user instead of
root [0]. However, because /var/lib/ceph/crash/posted belongs to root,
ceph-crash cannot actually post any crash logs now.

This commit fixes this by recursively updating the permissions of
'/var/lib/ceph/crash', which ensures that all files and directories
used by 'ceph-crash.service' are actually owned by the user configured
for Ceph. This also accounts for existing installations.

Additionally, quote interpolated variables and use curly braces [1].

[0]: ceph#48713
[1]: https://www.shellcheck.net/wiki/SC2086

Fixes: https://tracker.ceph.com/issues/64548
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
(cherry picked from commit 97a1ec4)
Aequitosh added a commit to Aequitosh/ceph that referenced this pull request Sep 12, 2025
A rather recent PR made ceph-crash run as "ceph" user instead of
root [0]. However, because /var/lib/ceph/crash/posted belongs to root,
ceph-crash cannot actually post any crash logs now.

This commit fixes this by recursively updating the permissions of
'/var/lib/ceph/crash', which ensures that all files and directories
used by 'ceph-crash.service' are actually owned by the user configured
for Ceph.

The previously existing loop has also been replaced by an invocation
of `find | xargs`.

[0]: ceph#48713

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Aequitosh added a commit to Aequitosh/ceph that referenced this pull request Sep 12, 2025
A rather recent PR made ceph-crash run as "ceph" user instead of
root [0]. However, because /var/lib/ceph/crash/posted belongs to root,
ceph-crash cannot actually post any crash logs now.

This commit fixes this by recursively updating the permissions of
'/var/lib/ceph/crash', which ensures that all files and directories
used by 'ceph-crash.service' are actually owned by the user configured
for Ceph.

The previously existing loop has also been replaced by an invocation
of `find | xargs`.

[0]: ceph#48713

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Aequitosh added a commit to Aequitosh/ceph that referenced this pull request Sep 12, 2025
A rather recent PR made ceph-crash run as "ceph" user instead of
root [0]. However, because /var/lib/ceph/crash/posted belongs to root,
ceph-crash cannot actually post any crash logs now.

This commit fixes this by also updating the permissions of
/var/lib/ceph/*/* - the subdirectories and files of the directories in
/var/lib/ceph - by using `find` instead of a loop over a glob pattern.

[0]: ceph#48713

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Aequitosh added a commit to Aequitosh/ceph that referenced this pull request Sep 12, 2025
A rather recent PR made ceph-crash run as "ceph" user instead of
root [0]. However, because /var/lib/ceph/crash/posted belongs to root,
ceph-crash cannot actually post any crash logs now.

This commit fixes this by also updating the permissions of
/var/lib/ceph/*/* - the subdirectories and files of the directories in
/var/lib/ceph - by using `find` instead of a loop over a glob pattern.

[0]: ceph#48713

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Aequitosh added a commit to Aequitosh/ceph that referenced this pull request Sep 12, 2025
A rather recent PR made ceph-crash run as "ceph" user instead of
root [0]. However, because /var/lib/ceph/crash/posted belongs to root,
ceph-crash cannot actually post any crash logs now.

This commit fixes this by also updating the permissions of
/var/lib/ceph/*/* - the subdirectories of the directories in
/var/lib/ceph.

[0]: ceph#48713

Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants