ceph-crash: drop privleges to run as "ceph" user, rather than root (CVE-2022-3650)#48713
ceph-crash: drop privleges to run as "ceph" user, rather than root (CVE-2022-3650)#48713
Conversation
Popen.communicate() returns a tuple (stdout, stderr), and stderr will be of type bytes, hence the need to decode it before checking if it's an empty string or not. Fixes: a77b47e Signed-off-by: Tim Serong <tserong@suse.com>
If privileges cannot be dropped, log an error and exit. This commit also catches and logs exceptions when scraping the crash path, without which ceph-crash would just exit if it encountered an error. Fixes: CVE-2022-3650 Fixes: https://tracker.ceph.com/issues/57967 Signed-off-by: Tim Serong <tserong@suse.com>
|
jenkins test make check |
|
@tserong @dmick before this is backported to P and Q, I have identified a possible regression in the rados suite from this PR that is currently being investigated: https://tracker.ceph.com/issues/58098 I left a request for changes on P and Q so they aren't backported until we can confirm for sure. |
|
Thanks @ljflores, I've commented on the bug. |
|
Thanks @tserong. I reverted the two commits on a test branch that was failing and reran the tests, which passed. So, it seems like this has caused a regression in the testing suite. I have recorded all efforts on https://tracker.ceph.com/issues/58098 and attached the journalctl file you were interested in. Let me know if I can do anything else to help debug. |
A rather recent PR made ceph-crash run as "ceph" user instead of root [0]. However, because /var/lib/ceph/crash/posted belongs to root, ceph-crash cannot actually post any crash logs now. This commit fixes this by recursively updating the permissions of '/var/lib/ceph/crash', which ensures that all files and directories used by 'ceph-crash.service' are actually owned by the user configured for Ceph. The previously existing loop has also been replaced by an invocation of `find | xargs`. [0]: ceph#48713 Fixes: https://tracker.ceph.com/issues/64548 Signed-off-by: Max Carrara <m.carrara@proxmox.com>
A rather recent PR made ceph-crash run as "ceph" user instead of root [0]. However, because /var/lib/ceph/crash/posted belongs to root, ceph-crash cannot actually post any crash logs now. This commit fixes this by recursively updating the permissions of '/var/lib/ceph/crash', which ensures that all files and directories used by 'ceph-crash.service' are actually owned by the user configured for Ceph. The previously existing loop has also been replaced by an invocation of `find | xargs`. [0]: ceph#48713 Fixes: https://tracker.ceph.com/issues/64548 Signed-off-by: Max Carrara <m.carrara@proxmox.com>
|
A regression similar to #48713 (comment) exists in cephadm as listed in https://tracker.ceph.com/issues/61589 and Debian package in https://tracker.ceph.com/issues/64548 ; adding the links here for the next person trying to figure out why crash is throwing errors constantly. |
Due to Ceph dropping privileges when running the 'ceph-crash' daemon
[0], it is necessary to allow the daemon to authenticate with its
cluster in a safe manner.
In order to avoid exposing sensitive keyrings or somehow escalating
its privileges again, 'ceph-crash' is therefore provided with its own
keyring in the '/etc/pve/ceph' directory. This directory, due to being
on 'pmxcfs', may be read by members of the 'www-data' group, which
'ceph-crash' is made part of [1].
Expected Configuration
----------------------
1. A keyring file named '/etc/pve/ceph/ceph.client.crash.keyring'
exists
2. A section named 'client.crash' exists in '/etc/pve/ceph.conf'
3. The 'client.crash' section has a key named 'keyring' which
references the keyring file as '/etc/pve/ceph/$cluster.$name.keyring'
4. The 'client.crash' section has *no* key named 'key'
New Clusters
------------
The keyring file is created and the conf file is updated after the first
monitor has been created (when calling `pveceph mon create`).
Existing Clusters
-----------------
A new helper script creates and configures the 'client.crash' keyring in
`postinst`, if:
* Ceph is installed
* Ceph is initialized ('/etc/pve/ceph.conf' and '/etc/pve/ceph' exist)
* Connection to RADOS is successful
If the above conditions are met, the helper script ensures that the
existing configuration matches the expected configuration mentioned
above.
The configuration is not changed if it is already as expected.
The helper script may be called again manually if the `postinst` hook
fails. It is installed to '/usr/share/pve-manager/helpers/pve-init-ceph-crash'.
Existing `client.crash` Key
---------------------------
If a key named 'client.crash' already exists within the cluster, it is
reused and not regenerated.
[0]: ceph/ceph#48713
[1]: https://git.proxmox.com/?p=ceph.git;a=commitdiff;h=f72c698a55905d93e9a0b7b95674616547deba8a
Signed-off-by: Max Carrara <m.carrara@proxmox.com>
Tested-by: Friedrich Weber <f.weber@proxmox.com>
A rather recent PR made ceph-crash run as "ceph" user instead of root [0]. However, because /var/lib/ceph/crash/posted belongs to root, ceph-crash cannot actually post any crash logs now. This commit fixes this by recursively updating the permissions of '/var/lib/ceph/crash', which ensures that all files and directories used by 'ceph-crash.service' are actually owned by the user configured for Ceph. This also accounts for existing installations. Additionally, quote interpolated variables and use curly braces [1]. [0]: ceph#48713 [1]: https://www.shellcheck.net/wiki/SC2086 Fixes: https://tracker.ceph.com/issues/64548 Signed-off-by: Max Carrara <m.carrara@proxmox.com>
A rather recent PR made ceph-crash run as "ceph" user instead of root [0]. However, because /var/lib/ceph/crash/posted belongs to root, ceph-crash cannot actually post any crash logs now. This commit fixes this by recursively updating the permissions of '/var/lib/ceph/crash', which ensures that all files and directories used by 'ceph-crash.service' are actually owned by the user configured for Ceph. This also accounts for existing installations. Additionally, quote interpolated variables and use curly braces [1]. [0]: ceph#48713 [1]: https://www.shellcheck.net/wiki/SC2086 Fixes: https://tracker.ceph.com/issues/64548 Signed-off-by: Max Carrara <m.carrara@proxmox.com> (cherry picked from commit 97a1ec4)
A rather recent PR made ceph-crash run as "ceph" user instead of root [0]. However, because /var/lib/ceph/crash/posted belongs to root, ceph-crash cannot actually post any crash logs now. This commit fixes this by recursively updating the permissions of '/var/lib/ceph/crash', which ensures that all files and directories used by 'ceph-crash.service' are actually owned by the user configured for Ceph. This also accounts for existing installations. Additionally, quote interpolated variables and use curly braces [1]. [0]: ceph#48713 [1]: https://www.shellcheck.net/wiki/SC2086 Fixes: https://tracker.ceph.com/issues/64548 Signed-off-by: Max Carrara <m.carrara@proxmox.com> (cherry picked from commit 97a1ec4)
A rather recent PR made ceph-crash run as "ceph" user instead of root [0]. However, because /var/lib/ceph/crash/posted belongs to root, ceph-crash cannot actually post any crash logs now. This commit fixes this by recursively updating the permissions of '/var/lib/ceph/crash', which ensures that all files and directories used by 'ceph-crash.service' are actually owned by the user configured for Ceph. This also accounts for existing installations. Additionally, quote interpolated variables and use curly braces [1]. [0]: ceph#48713 [1]: https://www.shellcheck.net/wiki/SC2086 Fixes: https://tracker.ceph.com/issues/64548 Signed-off-by: Max Carrara <m.carrara@proxmox.com> (cherry picked from commit 97a1ec4)
A rather recent PR made ceph-crash run as "ceph" user instead of root [0]. However, because /var/lib/ceph/crash/posted belongs to root, ceph-crash cannot actually post any crash logs now. This commit fixes this by recursively updating the permissions of '/var/lib/ceph/crash', which ensures that all files and directories used by 'ceph-crash.service' are actually owned by the user configured for Ceph. This also accounts for existing installations. Additionally, quote interpolated variables and use curly braces [1]. [0]: ceph#48713 [1]: https://www.shellcheck.net/wiki/SC2086 Fixes: https://tracker.ceph.com/issues/64548 Signed-off-by: Max Carrara <m.carrara@proxmox.com> (cherry picked from commit 97a1ec4)
A rather recent PR made ceph-crash run as "ceph" user instead of root [0]. However, because /var/lib/ceph/crash/posted belongs to root, ceph-crash cannot actually post any crash logs now. This commit fixes this by recursively updating the permissions of '/var/lib/ceph/crash', which ensures that all files and directories used by 'ceph-crash.service' are actually owned by the user configured for Ceph. This also accounts for existing installations. Additionally, quote interpolated variables and use curly braces [1]. [0]: ceph#48713 [1]: https://www.shellcheck.net/wiki/SC2086 Fixes: https://tracker.ceph.com/issues/64548 Signed-off-by: Max Carrara <m.carrara@proxmox.com> (cherry picked from commit 97a1ec4)
A rather recent PR made ceph-crash run as "ceph" user instead of root [0]. However, because /var/lib/ceph/crash/posted belongs to root, ceph-crash cannot actually post any crash logs now. This commit fixes this by recursively updating the permissions of '/var/lib/ceph/crash', which ensures that all files and directories used by 'ceph-crash.service' are actually owned by the user configured for Ceph. This also accounts for existing installations. Additionally, quote interpolated variables and use curly braces [1]. [0]: ceph#48713 [1]: https://www.shellcheck.net/wiki/SC2086 Fixes: https://tracker.ceph.com/issues/64548 Signed-off-by: Max Carrara <m.carrara@proxmox.com> (cherry picked from commit 97a1ec4)
A rather recent PR made ceph-crash run as "ceph" user instead of root [0]. However, because /var/lib/ceph/crash/posted belongs to root, ceph-crash cannot actually post any crash logs now. This commit fixes this by recursively updating the permissions of '/var/lib/ceph/crash', which ensures that all files and directories used by 'ceph-crash.service' are actually owned by the user configured for Ceph. This also accounts for existing installations. Additionally, quote interpolated variables and use curly braces [1]. [0]: ceph#48713 [1]: https://www.shellcheck.net/wiki/SC2086 Fixes: https://tracker.ceph.com/issues/64548 Signed-off-by: Max Carrara <m.carrara@proxmox.com> (cherry picked from commit 97a1ec4)
A rather recent PR made ceph-crash run as "ceph" user instead of root [0]. However, because /var/lib/ceph/crash/posted belongs to root, ceph-crash cannot actually post any crash logs now. This commit fixes this by recursively updating the permissions of '/var/lib/ceph/crash', which ensures that all files and directories used by 'ceph-crash.service' are actually owned by the user configured for Ceph. The previously existing loop has also been replaced by an invocation of `find | xargs`. [0]: ceph#48713 Signed-off-by: Max Carrara <m.carrara@proxmox.com>
A rather recent PR made ceph-crash run as "ceph" user instead of root [0]. However, because /var/lib/ceph/crash/posted belongs to root, ceph-crash cannot actually post any crash logs now. This commit fixes this by recursively updating the permissions of '/var/lib/ceph/crash', which ensures that all files and directories used by 'ceph-crash.service' are actually owned by the user configured for Ceph. The previously existing loop has also been replaced by an invocation of `find | xargs`. [0]: ceph#48713 Signed-off-by: Max Carrara <m.carrara@proxmox.com>
A rather recent PR made ceph-crash run as "ceph" user instead of root [0]. However, because /var/lib/ceph/crash/posted belongs to root, ceph-crash cannot actually post any crash logs now. This commit fixes this by also updating the permissions of /var/lib/ceph/*/* - the subdirectories and files of the directories in /var/lib/ceph - by using `find` instead of a loop over a glob pattern. [0]: ceph#48713 Signed-off-by: Max Carrara <m.carrara@proxmox.com>
A rather recent PR made ceph-crash run as "ceph" user instead of root [0]. However, because /var/lib/ceph/crash/posted belongs to root, ceph-crash cannot actually post any crash logs now. This commit fixes this by also updating the permissions of /var/lib/ceph/*/* - the subdirectories and files of the directories in /var/lib/ceph - by using `find` instead of a loop over a glob pattern. [0]: ceph#48713 Signed-off-by: Max Carrara <m.carrara@proxmox.com>
A rather recent PR made ceph-crash run as "ceph" user instead of root [0]. However, because /var/lib/ceph/crash/posted belongs to root, ceph-crash cannot actually post any crash logs now. This commit fixes this by also updating the permissions of /var/lib/ceph/*/* - the subdirectories of the directories in /var/lib/ceph. [0]: ceph#48713 Signed-off-by: Max Carrara <m.carrara@proxmox.com>
This makes the ceph-crash script drop privileges and run as the "ceph" user, much the same as the other ceph daemons already do. It also incidentally fixes stderr handling.
I've tested this by applying it to SUSE's downstream Pacific branch, and running the exploit script linked off https://www.openwall.com/lists/oss-security/2022/10/25/1 before and after. Without the fix,
/usr/bin/mountis replaced with evil code. With the fix applied, ceph-crash logs an appropriate error instead ("ERROR:ceph-crash:Error scraping /var/lib/ceph/crash: [Errno 13] Permission denied: '/var/lib/ceph/crash/mount' -> '/var/lib/ceph/crash/posted/mount'").Contribution Guidelines
To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
Checklist
Show available Jenkins commands
jenkins retest this pleasejenkins test classic perfjenkins test crimson perfjenkins test signedjenkins test make checkjenkins test make check arm64jenkins test submodulesjenkins test dashboardjenkins test dashboard cephadmjenkins test apijenkins test docsjenkins render docsjenkins test ceph-volume alljenkins test ceph-volume toxjenkins test windows