reef: mon/AuthMonitor: provide command to rotate the key for a user credential by batrick · Pull Request #58236 · ceph/ceph

batrick · 2024-06-24T15:54:50Z

backport tracker: https://tracker.ceph.com/issues/66618

backport of #58121
parent tracker: https://tracker.ceph.com/issues/66509

this backport was staged using ceph-backport.sh version 16.0.0.6848
find the latest version at https://github.com/ceph/ceph/blob/main/src/script/ceph-backport.sh

batrick · 2024-08-23T02:57:43Z

jenkins test make check

batrick · 2024-08-23T02:57:47Z

jenkins test windows

batrick · 2024-08-23T02:57:53Z

jenkins test api

batrick · 2024-09-11T16:18:31Z

jenkins test api

batrick · 2024-09-11T16:18:35Z

jenkins test windows

batrick · 2024-09-11T16:18:42Z

jenkins test make check

mchangir · 2025-02-10T05:37:46Z

This PR is under test in https://tracker.ceph.com/issues/69881.

shraddhaag · 2025-02-19T06:55:41Z

Hey @batrick, while going through the failures for the batch that included this PR the following error was encountered that seems to be related to this PR:

2025-02-15T00:47:12.574 INFO:tasks.workunit.client.0.smithi102.stdout:  key = AQAQ5K9n8Q6fGRAAeif4RSrb0bR7QEiWR73JfA==
2025-02-15T00:47:12.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:614: test_auth:  ceph auth get client.admin2
2025-02-15T00:47:13.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:615: test_auth:  env CEPH_KEYRING=keyring1 ceph -n client.admin2 auth get client.admin2
2025-02-15T00:47:13.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:617: test_auth:  expect_true diff -au keyring1 keyring2
2025-02-15T00:47:13.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:41: expect_true:  set -x
2025-02-15T00:47:13.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:42: expect_true:  diff -au keyring1 keyring2
2025-02-15T00:47:13.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:42: expect_true:  return 0
2025-02-15T00:47:13.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:619: test_auth:  env CEPH_KEYRING=keyring1 ceph -n client.admin2 auth rotate client.admin2
2025-02-15T00:47:14.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:621: test_auth:  diff -au keyring1 keyring3
2025-02-15T00:47:14.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:621: test_auth:  grep -E '^[-+][^-+]'
2025-02-15T00:47:14.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:621: test_auth:  expect_false grep -v key
2025-02-15T00:47:14.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false:  set -x
2025-02-15T00:47:14.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false:  grep -v key
2025-02-15T00:47:14.042 INFO:tasks.workunit.client.0.smithi102.stdout:- caps mon = "allow *"
2025-02-15T00:47:14.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false:  return 1
2025-02-15T00:47:14.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:1: test_auth:  rm -fr /tmp/cephtool.uDS
2025-02-15T00:47:14.043 DEBUG:teuthology.orchestra.run:got remote process result: 1
2025-02-15T00:47:14.044 INFO:tasks.workunit:Stopping ['cephtool'] on client.0...
2025-02-15T00:47:14.044 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0
2025-02-15T00:47:14.343 ERROR:teuthology.run_tasks:Saw exception from tasks.

Could you please take a look? Meanwhile I'll request @SrinivasaBharath to rerun the batch excluding this PR to be sure.

SrinivasaBharath · 2025-02-26T06:41:39Z

@shraddhaag Excluded the PR from the batch

mchangir · 2025-02-26T08:53:11Z

This PR is under test in https://tracker.ceph.com/issues/70178.

Naveenaidu · 2025-04-01T13:26:34Z

while going through the failures of a batch that included this PR, we see the following error:

2025-03-18T03:54:56.338 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:613: test_auth:  ceph auth get-or-create client.admin2 mon 'allow *'
2025-03-18T03:54:56.856 INFO:tasks.workunit.client.0.smithi037.stdout:[client.admin2]
2025-03-18T03:54:56.856 INFO:tasks.workunit.client.0.smithi037.stdout:  key = AQCQ7thnz39PKRAACyv9fwObuJ6S+Q3aHRh/Tw==
2025-03-18T03:54:56.868 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:614: test_auth:  ceph auth get client.admin2
2025-03-18T03:54:57.390 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:615: test_auth:  env CEPH_KEYRING=keyring1 ceph -n client.admin2 auth get client.admin2
2025-03-18T03:54:57.886 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:617: test_auth:  expect_true diff -au keyring1 keyring2
2025-03-18T03:54:57.886 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:41: expect_true:  set -x
2025-03-18T03:54:57.886 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:42: expect_true:  diff -au keyring1 keyring2
2025-03-18T03:54:57.886 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:42: expect_true:  return 0
2025-03-18T03:54:57.886 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:619: test_auth:  env CEPH_KEYRING=keyring1 ceph -n client.admin2 auth rotate client.admin2
2025-03-18T03:54:58.430 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:621: test_auth:  diff -au keyring1 keyring3
2025-03-18T03:54:58.430 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:621: test_auth:  grep -E '^[-+][^-+]'
2025-03-18T03:54:58.430 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:621: test_auth:  expect_false grep -v key
2025-03-18T03:54:58.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false:  set -x
2025-03-18T03:54:58.431 INFO:tasks.workunit.client.0.smithi037.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false:  grep -v key

The error lines seems to match the changes that were done in this PR. These are the jobs that had this errors in the run of the batch

/a/yuriw-2025-03-18_01:22:46-rados-wip-yuri5-testing-2025-03-17-1441-reef-distro-default-smithi/8196407
/a/yuriw-2025-03-18_01:22:46-rados-wip-yuri5-testing-2025-03-17-1441-reef-distro-default-smithi/8196610
/a/yuriw-2025-03-18_01:22:46-rados-wip-yuri5-testing-2025-03-17-1441-reef-distro-default-smithi/8196542
/a/yuriw-2025-03-18_01:22:46-rados-wip-yuri5-testing-2025-03-17-1441-reef-distro-default-smithi/8196473

@batrick it would be nice if you could take a look at this! Meanwhile have requested @SrinivasaBharath to rerun the batch without this PR to be sure

Naveenaidu · 2025-04-29T13:24:40Z

while going through the failures of a batch that included this PR, we see the following error:

2025-04-24T14:44:34.575 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:613: test_auth:  ceph auth get-or-create client.admin2 mon 'allow *'
2025-04-24T14:44:35.057 INFO:tasks.workunit.client.0.smithi104.stdout:[client.admin2]
2025-04-24T14:44:35.057 INFO:tasks.workunit.client.0.smithi104.stdout:  key = AQBSTgpo/GCwNRAAN2ucL4YR04DolOHZ9fBywA==
2025-04-24T14:44:35.073 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:614: test_auth:  ceph auth get client.admin2
2025-04-24T14:44:35.556 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:615: test_auth:  env CEPH_KEYRING=keyring1 ceph -n client.admin2 auth get client.admin2
2025-04-24T14:44:36.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:617: test_auth:  expect_true diff -au keyring1 keyring2
2025-04-24T14:44:36.037 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:41: expect_true:  set -x
2025-04-24T14:44:36.038 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:42: expect_true:  diff -au keyring1 keyring2
2025-04-24T14:44:36.039 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:42: expect_true:  return 0
2025-04-24T14:44:36.039 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:619: test_auth:  env CEPH_KEYRING=keyring1 ceph -n client.admin2 auth rotate client.admin2
2025-04-24T14:44:36.545 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:621: test_auth:  diff -au keyring1 keyring3
2025-04-24T14:44:36.546 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:621: test_auth:  grep -E '^[-+][^-+]'
2025-04-24T14:44:36.546 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:621: test_auth:  expect_false grep -v key
2025-04-24T14:44:36.546 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:35: expect_false:  set -x
2025-04-24T14:44:36.546 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false:  grep -v key
2025-04-24T14:44:36.546 INFO:tasks.workunit.client.0.smithi104.stdout:- caps mon = "allow *"
2025-04-24T14:44:36.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:36: expect_false:  return 1
2025-04-24T14:44:36.547 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:1: test_auth:  rm -fr /tmp/cephtool.R1f
2025-04-24T14:44:36.548 DEBUG:teuthology.orchestra.run:got remote process result: 1

The error lines seems to match the changes that were done in this PR. These are the jobs that had this errors in the run of the batch

/a/skanta-2025-04-24_14:20:20-rados-wip-bharath8-testing-2025-04-23-0519-reef-distro-default-smithi/8257810
/a/skanta-2025-04-24_14:20:20-rados-wip-bharath8-testing-2025-04-23-0519-reef-distro-default-smithi/8257816
/a/skanta-2025-04-24_14:20:20-rados-wip-bharath8-testing-2025-04-23-0519-reef-distro-default-smithi/8257830
/a/skanta-2025-04-24_14:20:20-rados-wip-bharath8-testing-2025-04-23-0519-reef-distro-default-smithi/8257824

@batrick it would be nice if you could take a look at this! Meanwhile have requested @SrinivasaBharath to rerun the batch without this PR to be sure

batrick · 2025-04-29T18:22:16Z

I will take over QA of this. Need to figure out what's going on.

batrick · 2025-04-29T18:23:23Z

This PR is under test in https://tracker.ceph.com/issues/71132.

* refs/pull/58236/head: doc: add documentation for `ceph auth rotate` PendingReleaseNotes: add note for new `auth rotate` qa: test `auth rotate` mon/AuthMonitor: add `ceph auth rotate` command

* refs/pull/58236/head: test doc: add documentation for `ceph auth rotate` PendingReleaseNotes: add note for new `auth rotate` qa: test `auth rotate` mon/AuthMonitor: add `ceph auth rotate` command

batrick · 2025-04-30T17:59:52Z

Fixed with this patch:

diff --git a/src/mon/AuthMonitor.cc b/src/mon/AuthMonitor.cc
index c1e70e91c109..4c88c7d74a71 100644
--- a/src/mon/AuthMonitor.cc
+++ b/src/mon/AuthMonitor.cc
@@ -1941,9 +1941,8 @@ bool AuthMonitor::prepare_command(MonOpRequestRef op)
 
     {
       KeyRing kr;
-      kr.add(entity, entity_auth.key, entity_auth.pending_key);
+      kr.add(entity, entity_auth);
       if (f) {
-        kr.set_caps(entity, entity_auth.caps);
         kr.encode_formatted("auth", f.get(), rdata);
       } else {
         kr.encode_plaintext(rdata);

https://pulpito.ceph.com/pdonnell-2025-04-30_17:14:51-rados:singleton-bluestore-wip-pdonnell-testing-20250430.151812-reef-debug-distro-default-smithi/

Add command to rotate the permanent key of an entity. This avoids the need to delete / recreate the key when it is compromised, lost, or just scheduled for rotation. Fixes: https://tracker.ceph.com/issues/66509 Signed-off-by: Patrick Donnelly <pdonnell@redhat.com> (cherry picked from commit d57326f) Conflicts: src/mon/AuthMonitor.cc: _encode_auth not in reef

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com> (cherry picked from commit 2ae027f)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com> (cherry picked from commit deab044)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com> (cherry picked from commit b871bbe)

batrick requested review from a team as code owners June 24, 2024 15:54

batrick added this to the reef milestone Jun 24, 2024

batrick added the core label Jun 24, 2024

github-actions bot added documentation mon tests labels Jun 24, 2024

github-actions bot added the stale label Nov 26, 2024

batrick removed the stale label Nov 26, 2024

batrick force-pushed the wip-66618-reef branch 2 times, most recently from 4c774f6 to 178ea2a Compare November 26, 2024 15:21

ceph deleted a comment from github-actions bot Dec 30, 2024

batrick added the needs-qa label Dec 30, 2024

SrinivasaBharath added the wip-bharath5-testing label Feb 10, 2025

SrinivasaBharath removed the wip-bharath5-testing label Feb 26, 2025

yuriw added the wip-yuri5-testing label Mar 17, 2025

yuriw removed the wip-yuri5-testing label Apr 7, 2025

SrinivasaBharath added the wip-bharath8-testing label Apr 22, 2025

batrick added wip-pdonnell-testing3 and removed needs-qa wip-bharath8-testing labels Apr 29, 2025

batrick added 4 commits April 30, 2025 14:00

qa: test auth rotate

f2ca73e

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com> (cherry picked from commit 2ae027f)

PendingReleaseNotes: add note for new auth rotate

fc4933d

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com> (cherry picked from commit deab044)

doc: add documentation for ceph auth rotate

df0863f

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com> (cherry picked from commit b871bbe)

batrick force-pushed the wip-66618-reef branch from a983bd2 to df0863f Compare April 30, 2025 18:00

batrick added needs-qa and removed wip-pdonnell-testing3 labels Apr 30, 2025

SrinivasaBharath added wip-bharath1-testing and removed wip-bharath1-testing labels May 6, 2025

SrinivasaBharath added wip-bharath4-testing and removed wip-bharath4-testing labels Jun 7, 2025

ljflores approved these changes Jun 11, 2025

View reviewed changes

batrick merged commit e5663f0 into ceph:reef Jun 11, 2025
8 checks passed

batrick deleted the wip-66618-reef branch June 11, 2025 16:22

SrinivasaBharath removed the wip-bharath4-testing label Jun 16, 2025

Conversation

batrick commented Jun 24, 2024

Uh oh!

batrick commented Aug 23, 2024

Uh oh!

batrick commented Aug 23, 2024

Uh oh!

batrick commented Aug 23, 2024

Uh oh!

batrick commented Sep 11, 2024

Uh oh!

batrick commented Sep 11, 2024

Uh oh!

batrick commented Sep 11, 2024

Uh oh!

mchangir commented Feb 10, 2025

Uh oh!

shraddhaag commented Feb 19, 2025

Uh oh!

SrinivasaBharath commented Feb 26, 2025

Uh oh!

mchangir commented Feb 26, 2025

Uh oh!

Naveenaidu commented Apr 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Naveenaidu commented Apr 29, 2025

Uh oh!

batrick commented Apr 29, 2025

Uh oh!

batrick commented Apr 29, 2025

Uh oh!

batrick commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Naveenaidu commented Apr 1, 2025 •

edited

Loading

batrick commented Apr 30, 2025 •

edited

Loading