Skip to content

mds: fix session/client evict command.#49974

Merged
vshankar merged 3 commits intoceph:mainfrom
neesingh-rh:wip-58619
Jul 15, 2024
Merged

mds: fix session/client evict command.#49974
vshankar merged 3 commits intoceph:mainfrom
neesingh-rh:wip-58619

Conversation

@neesingh-rh
Copy link
Contributor

@neesingh-rh neesingh-rh commented Feb 2, 2023

Daemon commands are getting executed ignoring the ceph flags
like --help, --status etc.
For eg: ceph --admin-daemon $socketfile client evict --help is run
ignoring the --help flag which the user doesn't want and can be
dangerous sometimes as with the above command which will
terminate all active connections which isn't supposed to be.

This PR fixes the issues using these solutions:
(1) client evict without filter is forbidden
(2) SessionFilter::parse is modified to support id=* (id=0 is not allowed)
(3) Invalid id error is handled
User can use client evict id=* to evict all clients.

Fixes: https://tracker.ceph.com/issues/58619
Signed-off-by: Neeraj Pratap Singh neesingh@redhat.com

Contribution Guidelines

Checklist

  • Tracker (select at least one)
    • References tracker ticket
    • Very recent bug; references commit where it was introduced
    • New feature (ticket optional)
    • Doc update (no ticket needed)
    • Code cleanup (no ticket needed)
  • Component impact
    • Affects Dashboard, opened tracker ticket
    • Affects Orchestrator, opened tracker ticket
    • No impact that needs to be tracked
  • Documentation (select at least one)
    • Updates relevant documentation
    • No doc update is appropriate
  • Tests (select at least one)
Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard cephadm
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox
  • jenkins test windows

@neesingh-rh neesingh-rh requested a review from a team as a code owner February 2, 2023 13:14
@github-actions github-actions bot added the core label Feb 2, 2023
@neesingh-rh neesingh-rh requested a review from a team February 2, 2023 13:15
@neesingh-rh neesingh-rh changed the title mds: daemon commands run ignoring the ceph flags. ceph.in: admin_socket commands run ignoring the passed ceph flags. Feb 2, 2023
@neesingh-rh neesingh-rh requested a review from a team February 2, 2023 14:28
@gregsfortytwo
Copy link
Member

So this PR makes a pretty fundamental change to how admin sockets command parsing works: namely, it refuses to pass commands to the admin socket if there are "--" options in the command line. Are people okay with this? If not, we will need to handle stuff like that in every individual daemon, or build a more complicated harness that detects stuff like "--help" and then...doesn't invoke the command.

@vshankar
Copy link
Contributor

vshankar commented Feb 3, 2023

So this PR makes a pretty fundamental change to how admin sockets command parsing works: namely, it refuses to pass commands to the admin socket if there are "--" options in the command line. Are people okay with this? If not, we will need to handle stuff like that in every individual daemon, or build a more complicated harness that detects stuff like "--help" and then...doesn't invoke the command.

The ceph tell interface doesn't have this problem, so I guess doing this should be fine. But we still need this to be vetted (and probably run through tests) for each component.

@dparmar18
Copy link
Contributor

So this PR makes a pretty fundamental change to how admin sockets command parsing works: namely, it refuses to pass commands to the admin socket if there are "--" options in the command line. Are people okay with this? If not, we will need to handle stuff like that in every individual daemon, or build a more complicated harness that detects stuff like "--help" and then...doesn't invoke the command.

maybe we can add an exception here for value --help instead of refusing all the values?

Copy link
Contributor

@idryomov idryomov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this PR makes a pretty fundamental change to how admin sockets command parsing works: namely, it refuses to pass commands to the admin socket if there are "--" options in the command line. Are people okay with this?

Absolutely not. I don't think admin socket command accept --foo options but this breaks use cases like ceph --cluster my-other-cluster daemon osd.0 perf dump, etc.

@neesingh-rh
Copy link
Contributor Author

So this PR makes a pretty fundamental change to how admin sockets command parsing works: namely, it refuses to pass commands to the admin socket if there are "--" options in the command line. Are people okay with this?

Absolutely not. I don't think admin socket command accept --foo options but this breaks use cases like ceph --cluster my-other-cluster daemon osd.0 perf dump, etc.

Okay, if there are use cases which uses ceph flags with daemon commands like --cluster then we should modify this fix.Thanks @idryomov for pointing it out.

@vshankar
Copy link
Contributor

Spoke to @neesingh-rh regarding this - the --help (and many other flags) are parsed and recorded in parsed_args variable, effectively turning the command into client evict command string that gets handed over to the MDS.

We could add filter checks in ceph.in but that doesn't look neat (those need to be updated/maintained if we add/remove asok commands).

@neesingh-rh
Copy link
Contributor Author

I also checked the daemon command along with the ceph flags which need arguments to be passed , its buggy with them too ( not with all but some of them atleast).
for eg: ./bin/ceph --name invalid_client_name daemon mds.c client evict
It will execute the command client evict even if the passed client name doesn't exist.

@vshankar
Copy link
Contributor

vshankar commented Feb 22, 2023

I also checked the daemon command along with the ceph flags which need arguments to be passed , its buggy with them too ( not with all but some of them atleast). for eg: ./bin/ceph --name invalid_client_name daemon mds.c client evict It will execute the command client evict even if the passed client name doesn't exist.

I guess this happens since the asok command is tried before even connecting (and authenticating the user) to the cluster.

EDIT: BTW, this has helped us in situations where for some reason its not possible to connect to the cluster but we want to invoke a command (in the mds), so using asok was immensely helpful.

@vshankar
Copy link
Contributor

Looks like we never made progress on this since nothing was finalised w.r.t. the approach to fix this. I've also lost some context on this - will revisit asap.

@vshankar
Copy link
Contributor

Discussed this with @neesingh-rh - One approach would be to add --yes-i-really-mean-it to client evict command without which clients are not evicted (when no specific client-id is provided). Does that seem reasonable to everyone?

@idryomov
Copy link
Contributor

One approach would be to add --yes-i-really-mean-it to client evict command without which clients are not evicted (when no specific client-id is provided). Does that seem reasonable to everyone?

Yup, sounds reasonable to me.

@github-actions github-actions bot added the cephfs Ceph File System label Jun 27, 2023
@neesingh-rh neesingh-rh changed the title ceph.in: admin_socket commands run ignoring the passed ceph flags. mds: adding '--yes_i_really_mean_it' flag to session/client evict command. Jun 27, 2023
@dparmar18
Copy link
Contributor

Would be good to have tests for this

@github-actions github-actions bot added the tests label Jun 27, 2023
@neesingh-rh neesingh-rh force-pushed the wip-58619 branch 4 times, most recently from 77370fd to c18004b Compare June 27, 2023 11:15
Copy link
Member

@batrick batrick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please ping me on slack when you've addressed this.

self.assertEqual(len(info), 0) # multiple clients are evicted

def test_session_evict(self):
self._session_client_evict_without_filter(['session'])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something like:

@classhook('_add_session_evictions')
class TestSessionClientEvict
...
    def _add_session_evictions(cls):
        tests = [
"_session_client_evict_without_filter"
...
]
      def create_test(t, cmd):
        def test(self):
          getattr(self, t)([cmd])
      for t in tests:
        setattr(cls, 'test_session_'+t, create_test(t, ['session']))
        setattr(cls, 'test_client_'+t, create_test(t, ['client']))

(fix formatting but that's roughly what it'd look like)

@neesingh-rh neesingh-rh requested a review from a team as a code owner June 6, 2024 09:45
@neesingh-rh neesingh-rh requested review from Pegonzal and nizamial09 and removed request for a team June 6, 2024 09:45
@neesingh-rh neesingh-rh removed the request for review from a team June 6, 2024 09:48
Copy link
Member

@batrick batrick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mds: fix client evict command needs to explain the change. The title would be better as mds: require filter for evict command.

qa: delete the symbolic link 1-mds-2-client.yaml needs an explanation in the commit message.

qa: add test for fix of client/session evict command looks good.

Please post results from a vstart_runner.py run to confirm the QA tests work as expected before requesting general QA.

neeraj pratap singh added 3 commits June 8, 2024 08:09
This commit fixes the issues in client/session evict command
using below solutions:
(1) client evict without filter is forbidden
(2) SessionFilter::parse is modified to support id=* (id=0 is not allowed)
(3) Invalid id error is handled
User can use client evict id=* to evict all clients.

Fixes:  https://tracker.ceph.com/issues/58619
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>
Since the tests in the class test_misc.TestSessionClientEvict
requires 3 clients, this should be run with a yaml that
configures 3 clients. Hence, removing the link to 1-mds-2-client.yaml
from qa/suits/fs/multiclient/clusters/

Fixes: https://tracker.ceph.com/issues/58619
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>
Adds a test class test_misc.TestSessionClientEvict
which contains test for the issues mentioned in this PR.

Fixes: https://tracker.ceph.com/issues/58619
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>
@neesingh-rh
Copy link
Contributor Author

vstart_runner test results:

2024-06-08 08:26:59,432.432 INFO:__main__:test_session_evict_all_clients (tasks.cephfs.test_misc.TestSessionClientEvict.test_session_evict_all_clients) ... ok
2024-06-08 08:26:59,432.432 INFO:__main__:test_session_evict_with_id_zero (tasks.cephfs.test_misc.TestSessionClientEvict.test_session_evict_with_id_zero) ... ok
2024-06-08 08:26:59,432.432 INFO:__main__:test_session_evict_with_invalid_id (tasks.cephfs.test_misc.TestSessionClientEvict.test_session_evict_with_invalid_id) ... ok
2024-06-08 08:26:59,432.432 INFO:__main__:test_session_evict_with_negative_id (tasks.cephfs.test_misc.TestSessionClientEvict.test_session_evict_with_negative_id) ... ok
2024-06-08 08:26:59,432.432 INFO:__main__:test_session_evict_with_valid_id (tasks.cephfs.test_misc.TestSessionClientEvict.test_session_evict_with_valid_id) ... ok
2024-06-08 08:26:59,432.432 INFO:__main__:test_session_evict_without_filter (tasks.cephfs.test_misc.TestSessionClientEvict.test_session_evict_without_filter) ... ok
2024-06-08 08:26:59,432.432 INFO:__main__:
2024-06-08 08:26:59,432.432 INFO:__main__:Stopped test: test_session_evict_without_filter (tasks.cephfs.test_misc.TestSessionClientEvict.test_session_evict_without_filter) in 39.570571s

2024-06-08 08:31:55,173.173 INFO:__main__:test_client_evict_all_clients (tasks.cephfs.test_misc.TestSessionClientEvict.test_client_evict_all_clients) ... ok
2024-06-08 08:31:55,173.173 INFO:__main__:test_client_evict_with_id_zero (tasks.cephfs.test_misc.TestSessionClientEvict.test_client_evict_with_id_zero) ... ok
2024-06-08 08:31:55,173.173 INFO:__main__:test_client_evict_with_invalid_id (tasks.cephfs.test_misc.TestSessionClientEvict.test_client_evict_with_invalid_id) ... ok
2024-06-08 08:31:55,173.173 INFO:__main__:test_client_evict_with_negative_id (tasks.cephfs.test_misc.TestSessionClientEvict.test_client_evict_with_negative_id) ... ok
2024-06-08 08:31:55,173.173 INFO:__main__:test_client_evict_with_valid_id (tasks.cephfs.test_misc.TestSessionClientEvict.test_client_evict_with_valid_id) ... ok
2024-06-08 08:31:55,173.173 INFO:__main__:test_client_evict_without_filter (tasks.cephfs.test_misc.TestSessionClientEvict.test_client_evict_without_filter) ... ok
2024-06-08 08:31:55,173.173 INFO:__main__:
2024-06-08 08:31:55,173.173 INFO:__main__:Stopped test: test_client_evict_without_filter (tasks.cephfs.test_misc.TestSessionClientEvict.test_client_evict_without_filter) in 34.422433s

@batrick
Copy link
Member

batrick commented Jun 10, 2024

jenkins test api

@neesingh-rh
Copy link
Contributor Author

jenkins test api

Copy link
Contributor

@vshankar vshankar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (TIL about classhook :)

@vshankar
Copy link
Contributor

This PR is under test in https://tracker.ceph.com/issues/66521.

Copy link
Contributor

@vshankar vshankar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants