Skip to content

mgr/apicli: add new API CLI module#34840

Closed
epuertat wants to merge 1 commit intoceph:mainfrom
rhcs-dashboard:wip-45264-master
Closed

mgr/apicli: add new API CLI module#34840
epuertat wants to merge 1 commit intoceph:mainfrom
rhcs-dashboard:wip-45264-master

Conversation

@epuertat
Copy link
Member

@epuertat epuertat commented Apr 29, 2020

Fixes: https://tracker.ceph.com/issues/45264

Sample output:

# ceph cli -h
 Monitor commands: 
 =================
cli cluster_log <channel> <priority:int> <message>   Send message to the cluster log. :param channel: The 
                                                      log channel. This can be 'cluster', 'audit', ... :
                                                      type channel: str :param priority: The log message 
                                                      priority. This can be CLUSTER_LOG_PRIO_DEBUG, 
                                                      CLUSTER_LOG_PRIO_INFO, CLUSTER_LOG_PRIO_SEC, 
                                                      CLUSTER_LOG_PRIO_WARN or CLUSTER_LOG_PRIO_ERROR. :
                                                      type priority: int :param message: The message to 
                                                      log. :type message: str
cli get <data_name>                                  Called by the plugin to fetch named cluster-wide 
                                                      objects from ceph-mgr. :param str data_name: Valid 
                                                      things to fetch are osd_crush_map_text, osd_map, 
                                                      osd_map_tree, osd_map_crush, config, mon_map, fs_
                                                      map, osd_metadata, pg_summary, io_rate, pg_dump, df,
                                                      osd_stats, health, mon_status, devices, device 
                                                      <devid>, pg_stats, pool_stats, pg_ready, osd_ping_
                                                      times. Note: All these structures have their own 
                                                      JSON representations: experiment or look at the C++ 
                                                      ``dump()`` methods to learn about them.
cli get_all_perf_counters [<prio_limit:int>]         Return the perf counters currently known to this 
 [<services>]                                         ceph-mgr instance, filtered by priority equal to or 
                                                      greater than `prio_limit`. The result is a map of 
                                                      string to dict, associating services (like "osd.123"
                                                      ) with their counters. The counter dict for each 
                                                      service maps counter paths to a counter info 
                                                      structure, which is the information from the schema,
                                                      plus an additional "value" member with the latest 
                                                      value.

Signed-off-by: Ernesto Puerta epuertat@redhat.com

Checklist

  • References tracker ticket
  • Updates documentation if necessary
  • Includes tests for new functionality or reproducer for bug

Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard backend
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox

setattr(
klass,
f'_cli_{name}',
CLICommand(**CephCommander(func).to_ceph_signature())(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like CLICommand, as it has global side effects that prevents me from using it. I had to build my own wrapper to work around the problems:

https://github.com/ceph/ceph/search?q=CLICommandMeta&unscoped_q=CLICommandMeta

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I don't like it either, but it was there and wanted to reuse as much code as possible for this module.

Comment on lines +17 to +21
class CephCommander:
"""
Utility class to inspect Python functions and generate corresponding
CephCommand signatures (see src/mon/MonCommand.h for details)
"""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like forever, I wanted to have this class to automatically convert python signatures to mon commands.

Would be great to have this also work standalone and directly usable for me.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably we could take this one and came up with an alternative to @CLICommand that auto-guesses the signature (when Ricardo wrote CLICommand we couldn't use yet type annotations).

Comment on lines +105 to +121
class CLI(MgrModule, metaclass=MgrAPIReflector):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.stop = Event()

def serve(self):
logger.info("Starting")
while not self.stop.wait(self.get_ceph_option('mgr_tick_period')):
pass

def shutdown(self):
"""
This method is called by the mgr when the module needs to shut
down (i.e., when the serve() function needs to exit).
"""
logger.info('Stopping')
self.stop.set()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a against introducing a huge bunch of complexity and a whole new module for just printing some documentation. We've already plenty of modules already.

This seems redundant to https://docs.ceph.com/docs/master/mgr/modules/ .

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean about this module? This module is not for documenting anything (while it may provide docs, and more updated than that page), but to allow usage of the internal Ceph-mgr Python API from the CLI: either for development purposes (new module PoC or extension) or for debugging.

For example, we may explore the undocumented get('modified_config_options'):

# ceph cli get modified_config_options
{
    "options": [
        "admin_socket",
        "auth_client_required",
        "auth_cluster_required",
        "auth_service_required",
        "bluestore_block_create",
        "bluestore_block_db_create",
        "bluestore_block_db_path",
        "bluestore_block_db_size",
        "bluestore_block_wal_create",
        "bluestore_block_wal_path",
        "bluestore_block_wal_size",
        "bluestore_fsck_on_mount",
        "cephadm_path",
        "chdir",
        "crash_dir",
        "debug_asok_assert_abort",
        "debug_mon",
        "debug_ms",
        "enable_experimental_unrecoverable_data_corrupting_features",
        "erasure_code_dir",
        "filestore_fd_cache_size",
        "filestore_wbthrottle_btrfs_inodes_hard_limit",
        "filestore_wbthrottle_btrfs_ios_hard_limit",
        "filestore_wbthrottle_btrfs_ios_start_flusher",
        "filestore_wbthrottle_xfs_inodes_hard_limit",
        "filestore_wbthrottle_xfs_ios_hard_limit",
        "filestore_wbthrottle_xfs_ios_start_flusher",
        "heartbeat_file",
        "keyring",
        "kstore_fsck_on_mount",
        "leveldb_block_size",
        "leveldb_cache_size",
        "leveldb_compression",
        "leveldb_log",
        "leveldb_write_buffer_size",
        "lockdep",
        "log_file",
        "mgr_data",
        "mgr_initial_modules",
        "mgr_module_path",
        "mon_allow_pool_delete",
        "mon_allow_pool_size_one",
        "mon_cluster_log_file",
        "mon_data",
        "mon_data_avail_crit",
        "mon_data_avail_warn",
        "mon_host",
        "mon_max_pg_per_osd",
        "mon_osd_backfillfull_ratio",
        "mon_osd_full_ratio",
        "mon_osd_nearfull_ratio",
        "mon_osd_reporter_subtree_level",
        "ms_bind_msgr1",
        "ms_bind_msgr2",
        "no_config_file",
        "osd_check_max_object_name_len_on_startup",
        "osd_class_default_list",
        "osd_class_dir",
        "osd_class_load_list",
        "osd_copyfrom_max_chunk",
        "osd_crush_chooseleaf_type",
        "osd_data",
        "osd_debug_misdirected_ops",
        "osd_debug_op_order",
        "osd_failsafe_full_ratio",
        "osd_fast_shutdown",
        "osd_journal",
        "osd_journal_size",
        "osd_objectstore",
        "osd_pool_default_erasure_code_profile",
        "osd_pool_default_min_size",
        "osd_pool_default_size",
        "osd_scrub_load_threshold",
        "pid_file",
        "plugin_dir",
        "rbd_default_features",
        "run_dir"
    ]
}

Fixes: https://tracker.ceph.com/issues/45264
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
@stale stale bot added the stale label Aug 23, 2020
@ceph ceph deleted a comment from stale bot Aug 24, 2020
@stale stale bot removed the stale label Aug 24, 2020
@stale
Copy link

stale bot commented Oct 31, 2020

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

@stale stale bot added the stale label Oct 31, 2020
@github-actions
Copy link

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

@stale
Copy link

stale bot commented Jul 21, 2021

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

@stale stale bot added the stale label Jul 21, 2021
@djgalloway djgalloway changed the base branch from master to main July 9, 2022 00:00
@github-actions github-actions bot removed the stale label Jul 15, 2022
@github-actions
Copy link

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

@github-actions github-actions bot added the stale label Sep 13, 2022
@epuertat epuertat closed this Sep 13, 2022
@epuertat epuertat deleted the wip-45264-master branch September 13, 2022 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants