Ceph » mgr

/a/teuthology-2025-12-07_20:00:23-rados-main-distro-default-smithi/8644597

2025-12-07T21:26:39.859 INFO:teuthology.orchestra.run.smithi060.stderr:+ jq -e '.status == "success"'
2025-12-07T21:26:39.863 INFO:teuthology.orchestra.run.smithi060.stdout:{"status":"success","data":{"yaml":"global:\n  scrape_interval: 10s\n  scrape_timeout: 10s\n  scrape_protocols:\n  - OpenMetricsText1.0.0\n  - OpenMetricsText0.0.1\n  - PrometheusText1.0.0\n  - PrometheusText0.0.4\n  evaluation_interval: 10s\n  external_labels:\n    cluster: e8525cbe-d3b1-11f0-87af-adfe0268badd\nruntime:\n  gogc: 75\nalerting:\n  alertmanagers:\n  - follow_redirects: true\n    enable_http2: true\n    scheme: http\n    timeout: 10s\n    api_version: v2\n    http_sd_configs:\n    - follow_redirects: true\n      enable_http2: true\n      refresh_interval: 1m\n      url: http://172.21.15.60:8765/sd/prometheus/sd-config?service=alertmanager\n    - follow_redirects: true\n      enable_http2: true\n      refresh_interval: 1m\n      url: http://172.21.15.99:8765/sd/prometheus/sd-config?service=alertmanager\nrule_files:\n- /etc/prometheus/alerting/*\nscrape_configs:\n- job_name: ceph\n  honor_labels: true\n  honor_timestamps: true\n  track_timestamps_staleness: false\n  scrape_interval: 10s\n  scrape_timeout: 10s\n  scrape_protocols:\n  - OpenMetricsText1.0.0\n  - OpenMetricsText0.0.1\n  - PrometheusText1.0.0\n  - PrometheusText0.0.4\n  always_scrape_classic_histograms: false\n  convert_classic_histograms_to_nhcb: false\n  metrics_path: /metrics\n  scheme: http\n  enable_compression: true\n  metric_name_validation_scheme: utf8\n  metric_name_escaping_scheme: allow-utf-8\n  follow_redirects: true\n  enable_http2: true\n  relabel_configs:\n  - source_labels: [__address__]\n    separator: ;\n    target_label: cluster\n    replacement: e8525cbe-d3b1-11f0-87af-adfe0268badd\n    action: replace\n  - source_labels: [instance]\n    separator: ;\n    target_label: instance\n    replacement: ceph_cluster\n    action: replace\n  http_sd_configs:\n  - follow_redirects: true\n    enable_http2: true\n    refresh_interval: 1m\n    url: http://172.21.15.60:8765/sd/prometheus/sd-config?service=ceph\n  - follow_redirects: true\n    enable_http2: true\n    refresh_interval: 1m\n    url: http://172.21.15.99:8765/sd/prometheus/sd-config?service=ceph\n- job_name: node-exporter\n  honor_labels: true\n  honor_timestamps: true\n  track_timestamps_staleness: false\n  scrape_interval: 10s\n  scrape_timeout: 10s\n  scrape_protocols:\n  - OpenMetricsText1.0.0\n  - OpenMetricsText0.0.1\n  - PrometheusText1.0.0\n  - PrometheusText0.0.4\n  always_scrape_classic_histograms: false\n  convert_classic_histograms_to_nhcb: false\n  metrics_path: /metrics\n  scheme: http\n  enable_compression: true\n  metric_name_validation_scheme: utf8\n  metric_name_escaping_scheme: allow-utf-8\n  follow_redirects: true\n  enable_http2: true\n  relabel_configs:\n  - source_labels: [__address__]\n    separator: ;\n    target_label: cluster\n    replacement: e8525cbe-d3b1-11f0-87af-adfe0268badd\n    action: replace\n  http_sd_configs:\n  - follow_redirects: true\n    enable_http2: true\n    refresh_interval: 1m\n    url: http://172.21.15.60:8765/sd/prometheus/sd-config?service=node-exporter\n  - follow_redirects: true\n    enable_http2: true\n    refresh_interval: 1m\n    url: http://172.21.15.99:8765/sd/prometheus/sd-config?service=node-exporter\notlp:\n  translation_strategy: UnderscoreEscapingWithSuffixes\n"}}true
2025-12-07T21:26:39.863 INFO:teuthology.orchestra.run.smithi060.stderr:+ curl -s http://172.21.15.99:9095/api/v1/alerts
2025-12-07T21:26:39.868 INFO:teuthology.orchestra.run.smithi060.stderr:+ curl -s http://172.21.15.99:9095/api/v1/alerts
2025-12-07T21:26:39.868 INFO:teuthology.orchestra.run.smithi060.stderr:+ jq -e '.data | .alerts | .[] | select(.labels | .alertname == "CephMonDown") | .state == "firing"'
2025-12-07T21:26:40.488 DEBUG:teuthology.orchestra.run:got remote process result: 4
2025-12-07T21:26:40.488 INFO:teuthology.orchestra.run.smithi060.stdout:{"status":"success","data":{"alerts":[{"labels":{"alertname":"CephMgrPrometheusModuleInactive","cluster":"e8525cbe-d3b1-11f0-87af-adfe0268badd","instance":"ceph_cluster","job":"ceph","oid":"1.3.6.1.4.1.50495.1.2.1.6.2","severity":"critical","type":"ceph_default"},"annotations":{"description":"The mgr/prometheus module at ceph_cluster is unreachable. This could mean that the module has been disabled or the mgr daemon itself is down. Without the mgr/prometheus module metrics and alerts will no longer function. Open a shell to an admin node or toolbox pod and use 'ceph -s' to to determine whether the mgr is active. If the mgr is not active, restart it, otherwise you can determine module status with 'ceph mgr module ls'. If it is not listed as enabled, enable it with 'ceph mgr module enable prometheus'.","summary":"The mgr/prometheus module is not available"},"state":"firing","activeAt":"2025-12-07T21:21:33.245200013Z","value":"0e+00"}]}}
2025-12-07T21:26:40.490 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_258eb6279f4d7fcd4b45c82e521f2a2e799d7f33/teuthology/run_tasks.py", line 105, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_258eb6279f4d7fcd4b45c82e521f2a2e799d7f33/teuthology/run_tasks.py", line 83, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_ceph_6ce249e0e13e12a74d5c855ed12d6b50671977c9/qa/tasks/cephadm.py", line 1467, in shell
    _shell(
  File "/home/teuthworker/src/git.ceph.com_ceph_6ce249e0e13e12a74d5c855ed12d6b50671977c9/qa/tasks/cephadm.py", line 41, in _shell
    return remote.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_258eb6279f4d7fcd4b45c82e521f2a2e799d7f33/teuthology/orchestra/remote.py", line 575, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_258eb6279f4d7fcd4b45c82e521f2a2e799d7f33/teuthology/orchestra/run.py", line 461, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_258eb6279f4d7fcd4b45c82e521f2a2e799d7f33/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_258eb6279f4d7fcd4b45c82e521f2a2e799d7f33/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi060 with status 4: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:6ce249e0e13e12a74d5c855ed12d6b50671977c9 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e8525cbe-d3b1-11f0-87af-adfe0268badd -- bash -c \'set -e\nset -x\nceph orch apply node-exporter\nceph orch apply grafana\nceph orch apply alertmanager\nceph orch apply prometheus\nsleep 240\nceph orch ls\nceph orch ps\nceph orch host ls\nMON_DAEMON=$(ceph orch ps --daemon-type mon -f json | jq -r \'"\'"\'last | .daemon_name\'"\'"\')\nGRAFANA_HOST=$(ceph orch ps --daemon-type grafana -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nPROM_HOST=$(ceph orch ps --daemon-type prometheus -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nALERTM_HOST=$(ceph orch ps --daemon-type alertmanager -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nGRAFANA_IP=$(ceph orch host ls -f json | jq -r --arg GRAFANA_HOST "$GRAFANA_HOST" \'"\'"\'.[] | select(.hostname==$GRAFANA_HOST) | .addr\'"\'"\')\nPROM_IP=$(ceph orch host ls -f json | jq -r --arg PROM_HOST "$PROM_HOST" \'"\'"\'.[] | select(.hostname==$PROM_HOST) | .addr\'"\'"\')\nALERTM_IP=$(ceph orch host ls -f json | jq -r --arg ALERTM_HOST "$ALERTM_HOST" \'"\'"\'.[] | select(.hostname==$ALERTM_HOST) | .addr\'"\'"\')\n# check each host node-exporter metrics endpoint is responsive\nALL_HOST_IPS=$(ceph orch host ls -f json | jq -r \'"\'"\'.[] | .addr\'"\'"\')\nfor ip in $ALL_HOST_IPS; do\n  curl -s http://${ip}:9100/metric\ndone\n# check grafana endpoints are responsive and database health is okay\ncurl -k -s https://${GRAFANA_IP}:3000/api/health\ncurl -k -s https://${GRAFANA_IP}:3000/api/health | jq -e \'"\'"\'.database == "ok"\'"\'"\'\n# stop mon daemon in order to trigger an alert\nceph orch daemon stop $MON_DAEMON\nsleep 120\n# check prometheus endpoints are responsive and mon down alert is firing\ncurl -s http://${PROM_IP}:9095/api/v1/status/config\ncurl -s http://${PROM_IP}:9095/api/v1/status/config | jq -e \'"\'"\'.status == "success"\'"\'"\'\ncurl -s http://${PROM_IP}:9095/api/v1/alerts\ncurl -s http://${PROM_IP}:9095/api/v1/alerts | jq -e \'"\'"\'.data | .alerts | .[] | select(.labels | .alertname == "CephMonDown") | .state == "firing"\'"\'"\'\n# check alertmanager endpoints are responsive and mon down alert is active\ncurl -s http://${ALERTM_IP}:9093/api/v2/status\ncurl -s http://${ALERTM_IP}:9093/api/v2/alerts\ncurl -s http://${ALERTM_IP}:9093/api/v2/alerts | jq -e \'"\'"\'.[] | select(.labels | .alertname == "CephMonDown") | .status | .state == "active"\'"\'"\'\n# check prometheus metrics endpoint is not empty and make sure we can get metrics\nMETRICS_URL=$(

/a/teuthology-2025-12-07_20:00:23-rados-main-distro-default-smithi/8644597/remote/smithi060/log/e8525cbe-d3b1-11f0-87af-adfe0268badd/ceph-mgr.a.log.gz

2025-12-07T21:26:41.895+0000 7ff8f7017640  0 [prometheus INFO cherrypy.error] [07/Dec/2025:21:26:41] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down
2025-12-07T21:26:41.895+0000 7ff8f7017640  0 [prometheus INFO cherrypy.error] [07/Dec/2025:21:26:41] ENGINE Bus STOPPED

I suspect this PR: https://github.com/ceph/ceph/pull/65245