TODO

• review default SSL/TLS configuration
• unify cephadm root CA management
  (addressed by mgr/cephadm: adding mTLS for ceph mgmt-gateway and backend services communication #58402)
• fix prometheus/alertmanager access when secure_monitoring_stack is enabled
  (addressed by mgr/cephadm: adding mTLS for ceph mgmt-gateway and backend services communication #58402)
• fix the nginx image (now image from quay is used)

This pull request introduces a new design for Ceph applications based on a modular, service-based architecture. A new cephadm service mgmt-gateway based on nginx an open-source, high-performance web server known for its scalability, efficiency, and versatility. It will act as the new front-end and single point entry to the cluster, providing unified access to all Ceph applications, including the Dashboard and monitoring applications. In addition, Nginx enhances security and simplifies access management due to its robust community and high security standards.

Benefits of the new service

• Unified Access: Centralizing access through Nginx improves security and provide a single entry to the cluster mgmt.
• Improved user experience: User shouldn't care anymore about where each application is running (ip/host).
• High Availability for dashboard: Nginx HA mechanisms are used to provide high availability for ceph dashboard.
• High Availability for monitoring: Nginx HA mechanisms are used to provide high availability for monitoring.

High availability enhancements

The current cephadm/dashboard implementation lacks HA when it comes to monitoring services. Even when cephadm is able to deploy N instances of services such as grafana, prometheus or alertmanager when configuring the dashboard (using dashboard set-<service>-api-host API) it just picks the last configured daemon. In case this daemons goes down there's no automated fail-over to use redundant healthy instance. The following diagram reflects the current architecture (notice dashboard is configured to access directly the different monitoring services).

This problem is solved by using upstream HA features provided by nginx. The proposed solution makes sure of a dedicated internal server to act as rev-proxy for monitoring services. Dashboard is configured to use nginx end-points instead of using directly ip/host of the monitoring daemons. Following is a diagram of the new architecture:

As we can see in the above diagram, in the new architecture there are two servers:

• External server: this server is responsible of attending and routing external user requests. The idea is for this server is use it also for any extra processing we would like to perform for external users such as authentication, authorization, etc. This server relies on nginx upstream feature to group the monitoring applications (by category). HA mechanism is implemented by selecting one of the available healthy servers.

• Internal server: this server is responsible of attending and routing internal requests only. Similarly to the external case, this server relies on nginx upstream feature to provide monitoring HA this time for internal services. This server uses its own self-signed certificates to secure the communication with other internal clients.

Usage

cephadm:
ceph orch apply mgmt-gateway --placement=<your-destination-node>

Or by providing a detailed spec file (for custom certificates i.e):

service_type: mgmt-gateway
placement:
  hosts:
    - ceph-node-1
spec:
 port: 9443
 ssl_protocols:
   - TLSv1.2
   - TLSv1.3
 ssl_ciphers:
   - AES128-SHA
   - AES256-SHA
   - RC4-SHA
 ssl_certificate: |
   -----BEGIN CERTIFICATE-----
   < YOU CERT DATA HERE >
   -----END CERTIFICATE-----
 ssl_certificate_key: |
  -----BEGIN RSA PRIVATE KEY-----
   < YOU PRIV KEY DATA HERE >
  -----END RSA PRIVATE KEY-----

Example of the generated nginx config:

[root@ceph-node-0 ~]# cat /var/lib/ceph/bd861cc8-28ae-11ef-92e9-525400cfad85/mgmt-gateway.ceph-node-0/etc/nginx_external_server.conf 

server {
    listen                    443 ssl;
    listen                    [::]:443 ssl;
    ssl_certificate            /etc/nginx/ssl/nginx.crt;
    ssl_certificate_key /etc/nginx/ssl/nginx.key;
    ssl_protocols            TLSv1.2 TLSv1.3;
    # from:  https://ssl-config.mozilla.org/#server=nginx
    ssl_ciphers              ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305;

    # Only return Nginx in server header, no extra info will be provided
    server_tokens             off;

    # Perfect Forward Secrecy(PFS) is frequently compromised without this
    ssl_prefer_server_ciphers on;

    # Enable SSL session caching for improved performance
    ssl_session_tickets       off;
    ssl_session_timeout       1d;
    ssl_session_cache         shared:SSL:10m;

    # OCSP stapling
    ssl_stapling              on;
    ssl_stapling_verify       on;
    resolver_timeout 5s;

    # Security headers
    ## X-Content-Type-Options: avoid MIME type sniffing
    add_header X-Content-Type-Options nosniff;
    ## Strict Transport Security (HSTS): Yes
    add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload";
    ## Enables the Cross-site scripting (XSS) filter in browsers.
    add_header X-XSS-Protection "1; mode=block";
    ## Content-Security-Policy (CSP): FIXME
    # add_header Content-Security-Policy "default-src 'self'; script-src 'self'; object-src 'none'; base-uri 'none'; require-trusted-types-for 'script'; frame-ancestors 'self';";



    location / {
        proxy_pass https://dashboard_servers;
        proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
    }

    location /grafana {
        rewrite ^/grafana/(.*) /$1 break;
        proxy_pass https://grafana_servers;
    }

    location /prometheus {
        proxy_pass http://prometheus_servers;
    }

    location /alertmanager {
        proxy_pass http://alertmanager_servers;
    }
}

Contribution Guidelines

• To sign and title your commits, please refer to Submitting Patches to Ceph.

• If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

• When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

• Tracker (select at least one)
  • References tracker ticket
  • Very recent bug; references commit where it was introduced
  • New feature (ticket optional)
  • Doc update (no ticket needed)
  • Code cleanup (no ticket needed)
• Component impact
  • Affects Dashboard, opened tracker ticket
  • Affects Orchestrator, opened tracker ticket
  • No impact that needs to be tracked
• Documentation (select at least one)
  • Updates relevant documentation
  • No doc update is appropriate
• Tests (select at least one)
  • Includes unit test(s)
  • Includes integration test(s)
  • Includes bug reproducer
  • No tests