Motivation

In a Kubernetes production with Istio and prometheus metrics.

• Istio "installs" a sidecar (also known as istio-proxy) that intercepts inbound and outbound traffic
• Istio is used to monitor the success rate from outside the JVM
• Servicemonitor (Prometheus) is configured to look at /actuator/prometheus

We noticed that servicemonitor starts trying to fetch prometheus metrics very early, before the application is ready, this results in a noticeable 503 during the rollout.

Since grabbing the metrics is a separate concern than serving metrics, we wanted to expose those on a different port in order to exclude the port from Istio.

    annotations:
      # Make Istio not listening on management.server.port 8081
      traffic.sidecar.istio.io/excludeInboundPorts: "8081"

This is possible as documented here : https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-features.html#production-ready-customizing-management-server-port

There's always the possibility to change the management (actuator) port, but the documentation actually warns about trusting the health endpoints with this setup: https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-features.html#production-ready-kubernetes-probes

If your Actuator endpoints are deployed on a separate management context, be aware that endpoints are then not using the same web infrastructure (port, connection pools, framework components) as the main application. In this case, a probe check could be successful even if the main application does not work properly (for example, it cannot accept new connections).

Suggestion

Could it be worth to distinguish two class of actuators ?

1. The ones that represent the health of the service, and exposed on they same port, connection pool, framework than the service.
2. The others that are here to provide other features, like metrics, and can be declared to use a different web infrastructure without health endpoint.