What did you do?
I am deploying Elastic using the elastic operator.
When deploying the Elastic custom resource the operator creates elastic pods, as well as secrets that contain certificates and the password to the instance.
I want to scrape metrics from the Elastic instance, so I deploy a serviceMonitor to instruct the prometheus operator to do this.
The serviceMonitor needs the username, password and CA certificate in order to connect to the elastic instance.
This information is contained within secrets that is created by the elastic operator.

I am deploying both the elastic custom resource and the ServiceMonitor using helm.
After applying the elastic custom resource it takes a little while for the operator to deploy the elastic pods and create the secrets containing the certificated and password for the instance.
This means that the ServiceMonitor is created before these secrets exists.
When this happens prometheus disables the ServiceMonitor and does not re-try it.

level=warn ts=2023-10-17T11:32:39.147133443Z caller=operator.go:2255 component=prometheusoperator msg="skipping servicemonitor" error="failed to get basic auth username: unable to get secret \"prometheus-elastic-basic-auth-username\": secrets \"prometheus-elastic-basic-auth-username\" not found" servicemonitor=NAMESPACE/elk-prometheus-metrics namespace=CLUSTER-NAMESPACE prometheus=user-workload

A possible solution to this is to deploy the secrets using helm, but empty. The operator will then update the secrets with the correct information later.
I did this, but got another error

ts=2023-10-18T09:53:44.698Z caller=manager.go:216 level=error component="scrape manager" msg="error creating new scrape pool" 
err="error creating HTTP client: unable to load specified CA cert /etc/prometheus/certs/secret_NAMESPACE_-s-http-ca-internal_tls.crt: 
open /etc/prometheus/certs/secret_NAMESPACE-es-http-ca-internal_tls.crt: no such file or directory" scrape_pool=serviceMonitor/NAMESPACE/elk-prometheus-metrics-data/0

Did you expect to see some different?

First I expected prometheus to re-try adding the ServiceMonitor even after the secret did not exist at the first attempt.
This would be the best solution and would require no work-around on my part.

I then have a question. If I add empty secrets instead, will prometheus re-load the info from the secrets when the content changes? It looks like prometheus mounts the secret into itself as a file and reads it. When the secret updates it will not automatically reflect on the prometheus server as it needs to reload the secret. Is there a mechanism for this?
And does this mechanism work, even if it fails to load the certificate the first attempt because it is empty?

Environment

• Prometheus Operator version:

operator: 0.60.1
prometheus: 2.39.1

• Kubernetes version information:

v1.25.4+a34b9e9

• Kubernetes cluster kind:

  OpenShift

• Manifests:

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: elk-prometheus-metrics-master
  labels:
    app: elk
spec:
  selector:
    matchLabels:
      common.k8s.elastic.co/type: elasticsearch
      elasticsearch.k8s.elastic.co/statefulset-name: {{ .Release.Name }}-es-master
  endpoints:
    - path: "/_prometheus/metrics"
      port: https
      scheme: https
      basicAuth:
        password:
          name: {{ .Release.Name }}-es-elastic-user
          key: elastic
        username:
          name: prometheus-elastic-basic-auth-username
          key: username
      tlsConfig:
        ca: 
          secret: 
            name: {{ .Release.Name }}-es-http-ca-internal
            key: tls.crt
        serverName: {{ .Release.Name }}-es-http.{{ .Release.Namespace }}.es.local

Anything else we need to know?: