We get regular requests from users to add more fields to the Prometheus, Alertmanager and Thanos Ruler CRDs. The situation is particularly acute for fields that exist in the statefulset's pod template spec.

As of today, the following pod template's fields are exposed by the CRDs:

• affinity
• containers (via strategic merge patch)
• hostNetwork
  • alertmanager: add hostNetwork support #8281
  • feat : enable support for "hostNetwork" in Prometheus CRD #5010
• imagePullSecrets
• initContainers (via strategic merge patch)
• nodeSelector
• priorityClassName
• securityContext
• serviceAccountName
• tolerations
• topologySpreadConstraints
• volumes
• dnsConfig
  • Add dnsConfig parameter on PrometheusSpec and to AlertmanagerSpec #3836
  • Add ability for custom DNSConfig and DNSPolicy #3899
• dnsPolicy
  • Add ability for custom DNSConfig and DNSPolicy #3899
• schedulerName
  • Allow to define custom scheduler in the pod definition #4023
• hostAliases
  • Add an option to add hostAlias for alert manager #3488
  • pkg/alertmanager: added hostAliases to pod spec (fixes #3488) #3944
  • Add hostAlias support in Prometheus CRD #4298
  • Support StatefulSet patching #4637
• enableServiceLinks
  • feat: support pod spec field "enableServiceLinks" (PR included) #7383
  • feat: support pod spec field "enableServiceLinks" (#7383) #7384

The following fields are controlled by the operator and their values are hardcoded:

• automountServiceAccountToken (always true)
• dnsPolicy (when hostNetwork is true)
• terminationGracePeriodSeconds (600s)
  • Make termination grace seconds configurable #4681
  • hope add terminationGracePeriodSeconds support #3433

The following fields are not exposed by the CRD and not controlled by the operator:

• activeDeadlineSeconds
• ephemeralContainers
• hostIPC
• hostPID
• hostname
• nodeName
• overhead
• preemptionPolicy
• priority (priorityClassName instead)
• readinessGates
• restartPolicy
• runtimeClassName
• setHostnameAsFQDN
• shareProcessNamespace
• subdomain

As we can see, we've got requests for customizing a few fields but not the majority of them.

We've discussed possible options to circumvent the problem:

• Add new fields to the Prometheus, Alertmanager and Thanos Ruler CRDs on a case-by-case basis (e.g. the demand is backed by a legitimate scenario and no workaround exists).
  • Pros
    • First-class support by the project.
    • Easy to use and document.
  • Cons
    • Users have to convince the maintainers that the addition is worthwhile.
    • Additional burden on the maintainers who aren't necessary familiar with the details of how the field is going to be used.
• Offer a mechanism where users could patch the template generated by the operator. This would be similar to the initContainers and containers fields but for the pod template.
  • Pros
    • Total flexibility.
    • One-time operation.
  • Cons
    • Easier for users to get things wrong. It requires deeper knowledge of the operator and it may trigger more support requests.
    • The size of the CRDs increases by a lot.
    • No guarantee that it won't break you in the future (this would be marked as experimental).
    • Redundancy with the initContainers and containers fields.
• Use the server-side apply pattern
  • Pros
    • Aligned with the latest Kubernetes practices.
  • Cons
    • If/when the statefulset is recreated, the user changes are lost and need to be reconciled again.

cc @prometheus-operator/prometheus-operator-reviewers