Skip to content

[Bug]: Operator deletes PodMonitor that it didn't create due to lack of owner reference check #6109

@ari-becker

Description

@ari-becker

Is there an existing issue already for this bug?

  • I have searched for an existing issue, and could not find anything. I believe this is a new bug.

I have read the troubleshooting guide

  • I have read the troubleshooting guide and I think this is a new bug.

I am running a supported version of CloudNativePG

  • I have read the troubleshooting guide and I think this is a new bug.

Contact Details

No response

Version

1.24.0

What version of Kubernetes are you using?

1.30

What is your Kubernetes environment?

Cloud: Google GKE

How did you install the operator?

YAML manifest

What happened?

  1. [Bug]: Prometheus v2.52+ fails to scrape metrics: Error on ingesting samples with different value but same timestamp #6108 , decide to set monitoring.enablePodMonitor: false. In the same commit, create a PodMonitor to be applied manually, with the same name (i.e. the cluster name) and namespace.
  2. Apply both changes simultaneously
  3. Wonder why I don't have a PodMonitor
  4. Go spelunking in the code:
    client.ObjectKeyFromObject(expectedPodMonitor),
    gets a reference to the PodMonitor based on namespace and name only, not looking at the owner reference. Therefore
    case !manager.IsPodMonitorEnabled() && podMonitor != nil:
    indeed finds the PodMonitor (that it doesn't have an owner reference on) and deletes it.

I worked around the issue by adding a short suffix to the PodMonitor name so that it wouldn't be deleted, but this isn't ideal because it changes the job label that Prometheus attaches to it.

Cluster resource

N/A

Relevant log output

N/A

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

Labels

triagePending triage

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions