error finalizing order message is light on details #7267
Description
We're running: quay.io/jetstack/cert-manager-controller:v1.15.0
It'd help if I had access to at least the URL and IP address for an error like this, preferably the response headers (perhaps even the request headers, but maybe that's being greedy). There are so many nginx servers floating around that I have no idea who generated this error. As an end user, I can't determine which part of this process is involved / whom I should address to get it resolved.
- Is it my application's nginx that failed?
- This seems unlikely, as it shouldn't have an upstream and I don't think I'm using nginx
- Is it my ingress's nginx that failed?
- If so, I'd like to know what it was trying to talk to, but if I don't have any logs (url/ip/server headers), I won't be able to easily identify it
- Was it because a solver pod didn't do its job?
- Was it my normal application? (again, this seems unlikely)
- If so, I'd like to know what it was trying to talk to, but if I don't have any logs (url/ip/server headers), I won't be able to easily identify it
- Is it one of the acme endpoints that failed?
- Without headers, I can't easily complain to my acme provider
cert-manager pod logs
I0911 04:41:07.464144 1 trigger_controller.go:215] "Certificate must be re-issued" logger="cert-manager.controller" key="ingress-nginx/integrations-tls" reason="DoesNotExist" message="Issuing certificate as Secret does not exist"
I0911 04:41:07.464163 1 conditions.go:203] Setting lastTransitionTime for Certificate "integrations-tls" condition "Issuing" to 2024-09-11 04:41:07.464157843 +0000 UTC m=+1621694.000239082
I0911 04:41:14.680051 1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "integrations-tls-1" condition "Approved" to 2024-09-11 04:41:14.680037992 +0000 UTC m=+1621701.216119231
I0911 04:41:14.864056 1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "integrations-tls-1" condition "Ready" to 2024-09-11 04:41:14.864047216 +0000 UTC m=+1621701.400128439
E0911 07:43:30.642287 1 sync.go:73] "failed to update status" err=<
error finalizing order: 504 : <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>
> logger="cert-manager.controller" resource_name="integrations-tls-1-2417966242" resource_namespace="ingress-nginx" resource_kind="Order" resource_version="v1"
E0911 07:43:30.642333 1 controller.go:162] "re-queuing item due to error processing" err=<
[error finalizing order: 504 : <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>
, Operation cannot be fulfilled on orders.acme.cert-manager.io "integrations-tls-1-2417966242": StorageError: invalid object, Code: 4, Key: /registry/acme.cert-manager.io/orders/ingress-nginx/integrations-tls-1-2417966242, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 957f1a1e-cc2d-4040-a761-cced38d2b91e, UID in object meta: 0432c423-c07d-4731-ab05-a9f27e09de64]
> logger="cert-manager.controller" key="ingress-nginx/integrations-tls-1-2417966242"
I0911 07:43:50.604573 1 acme.go:233] "certificate issued" logger="cert-manager.controller.sign" resource_name="integrations-tls-1" resource_namespace="ingress-nginx" resource_kind="CertificateRequest" resource_version="v1" related_resource_name="integrations-tls-1-2417966242" related_resource_namespace="ingress-nginx" related_resource_kind="Order" related_resource_version="v1"
I0911 07:43:50.604737 1 conditions.go:252] Found status change for CertificateRequest "integrations-tls-1" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2024-09-11 07:43:50.604717999 +0000 UTC m=+1632657.140799234
I0911 07:43:50.763747 1 conditions.go:192] Found status change for Certificate "integrations-tls" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2024-09-11 07:43:50.680625145 +0000 UTC m=+1632657.216706384
I0911 07:43:50.964079 1 controller.go:157] "re-queuing item due to optimistic locking on resource" logger="cert-manager.controller" key="ingress-nginx/integrations-tls" error="Operation cannot be fulfilled on certificates.cert-manager.io \"integrations-tls\": the object has been modified; please apply your changes to the latest version and try again"
I0911 07:43:50.964941 1 conditions.go:192] Found status change for Certificate "integrations-tls" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2024-09-11 07:43:50.964933943 +0000 UTC m=+1632657.501015176
I0911 07:43:51.064054 1 controller.go:157] "re-queuing item due to optimistic locking on resource" logger="cert-manager.controller" key="ingress-nginx/integrations-tls" error="Operation cannot be fulfilled on certificates.cert-manager.io \"integrations-tls\": the object has been modified; please apply your changes to the latest version and try again"
I0911 07:43:51.065249 1 conditions.go:192] Found status change for Certificate "integrations-tls" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2024-09-11 07:43:51.06523912 +0000 UTC m=+1632657.601320358
E0911 07:43:51.250188 1 sync.go:562] "failed to finalize Order resource due to bad request, marking Order as failed" err="400 urn:ietf:params:acme:error:orderInvalid: Order is marked invalid." logger="cert-manager.controller" resource_name="integrations-tls-1-2417966242" resource_namespace="ingress-nginx" resource_kind="Order" resource_version="v1"
E0911 07:43:51.263854 1 sync.go:73] "failed to update status" logger="cert-manager.controller" resource_name="integrations-tls-1-2417966242" resource_namespace="ingress-nginx" resource_kind="Order" resource_version="v1"
I0911 07:43:51.263886 1 controller.go:157] "re-queuing item due to optimistic locking on resource" logger="cert-manager.controller" key="ingress-nginx/integrations-tls-1-2417966242" error="Operation cannot be fulfilled on orders.acme.cert-manager.io \"integrations-tls-1-2417966242\": the object has been modified; please apply your changes to the latest version and try again"