Skip to content

Nginx keeps using old upstream IP of Relay after Relay restart #3894

@jerbob92

Description

@jerbob92

Self-Hosted Version

25.7.0

CPU Architecture

x86_64

Docker Version

28.1.1

Docker Compose Version

2.35.1

Machine Specification

  • My system meets the minimum system requirements of Sentry

Steps to Reproduce

Sometimes (I'm not quite sure what triggers this yet), our Relay gets recreated/restarted and gets a new IP. Afterwards the Relay becomes unavailable because nginx is still using the old upstream IP. The only way to resolve this is to restart the nginx service.

This is probably because the nginx config does not have the resolve directive for the upstreams: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#resolve

By default, nginx resolvers the IPs of the upstreams once and caches them forever.

So perhaps changing the upstreams to:

	upstream relay {
		server relay:3000 resolve;
		keepalive 2;
	}

	upstream sentry {
		server web:9000 resolve;
		keepalive 2;
	}

Would already fix it.

This does require the resolver to be configured: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#resolver

So probably something like this:

    resolver 127.0.0.11 valid=5s;

5s might be a bit short, the default TTL is 600, which seems a bit long, maybe 60?

I already read somewhere that adding just the resolver directive on root level could be enough.

Expected Result

Nginx should pick up the new IP address.

Actual Result

nginx-1                                         | 2025/08/21 07:29:40 [error] 32#32: *1270951 connect() failed (111: Connection refused) while connecting to upstream, client: xxxx, server: , request: "POST /api/38/envelope/ HTTP/1.0", upstream: "http://10.201.8.65:3000/api/38/envelope/", host: "xxxx"

Event ID

No response

Metadata

Metadata

Assignees

No one assigned

    Projects

    Status

    No status

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions