Skip to content

nexus crashes when it can't reach internal DNS on startup #2614

@davepacheco

Description

@davepacheco

Found this in CI (originally this job):

{"msg":"Setting up resolver on subnet: Ipv6Subnet { net: Ipv6Net(Ipv6Network { addr: fd00:1122:3344::, prefix: 48 }) }","v":0,"name":"nexus","level":30,"time":"2023-03-17T20:19:23.752265804-07:00","hostname":"oxz_nexus","pid":7362,"component":"ServerContext","name":"f836cbe7-f69c-4642-b314-6f6c026276e7"}
{"msg":"Accessing DB url from DNS","v":0,"name":"nexus","level":30,"time":"2023-03-17T20:19:23.752407736-07:00","hostname":"oxz_nexus","pid":7362,"component":"ServerContext","name":"f836cbe7-f69c-4642-b314-6f6c026276e7"}
nexus: Failed to lookup IP: request timed out
[ Mar 17 20:19:38 Stopping because all processes in service exited. ]
[ Mar 17 20:19:38 Executing stop method (:kill). ]
[ Mar 17 20:19:38 Restarting too quickly, changing state to maintenance. ]

rendered:

[2023-03-17T20:19:23.752265804-07:00]  INFO: f836cbe7-f69c-4642-b314-6f6c026276e7/ServerContext/7362 on oxz_nexus: Setting up resolver on subnet: Ipv6Subnet { net: Ipv6Net(Ipv6Network { addr: fd00:1122:3344::, prefix: 48 }) }
[2023-03-17T20:19:23.752407736-07:00]  INFO: f836cbe7-f69c-4642-b314-6f6c026276e7/ServerContext/7362 on oxz_nexus: Accessing DB url from DNS
nexus: Failed to lookup IP: request timed out
[ Mar 17 20:19:38 Stopping because all processes in service exited. ]
[ Mar 17 20:19:38 Executing stop method (:kill). ]
[ Mar 17 20:19:38 Restarting too quickly, changing state to maintenance. ]

Nexus needs to try indefinitely to reach DNS (along with anything else it depends on). The impact of this is that if DNS is ever offline or just out to lunch, even if it comes back, the system won't recover without intervention from support.

(I'm not sure why it failed in this particular case, but it doesn't really matter.)

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions