Here's the issue: when using http health checks the host header that comes with the request will be simply the IP of the node that the check hits. This IP will be for any node in the cluster since the ELB knows about all of them and will hit any one at any given time. This node will then forward that request to a node that it knows is running the service based on the NodePort that was hit. This means that the IP in the host header of the request could be the IP of any node in the cluster, and not necessarily the one that has the IP from the host header. So every instance of the app running in the cluster will need to have all IPs of all nodes in its ALLOWED_HOSTS setting. This is even more a problem if a new node is added to the cluster as updating the ALLOWED_HOSTS setting for each instance of the app will mean at best a rolling-restart so that it can pick up the new list of IPs and be able to respond to health-checks on this new node.

My current thinking is that we should avoid putting a list of IPs in ALLOWED_HOSTS and instead write a custom middleware for checking the host header. This will get our IP range and verify that the host header, if it's an IP, is in that range. If not it will fall back to checking against a list of allowed hosts.

I'm definitely open to other ideas for this, but doing our own custom check seems to be the best option. Other options I considered:

• Create another custom subclass of WSGIRequest like we did for the HTTPS=on thing, overriding the get_host() method to do the above checking, falling back to super() if the IP was not validated.
• Add every IP in our range for the cluster to the ALLOWED_HOSTS setting. This seemed a bit extreme and possibly error prone.
• Somehow exempt /healthz/ from the host header check. I can't see a way to do this. I'm pretty sure that it's the fault of a middleware that the get_host() method is run on every request, but I'm not positive.
• Run some separate process to handle the healthcheck. Not a great healthcheck and would make things more complex and fragile.

/cc @metadave @glogiotatidis @jgmize