Envoy does not release access log fd on sigusr1 #4060
Description
Envoy does not release access log fd on sigusr1
Description:
In one of our more heavily used clusters we've seen that despite employing logrotate and kill -USR1 $ENVOY_PID the file descriptors are still present in /proc/$ENVOY_PID/fd. We have also seen the reported memory usage increase until we force a reload either by restarting the process or kill -HUP $ENVOY_HOT_RESTART_PID.
Version:
envoy 08fadcc41f34b5816f2dc9412a1614efca4d9667/1.8.0-dev/Clean/RELEASE live 2829 44974 1
Repro steps:
ls -l /var/log/envoy/200
lrwx------ 1 envoy envoy 64 Aug 6 12:39 130 -> /var/log/envoy/default_https.log
lrwx------ 1 envoy envoy 64 Aug 6 12:39 131 -> /var/log/envoy/200/default_https_access.log
lrwx------ 1 envoy envoy 64 Aug 6 12:39 146 -> /var/log/envoy/spark_default_https.log
lrwx------ 1 envoy envoy 64 Aug 6 12:39 61 -> /var/log/envoy/default.log
lrwx------ 1 envoy envoy 64 Aug 6 12:39 77 -> /var/log/envoy/spark_default.log
kill -USR1 $ENVOY_PIDsudo ls -l /proc/$ENVOY_PID/fd
lrwx------ 1 envoy envoy 64 Aug 6 12:39 130 -> /var/log/envoy/default_https.log
lrwx------ 1 envoy envoy 64 Aug 6 12:39 131 -> /var/log/envoy/200/default_https_access.log
lrwx------ 1 envoy envoy 64 Aug 6 12:39 146 -> /var/log/envoy/spark_default_https.log
lrwx------ 1 envoy envoy 64 Aug 6 12:39 147 -> /var/log/envoy/200/spark_default_https_access.log.1
lrwx------ 1 envoy envoy 64 Aug 6 12:39 61 -> /var/log/envoy/default.log
lrwx------ 1 envoy envoy 64 Aug 6 12:39 62 -> /var/log/envoy/200/default_access.log.1
lrwx------ 1 envoy envoy 64 Aug 6 12:39 77 -> /var/log/envoy/spark_default.log
lrwx------ 1 envoy envoy 64 Aug 6 12:39 78 -> /var/log/envoy/200/spark_default_access.log.1
Mitigation:
(this is for anyone that lands here from google searching in the future)
There are 2 ways to mitigate:
- Restart the envoy process.
- If using the envoy hot restarter script you can force a reload:
kill -HUP $ENVOY_HOT_RESTART_PID