Checking the code in log.go, it looks like close_removed/close_renamed are only checked once we reached EOF. As long as filebeat has an open file handle, but hasn't reached the end yet, the OS will never report an error. Due to this filebeat would keep files open, even the settings are set.