FileSystemWatcher may cause problems in containers - inotify limits and incorrect error message #27272
Description
MOVED FROM dotnet/aspnetcore#3475
Looking around the web I'm seeing years of issues with FileSystemWatcher saying "The configured user limit (n) on the number of inotify instances has been reached."
UPDATE: Looks like https://github.com/dotnet/corefx/blob/a10890f4ffe0fadf090c922578ba0e606ebdd16c/src/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Linux.cs#L371 will assume when inotify_add_watch fails with an ENOSPEC it must but an issue with inotify instances being out of range. In fact, ENOSPEC can also mean "the kernel failed to allocate a needed resource." We had no way to know it was anything other than "too many files open." The error message is misleading.
From the Man Page - The user limit on the total number of inotify watches was reached or the kernel failed to allocate a needed resource
Phrased differently. There's two Error Cases and we throw a message that implies there's just One.
This is becoming more prevalent in container situations in constrained sandboxes. I'm trying to deploy https://github.com/shanselman/superzeit (just clone and "now --public" or run locally with docker) to Zeit.co and I'm hitting this regularly. I don't think I'm hitting a limit. I think Zeit (and others) are blocking the syscall.
I think there are two issues here:
1 We should return a different error message if inotify_add_watch fails, and then circuit break so that FileSystemWatcher doesn't prevent the app from starting. If we CAN startup without a watch successfully, we should.
2 It seems DOTNET_USE_POLLING_FILE_WATCHER=1 is used in dotnet-watch and the aspnet file providers but the base System.IO FileSystemWatcher class doesn't support DOTNET_USE_POLLING_FILE_WATCHER? We should probably be consistent.
If I change reloadOnChange: false in Program.cs to bypass the first watch that is set on AppSettings.json, I end up hitting it later when Razor/MVC sets up its FileWatchers.
We need at a minimum, to have DOTNET_USE_POLLING_FILE_WATCHER respected everywhere. Another idea would be for a way to have FileSystemWatcher "fail gracefully." We need to test on systems with
Unhandled Exception: System.IO.IOException: The configured user limit (8192) on the number of inotify instances has been reached.
> [0] at System.IO.FileSystemWatcher.StartRaisingEvents()
> [0] at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
> [0] at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
> [0] at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
> [0] at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
> [0] at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
> [0] at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__0_0()
> [0] at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
> [0] at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
> [0] at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
> [0] at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
> [0] at Microsoft.AspNetCore.Hosting.WebHostBuilder.BuildCommonServices(AggregateException& hostingStartupErrors)
> [0] at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
> [0] at superzeit.Program.Main(String[] args) in /app/superzeit/Program.cs:line 17
Related issues?
- https://github.com/dotnet/corefx/issues/5660
- https://github.com/dotnet/corefx/search?q=inotify&type=Issues
- This issue may have been related AspNetCore IO Exception Invalid Arguement aspnet/Mvc#8299 because the FileWatcher failed to watch.
@stephentoub @natemcmaster @muratg @pranavkm
dotnet --info
.NET Core SDK (reflecting any global.json):
Version: 2.1.401
Commit: 91b1c13032
Runtime Environment:
OS Name: Windows
OS Version: 10.0.17134
OS Platform: Windows
RID: win10-x64
Base Path: C:\Program Files\dotnet\sdk\2.1.401\
Host (useful for support):
Version: 2.1.3-servicing-26724-03
Commit: 124038c13e
.NET Core SDKs installed:
2.1.400 [C:\Program Files\dotnet\sdk]
2.1.401 [C:\Program Files\dotnet\sdk]
.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.3-servicing-26724-03 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]