Npgsql 7.0.2 blocks threads NpgsqlConnector.PerformUserCancellation causing connection pool exhaustion #5032
Description
Steps to reproduce
I have not been able to create a sample to reproduce yet, and I'm not able to share our source code. I will keep trying and share if I can reproduce.
The issue
After upgrading Npgsql NuGet package rom 7.0.1 to 7.0.2 we are seeing idle connections not getting closed after several days. The number of open connections gradually increased until hitting our max pool size (1000), after which every query failed since the connection pool was exhausted.
Our process thread count also increased until it plateued around 1000 when the max pool size was reached.
Reverting the Npgsql NuGet package back to 7.0.1 fixed the issue.
Stack trace:
System.Private.CoreLib!System.Threading.ManualResetEventSlim.Wait(int32,value class System.Threading.CancellationToken)
System.Private.CoreLib!System.Threading.ManualResetEventSlim.Wait()
Npgsql!Npgsql.Internal.NpgsqlConnector.PerformUserCancellation()
Npgsql!Npgsql.Internal.NpgsqlConnector+<>c.<StartNestedCancellableOperation>b__262_0(class System.Object)
System.Private.CoreLib!System.Threading.CancellationTokenSource.Register(class System.Delegate,class System.Object,class System.Threading.SynchronizationContext,class System.Threading.ExecutionContext)
Npgsql!Npgsql.NpgsqlDataReader+<NextResult>d__47.MoveNext()
System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start(!!0&)
Npgsql!Npgsql.NpgsqlDataReader.NextResultAsync(value class System.Threading.CancellationToken)
Npgsql!Npgsql.NpgsqlCommand+<ExecuteReader>d__124.MoveNext()
System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start(!!0&)
Npgsql!Npgsql.NpgsqlCommand.ExecuteReader(value class System.Data.CommandBehavior,bool,value class System.Threading.CancellationToken)
Npgsql!Npgsql.NpgsqlCommand.ExecuteReaderAsync(value class System.Data.CommandBehavior,value class System.Threading.CancellationToken)
Npgsql!Npgsql.NpgsqlCommand+<ExecuteDbDataReaderAsync>d__117.MoveNext()
System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start(!!0&)
Npgsql!Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(value class System.Data.CommandBehavior,value class System.Threading.CancellationToken)
I noticed there is a new ManualResetEventSlim added and used in the PerformUserCancellation method that was not there in 7.0.1 as well as some other changes in this method.
https://github.com/npgsql/npgsql/blob/v7.0.2/src/Npgsql/Internal/NpgsqlConnector.cs#L1720
Further technical details
Npgsql version: 7.0.2 (issue is not present in 7.0.1)
PostgreSQL version: 13
Operating system: Ubuntu