Skip to content

Try using stream to avoid large buffers allocations in kestrel#9411

Draft
LukaszRozmej wants to merge 6 commits into
masterfrom
fix/large-rpc-buffers
Draft

Try using stream to avoid large buffers allocations in kestrel#9411
LukaszRozmej wants to merge 6 commits into
masterfrom
fix/large-rpc-buffers

Conversation

@LukaszRozmej

Copy link
Copy Markdown
Member

Fixes Closes Resolves #

Please choose one of the keywords above to refer to the issue this PR solves followed by the issue number (e.g. Fixes #000). If no issue number, remove the line. Also, remove everything marked optional that is not applicable. Remove this note after reading.

Changes

  • List the changes

Types of changes

What types of changes does your code introduce?

  • Bugfix (a non-breaking change that fixes an issue)
  • New feature (a non-breaking change that adds functionality)
  • Breaking change (a change that causes existing functionality not to work as expected)
  • Optimization
  • Refactoring
  • Documentation update
  • Build-related changes
  • Other: Description

Testing

Requires testing

  • Yes
  • No

If yes, did you write tests?

  • Yes
  • No

Notes on testing

Optional. Remove if not applicable.

Documentation

Requires documentation update

  • Yes
  • No

If yes, link the PR to the docs update or the issue with the details labeled docs. Remove if not applicable.

Requires explanation in Release Notes

  • Yes
  • No

If yes, fill in the details here. Remove if not applicable.

Remarks

Optional. Remove if not applicable.

@marcindsobczak

Copy link
Copy Markdown
Contributor

RPC requests are failing with these changes.

09 Oct 13:40:47 | Connection id "0HNG752A54IPG", Request id "0HNG752A54IPG:00000001": An unhandled exception was thrown by the application. System.InvalidOperationException: Synchronous operations are disallowed. Call WriteAsync or set AllowSynchronousIO to true instead.
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.IO.Stream.Write(ReadOnlySpan`1 buffer)
   at Nethermind.Serialization.Json.CountingStreamPipeWriter.FlushInternal(Boolean writeToStream) in C:\praca\nethermind\src\Nethermind\Nethermind.Serialization.Json\StreamPipeWriter.cs:line 445
   at System.Text.Json.Utf8JsonWriter.Flush()
   at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.Serialize(Utf8JsonWriter writer, T& rootValue, Object rootValueBoxed)
   at System.Text.Json.JsonSerializer.Serialize[TValue](Utf8JsonWriter writer, TValue value, JsonSerializerOptions options)
   at Nethermind.Facade.Eth.RpcTransaction.AccessListForRpc.JsonConverter.Write(Utf8JsonWriter writer, AccessListForRpc value, JsonSerializerOptions options) in C:\praca\nethermind\src\Nethermind\Nethermind.Facade\Eth\RpcTransaction\AccessListForRpc.cs:line 76
   at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.Converters.ArrayConverter`2.OnWriteResume(Utf8JsonWriter writer, TElement[] array, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonCollectionConverter`2.OnTryWrite(Utf8JsonWriter writer, TCollection value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonConverter`1.TryWriteAsObject(Utf8JsonWriter writer, Object value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonConverter`1.TryWriteAsObject(Utf8JsonWriter writer, Object value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.SerializeAsync(PipeWriter pipeWriter, T rootValue, Int32 flushThreshold, CancellationToken cancellationToken, Object rootValueBoxed)
   at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.SerializeAsync(PipeWriter pipeWriter, T rootValue, Int32 flushThreshold, CancellationToken cancellationToken, Object rootValueBoxed)
   at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.SerializeAsync(PipeWriter pipeWriter, T rootValue, Int32 flushThreshold, CancellationToken cancellationToken, Object rootValueBoxed)
   at Nethermind.Runner.JsonRpc.Startup.<>c__DisplayClass10_0.<<Configure>b__9>d.MoveNext() in C:\praca\nethermind\src\Nethermind\Nethermind.Runner\JsonRpc\Startup.cs:line 265
--- End of stack trace from previous location ---
   at Nethermind.Runner.JsonRpc.Startup.<>c__DisplayClass10_0.<<Configure>b__9>d.MoveNext() in C:\praca\nethermind\src\Nethermind\Nethermind.Runner\JsonRpc\Startup.cs:line 285
--- End of stack trace from previous location ---
   at Nethermind.Runner.JsonRpc.Startup.<>c__DisplayClass10_0.<<Configure>b__9>d.MoveNext() in C:\praca\nethermind\src\Nethermind\Nethermind.Runner\JsonRpc\Startup.cs:line 296
--- End of stack trace from previous location ---
   at Nethermind.Runner.JsonRpc.Startup.<>c__DisplayClass10_0.<<Configure>b__9>d.MoveNext() in C:\praca\nethermind\src\Nethermind\Nethermind.Runner\JsonRpc\Startup.cs:line 213
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

@LukaszRozmej

Copy link
Copy Markdown
Member Author

@marcindsobczak try now

@marcindsobczak

Copy link
Copy Markdown
Contributor

Without fix:
BeforeFix

With fix:
AfterFix

It was tested by running a script doing debug traces for a few minutes, turning it off and after ~1min doing a memory snapshot.
Changes from this PR are cleaning memory significantly.

@marcindsobczak marcindsobczak left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a significant improvement, worth to run it on some infra nodes to ensure stability

@LukaszRozmej

Copy link
Copy Markdown
Member Author

Yeah but it might compromise on some other performance stuff... So not sure if worth it with dotnet 10 around the corner. @benaadams ?

kamilchodola added a commit that referenced this pull request Feb 20, 2026
Apply CountingStreamPipeWriter(ctx.Response.Body) change inside the
refactored ProcessJsonRpcRequestCoreAsync method that master extracted
from the inline request processing block.
kamilchodola added a commit that referenced this pull request Feb 20, 2026
Head-to-head tests measuring UnflushedBytes (peak internal buffer)
during serialization. PipeWriter grows to full response size while
StreamPipeWriter stays bounded at ~26 bytes via auto-flush.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants