Skip to content
This repository was archived by the owner on Apr 4, 2026. It is now read-only.

Improved request body enumerator to use Concurrent.unicast push instead of buffering all the request bytes in memory#169

Merged
dlecan merged 1 commit into
play2war:developfrom
fdimuccio:develop
Jul 12, 2013
Merged

Improved request body enumerator to use Concurrent.unicast push instead of buffering all the request bytes in memory#169
dlecan merged 1 commit into
play2war:developfrom
fdimuccio:develop

Conversation

@fdimuccio

Copy link
Copy Markdown
Contributor

This should allow to stream bytes to the body parser iterator as they arrive avoiding OOM error when receiving large request body such as file uploads.

…ad of buffering all the request bytes in memory. This should avoid OOM error when receiving large request body such as file uploads.
dlecan added a commit that referenced this pull request Jul 1, 2013
@dlecan

dlecan commented Jul 1, 2013

Copy link
Copy Markdown
Member

Integration tests are OK.

@takezoe used to have hangs and/or deadlocks here on Tomcat 6 (servlet 2.5). I made this change for him months ago.

Before, inputstream was "streamed" thanks to Enumerator.fromStream(is).andThen(Enumerator.eof). This new solution form @fdimuccio may be stronger. I don't know how to evalutate it without a hanging test case from @takezoe.

dlecan added a commit that referenced this pull request Jul 12, 2013
Improved request body enumerator to use Concurrent.unicast push instead of buffering all the request bytes in memory
@dlecan dlecan merged commit b3871ec into play2war:develop Jul 12, 2013
@dlecan

dlecan commented Jul 12, 2013

Copy link
Copy Markdown
Member

Thank you for this contribution.

@fdimuccio

Copy link
Copy Markdown
Contributor Author

No problem, but i would wait before pushing it in production.

I was doing major investigation about the hanging test case reported by @takezoe but I couldn't reproduce it, instead I found another bug with Concurrent.push implementation. The request hangs when a POST/PUT action throws an exception that is not handled, because the callback in RequestHandler to redeem the result/error from eventuallyResult is never called.

The problem does not seems related to a particular container or servlet api (tested with jetty, tomcat, servlet25 and servlet30). I did not found the reason yet, but it could be that the iteratee execution context is not the same of the action handling code.

In Play 2.2.0-M1 is possible to pass an execution context to the iteratees, so maybe I could give it a try and let you know any progress.

@dlecan

dlecan commented Aug 5, 2013

Copy link
Copy Markdown
Member

Ok, I reverted it.

@takezoe

takezoe commented Aug 5, 2013

Copy link
Copy Markdown
Contributor

Enumerator.fromStream() in Play 2.0.x had contained a critical bug. It's not tail recursive so if large request is given, it had caused stack overflow. Moreover it had not been lifted up to the surface, so it seemed to have hung.

In Play 2.1, this method implementation has been changed fundamentally. So it might work correctly.

Anyway, this pull request does not depend on Enumerator.fromStream(). I think it's better solution than mine.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants