Skip to content

Allow opting out of parsing of MIME upload parsing in twisted.web #12412

@itamarst

Description

@itamarst

Current file upload MIME parsing in Twisted Web requires, based on the interface, requires parsing the whole thing into memory. So for large uploads this can result in very large memory usage. The API for users specifically involves accessing the mapping request.args.

How to solve this?

  1. Long term, one could do full-on async streaming parsing, but that is a large project. It also doesn't actually solve the problem in the short term, since for backwards compatibility request.args would still need to exist.
  2. On-demand parsing only when request.args is accessed. This is not ideal since it makes it requires blocking, which can be meaningful for large uploads.
  3. Allow users to opt-out so they can just do their own parsing (or use the streaming future API, if that ever happens). Tahoe-LAFS already wants to do this, for example, and right now overrides the parsing logic already in a pretty hacky way.

I propose to implement option 3.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions