Asynchronous Uploads by sumnerevans · Pull Request #15503 · matrix-org/synapse

sumnerevans · 2023-04-28T14:52:42Z

Implements the recently merged MSC2246 Asynchronous Uploads for Synapse's media repository.

Closes #12484

Configuration options rate limiting media creation
Configuration options for how long to wait until media IDs expire
POST /_matrix/media/v1/create
PUT /_matrix/media/v3/upload/{serverName}/{mediaId}
timeout_ms support on /download
timeout_ms support on /thumbnail
Option to restrict how many open media uploads a given user can have.

Signed-off-by: Sumner Evans sumner@beeper.com

Keywords: async

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
Pull request includes a sign off
Code style is correct
(run the linters)

clokep · 2023-05-01T15:19:59Z

@sumnerevans I'm going to remove the review request from this since you marked it as a draft; if you are still looking for a review please shout about what bits you're looking for a review of (or why it is in draft but you want a review). Thanks! 👍

sumnerevans · 2023-05-01T18:01:33Z

@clokep I accidentally opened as a non-draft PR. And then the automatic reviewer adder functionality kicked in and I could not remove it. Sorry about that!

clokep · 2023-05-01T18:04:57Z

Not an issue! If it doesn't kick in when un-drafting the PR just shout! 👍

sumnerevans · 2023-06-09T17:19:09Z

Blocked by matrix-org/complement#638

Signed-off-by: Sumner Evans <me@sumnerevans.com>

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

Signed-off-by: Sumner Evans <me@sumnerevans.com>

synapse/rest/media/create_resource.py

synapse/storage/databases/main/media_repository.py

synapse/rest/media/upload_resource.py

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

synapse/media/media_repository.py

clokep · 2023-11-13T20:02:07Z

With the last comment and updating the complement tests this is all passing for me locally again! 👍

I'm pretty certain it is good to merge.

Signed-off-by: Sumner Evans <me@sumnerevans.com>

clokep · 2023-11-14T17:26:29Z

Kicked off a new round of complement tests: https://github.com/matrix-org/synapse/actions/runs/6867273574

clokep

Should be good if the test run works. 🎉

synapse/storage/databases/main/media_repository.py

clokep · 2023-11-15T13:30:57Z

Complement with workers was failing, hopefully the latest change fixes it: https://github.com/matrix-org/synapse/actions/runs/6877944714

clokep · 2023-11-15T14:18:55Z

Tests passed minus a known flake (matrix-org/sytest#1201). 🎉 Thanks for working through this with me! 🚀

sumnerevans requested a review from a team as a code owner April 28, 2023 14:52

sumnerevans mentioned this pull request Apr 28, 2023

Asynchronous Uploads #15460

Closed

11 tasks

sumnerevans marked this pull request as draft April 28, 2023 14:53

github-actions bot deployed to PR Documentation Preview April 28, 2023 14:53 View deployment

clokep removed the request for review from a team May 1, 2023 15:20

This was referenced May 23, 2023

Declare support for Matrix v1.7 #15661

Closed

MSC2246 async uploads #12484

Closed

sumnerevans force-pushed the async-uploads branch from 298355c to a126708 Compare June 9, 2023 15:18

github-actions bot deployed to PR Documentation Preview June 9, 2023 15:19 View deployment

github-actions bot deployed to PR Documentation Preview June 9, 2023 15:58 View deployment

github-actions bot deployed to PR Documentation Preview June 9, 2023 16:00 View deployment

github-actions bot deployed to PR Documentation Preview June 9, 2023 17:02 View deployment

github-actions bot deployed to PR Documentation Preview June 9, 2023 17:05 View deployment

github-actions bot deployed to PR Documentation Preview June 9, 2023 17:25 View deployment

github-actions bot deployed to PR Documentation Preview June 9, 2023 19:53 View deployment

sumnerevans force-pushed the async-uploads branch from 5d31b44 to b0617e3 Compare June 9, 2023 19:56

github-actions bot deployed to PR Documentation Preview June 9, 2023 19:57 View deployment

github-actions bot deployed to PR Documentation Preview June 9, 2023 20:16 View deployment

github-actions bot deployed to PR Documentation Preview June 9, 2023 20:17 View deployment

sumnerevans mentioned this pull request Jun 9, 2023

unknown endpoints: allow PUT /_matrix/media/v3/upload matrix-org/complement#638

Merged

1 task

sumnerevans force-pushed the async-uploads branch from f05bb89 to 19a7e14 Compare June 9, 2023 21:47

github-actions bot deployed to PR Documentation Preview June 9, 2023 21:48 View deployment

github-actions bot deployed to PR Documentation Preview June 9, 2023 23:21 View deployment

github-actions bot deployed to PR Documentation Preview June 10, 2023 00:01 View deployment

sumnerevans marked this pull request as ready for review June 10, 2023 12:47

sumnerevans force-pushed the async-uploads branch from a91a9bd to 46ed200 Compare June 10, 2023 12:54

sumnerevans and others added 7 commits November 9, 2023 16:17

fixup! media/upload: add support for async uploads

b49a50b

Signed-off-by: Sumner Evans <me@sumnerevans.com>

fixup! media/{download,thumbnail}: support timeout_ms parameter

4e4e8d0

Signed-off-by: Sumner Evans <me@sumnerevans.com>

fixup! media/{download,thumbnail}: support timeout_ms parameter

37c7970

Signed-off-by: Sumner Evans <me@sumnerevans.com>

fixup! media/create: add /create endpoint

acadec0

Signed-off-by: Sumner Evans <me@sumnerevans.com>

fixup! media/create: don't store unused expiration time in database

58364ad

Signed-off-by: Sumner Evans <me@sumnerevans.com>

fixup! media/create: enforce limit on number of pending uploads

a514c84

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

fixup! media/create: enforce limit on number of pending uploads

b298258

Signed-off-by: Sumner Evans <me@sumnerevans.com>

clokep reviewed Nov 13, 2023

View reviewed changes

synapse/rest/media/create_resource.py Outdated Show resolved Hide resolved

clokep reviewed Nov 13, 2023

View reviewed changes

synapse/storage/databases/main/media_repository.py Outdated Show resolved Hide resolved

clokep reviewed Nov 13, 2023

View reviewed changes

synapse/rest/media/upload_resource.py Outdated Show resolved Hide resolved

sumnerevans and others added 3 commits November 13, 2023 12:31

fixup! media/create: add /create endpoint

1cff556

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

fixup! media/create: enforce limit on number of pending uploads

07afcfb

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

fixup! media/upload: split async servlet into separate class

c377b5d

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

clokep reviewed Nov 13, 2023

View reviewed changes

synapse/media/media_repository.py Show resolved Hide resolved

sumnerevans added 4 commits November 14, 2023 10:02

fixup! media/{download,thumbnail}: support timeout_ms parameter

6974380

Signed-off-by: Sumner Evans <me@sumnerevans.com>

fixup! media/{download,thumbnail}: support timeout_ms parameter

8b225ee

Signed-off-by: Sumner Evans <me@sumnerevans.com>

fixup! media/{download,thumbnail}: support timeout_ms parameter

c5e4f04

Signed-off-by: Sumner Evans <me@sumnerevans.com>

fixup! media/create: enforce limit on number of pending uploads

c3d3afe

Signed-off-by: Sumner Evans <me@sumnerevans.com>

clokep approved these changes Nov 14, 2023

View reviewed changes

clokep reviewed Nov 15, 2023

View reviewed changes

synapse/storage/databases/main/media_repository.py Outdated Show resolved Hide resolved

clokep reviewed Nov 15, 2023

View reviewed changes

synapse/storage/databases/main/media_repository.py Show resolved Hide resolved

Fix reference to potentially undefined config.

8debbdf

clokep reviewed Nov 15, 2023

View reviewed changes

synapse/storage/databases/main/media_repository.py Show resolved Hide resolved

Appease mypy

38badff

sumnerevans mentioned this pull request Nov 21, 2023

MSC2246: Asynchronous media uploads matrix-org/matrix-spec-proposals#2246

Merged

donjuanplatinum mentioned this pull request Feb 28, 2026

add MSC2246 support matrix-construct/tuwunel#347

Open

Uh oh!

Conversation

sumnerevans commented Apr 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Checklist

Uh oh!

clokep commented May 1, 2023

Uh oh!

sumnerevans commented May 1, 2023

Uh oh!

clokep commented May 1, 2023

Uh oh!

sumnerevans commented Jun 9, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

clokep commented Nov 13, 2023

Uh oh!

clokep commented Nov 14, 2023

Uh oh!

clokep left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

clokep commented Nov 15, 2023

Uh oh!

clokep commented Nov 15, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sumnerevans commented Apr 28, 2023 •

edited

Loading