Increase perf of handling presence when joining large rooms. by erikjohnston · Pull Request #9916 · matrix-org/synapse

erikjohnston · 2021-04-30T12:36:05Z

We ended up doing a lot of duplicate work, and e.g. ended up doing n^2
worth of is_mine_id(..) checks across all joined users.

Based on #9910

We ended up doing a *lot* of duplicate work, and e.g. ended up doing n^2 worth of `is_mine_id(..)` checks across all joined users.

richvdh

lgtm, modulo a couple of thoughts

richvdh · 2021-04-30T18:00:49Z

synapse/handlers/presence.py

+        prev_local_users = set()
+        prev_remote_hosts = set()
+        for user_id in prev_users:
+            if self.is_mine_id(user_id):
+                prev_local_users.add(user_id)
+            else:
+                prev_remote_hosts.add(get_domain_from_id(user_id))


I wonder if a general "partition a user list by homeserver" function would be generally useful. just a thought, no need to do anything about it.

Yeah, that would probably make a lot of sense

synapse/handlers/presence.py

richvdh · 2021-04-30T18:05:28Z

synapse/handlers/presence.py

+
+        # Get all the users who were already in the room, by fetching the
+        # current users in the room and removing the newly joined users.
+        users = await self.store.get_users_in_room(room_id)


can we rely on all deltas being for the same room id? and can we rely on there being at least one delta (so that room_id is set)?

Oh, that is a good point. I don't think we can rely on there only being one room per call (though that is the common path).

…esence_join

@aaronraimist

Synapse 1.34.0 (2021-05-17) =========================== This release deprecates the `room_invite_state_types` configuration setting. See the [upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.34.0/UPGRADE.rst#upgrading-to-v1340) for instructions on updating your configuration file to use the new `room_prejoin_state` setting. This release also deprecates the `POST /_synapse/admin/v1/rooms/<room_id>/delete` admin API route. Server administrators are encouraged to update their scripts to use the new `DELETE /_synapse/admin/v1/rooms/<room_id>` route instead. No significant changes since v1.34.0rc1. Synapse 1.34.0rc1 (2021-05-12) ============================== Features -------- - Add experimental option to track memory usage of the caches. ([\matrix-org#9881](matrix-org#9881)) - Add support for `DELETE /_synapse/admin/v1/rooms/<room_id>`. ([\matrix-org#9889](matrix-org#9889)) - Add limits to how often Synapse will GC, ensuring that large servers do not end up GC thrashing if `gc_thresholds` has not been correctly set. ([\matrix-org#9902](matrix-org#9902)) - Improve performance of sending events for worker-based deployments using Redis. ([\matrix-org#9905](matrix-org#9905), [\matrix-org#9950](matrix-org#9950), [\matrix-org#9951](matrix-org#9951)) - Improve performance after joining a large room when presence is enabled. ([\matrix-org#9910](matrix-org#9910), [\matrix-org#9916](matrix-org#9916)) - Support stable identifiers for [MSC1772](matrix-org/matrix-spec-proposals#1772) Spaces. `m.space.child` events will now be taken into account when populating the experimental spaces summary response. Please see [the upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.34.0/UPGRADE.rst#upgrading-to-v1340) if you have customised `room_invite_state_types` in your configuration. ([\matrix-org#9915](matrix-org#9915), [\matrix-org#9966](matrix-org#9966)) - Improve performance of backfilling in large rooms. ([\matrix-org#9935](matrix-org#9935)) - Add a config option to allow you to prevent device display names from being shared over federation. Contributed by @aaronraimist. ([\matrix-org#9945](matrix-org#9945)) - Update support for [MSC2946](matrix-org/matrix-spec-proposals#2946): Spaces Summary. ([\matrix-org#9947](matrix-org#9947), [\matrix-org#9954](matrix-org#9954)) Bugfixes -------- - Fix a bug introduced in v1.32.0 where the associated connection was improperly logged for SQL logging statements. ([\matrix-org#9895](matrix-org#9895)) - Correct the type hint for the `user_may_create_room_alias` method of spam checkers. It is provided a `RoomAlias`, not a `str`. ([\matrix-org#9896](matrix-org#9896)) - Fix bug where user directory could get out of sync if room visibility and membership changed in quick succession. ([\matrix-org#9910](matrix-org#9910)) - Include the `origin_server_ts` property in the experimental [MSC2946](matrix-org/matrix-spec-proposals#2946) support to allow clients to properly sort rooms. ([\matrix-org#9928](matrix-org#9928)) - Fix bugs introduced in v1.23.0 which made the PostgreSQL port script fail when run with a newly-created SQLite database. ([\matrix-org#9930](matrix-org#9930)) - Fix a bug introduced in Synapse 1.29.0 which caused `m.room_key_request` to-device messages sent from one user to another to be dropped. ([\matrix-org#9961](matrix-org#9961), [\matrix-org#9965](matrix-org#9965)) - Fix a bug introduced in v1.27.0 preventing users and appservices exempt from ratelimiting from creating rooms with many invitees. ([\matrix-org#9968](matrix-org#9968)) Updates to the Docker image --------------------------- - Add `startup_delay` to docker healthcheck to reduce waiting time for coming online and update the documentation with extra options. Contributed by @maquis196. ([\matrix-org#9913](matrix-org#9913)) Improved Documentation ---------------------- - Add `port` argument to the Postgres database sample config section. ([\matrix-org#9911](matrix-org#9911)) Deprecations and Removals ------------------------- - Mark as deprecated `POST /_synapse/admin/v1/rooms/<room_id>/delete`. ([\matrix-org#9889](matrix-org#9889)) Internal Changes ---------------- - Reduce the length of Synapse's access tokens. ([\matrix-org#5588](matrix-org#5588)) - Export jemalloc stats to Prometheus if it is being used. ([\matrix-org#9882](matrix-org#9882)) - Add type hints to presence handler. ([\matrix-org#9885](matrix-org#9885)) - Reduce memory usage of the LRU caches. ([\matrix-org#9886](matrix-org#9886)) - Add type hints to the `synapse.handlers` module. ([\matrix-org#9896](matrix-org#9896)) - Time response time for external cache requests. ([\matrix-org#9904](matrix-org#9904)) - Minor fixes to the `make_full_schema.sh` script. ([\matrix-org#9931](matrix-org#9931)) - Move database schema files into a common directory. ([\matrix-org#9932](matrix-org#9932)) - Add debug logging for lost/delayed to-device messages. ([\matrix-org#9959](matrix-org#9959))

@aaronraimist

Synapse 1.34.0 (2021-05-17) =========================== This release deprecates the `room_invite_state_types` configuration setting. See the [upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.34.0/UPGRADE.rst#upgrading-to-v1340) for instructions on updating your configuration file to use the new `room_prejoin_state` setting. This release also deprecates the `POST /_synapse/admin/v1/rooms/<room_id>/delete` admin API route. Server administrators are encouraged to update their scripts to use the new `DELETE /_synapse/admin/v1/rooms/<room_id>` route instead. No significant changes since v1.34.0rc1. Synapse 1.34.0rc1 (2021-05-12) ============================== Features -------- - Add experimental option to track memory usage of the caches. ([\matrix-org#9881](matrix-org#9881)) - Add support for `DELETE /_synapse/admin/v1/rooms/<room_id>`. ([\matrix-org#9889](matrix-org#9889)) - Add limits to how often Synapse will GC, ensuring that large servers do not end up GC thrashing if `gc_thresholds` has not been correctly set. ([\matrix-org#9902](matrix-org#9902)) - Improve performance of sending events for worker-based deployments using Redis. ([\matrix-org#9905](matrix-org#9905), [\matrix-org#9950](matrix-org#9950), [\matrix-org#9951](matrix-org#9951)) - Improve performance after joining a large room when presence is enabled. ([\matrix-org#9910](matrix-org#9910), [\matrix-org#9916](matrix-org#9916)) - Support stable identifiers for [MSC1772](matrix-org/matrix-spec-proposals#1772) Spaces. `m.space.child` events will now be taken into account when populating the experimental spaces summary response. Please see [the upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.34.0/UPGRADE.rst#upgrading-to-v1340) if you have customised `room_invite_state_types` in your configuration. ([\matrix-org#9915](matrix-org#9915), [\matrix-org#9966](matrix-org#9966)) - Improve performance of backfilling in large rooms. ([\matrix-org#9935](matrix-org#9935)) - Add a config option to allow you to prevent device display names from being shared over federation. Contributed by @aaronraimist. ([\matrix-org#9945](matrix-org#9945)) - Update support for [MSC2946](matrix-org/matrix-spec-proposals#2946): Spaces Summary. ([\matrix-org#9947](matrix-org#9947), [\matrix-org#9954](matrix-org#9954)) Bugfixes -------- - Fix a bug introduced in v1.32.0 where the associated connection was improperly logged for SQL logging statements. ([\matrix-org#9895](matrix-org#9895)) - Correct the type hint for the `user_may_create_room_alias` method of spam checkers. It is provided a `RoomAlias`, not a `str`. ([\matrix-org#9896](matrix-org#9896)) - Fix bug where user directory could get out of sync if room visibility and membership changed in quick succession. ([\matrix-org#9910](matrix-org#9910)) - Include the `origin_server_ts` property in the experimental [MSC2946](matrix-org/matrix-spec-proposals#2946) support to allow clients to properly sort rooms. ([\matrix-org#9928](matrix-org#9928)) - Fix bugs introduced in v1.23.0 which made the PostgreSQL port script fail when run with a newly-created SQLite database. ([\matrix-org#9930](matrix-org#9930)) - Fix a bug introduced in Synapse 1.29.0 which caused `m.room_key_request` to-device messages sent from one user to another to be dropped. ([\matrix-org#9961](matrix-org#9961), [\matrix-org#9965](matrix-org#9965)) - Fix a bug introduced in v1.27.0 preventing users and appservices exempt from ratelimiting from creating rooms with many invitees. ([\matrix-org#9968](matrix-org#9968)) Updates to the Docker image --------------------------- - Add `startup_delay` to docker healthcheck to reduce waiting time for coming online and update the documentation with extra options. Contributed by @maquis196. ([\matrix-org#9913](matrix-org#9913)) Improved Documentation ---------------------- - Add `port` argument to the Postgres database sample config section. ([\matrix-org#9911](matrix-org#9911)) Deprecations and Removals ------------------------- - Mark as deprecated `POST /_synapse/admin/v1/rooms/<room_id>/delete`. ([\matrix-org#9889](matrix-org#9889)) Internal Changes ---------------- - Reduce the length of Synapse's access tokens. ([\matrix-org#5588](matrix-org#5588)) - Export jemalloc stats to Prometheus if it is being used. ([\matrix-org#9882](matrix-org#9882)) - Add type hints to presence handler. ([\matrix-org#9885](matrix-org#9885)) - Reduce memory usage of the LRU caches. ([\matrix-org#9886](matrix-org#9886)) - Add type hints to the `synapse.handlers` module. ([\matrix-org#9896](matrix-org#9896)) - Time response time for external cache requests. ([\matrix-org#9904](matrix-org#9904)) - Minor fixes to the `make_full_schema.sh` script. ([\matrix-org#9931](matrix-org#9931)) - Move database schema files into a common directory. ([\matrix-org#9932](matrix-org#9932)) - Add debug logging for lost/delayed to-device messages. ([\matrix-org#9959](matrix-org#9959))

@aaronraimist

Synapse 1.34.0 (2021-05-17) =========================== This release deprecates the `room_invite_state_types` configuration setting. See the [upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.34.0/UPGRADE.rst#upgrading-to-v1340) for instructions on updating your configuration file to use the new `room_prejoin_state` setting. This release also deprecates the `POST /_synapse/admin/v1/rooms/<room_id>/delete` admin API route. Server administrators are encouraged to update their scripts to use the new `DELETE /_synapse/admin/v1/rooms/<room_id>` route instead. No significant changes since v1.34.0rc1. Synapse 1.34.0rc1 (2021-05-12) ============================== Features -------- - Add experimental option to track memory usage of the caches. ([\#9881](matrix-org/synapse#9881)) - Add support for `DELETE /_synapse/admin/v1/rooms/<room_id>`. ([\#9889](matrix-org/synapse#9889)) - Add limits to how often Synapse will GC, ensuring that large servers do not end up GC thrashing if `gc_thresholds` has not been correctly set. ([\#9902](matrix-org/synapse#9902)) - Improve performance of sending events for worker-based deployments using Redis. ([\#9905](matrix-org/synapse#9905), [\#9950](matrix-org/synapse#9950), [\#9951](matrix-org/synapse#9951)) - Improve performance after joining a large room when presence is enabled. ([\#9910](matrix-org/synapse#9910), [\#9916](matrix-org/synapse#9916)) - Support stable identifiers for [MSC1772](matrix-org/matrix-spec-proposals#1772) Spaces. `m.space.child` events will now be taken into account when populating the experimental spaces summary response. Please see [the upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.34.0/UPGRADE.rst#upgrading-to-v1340) if you have customised `room_invite_state_types` in your configuration. ([\#9915](matrix-org/synapse#9915), [\#9966](matrix-org/synapse#9966)) - Improve performance of backfilling in large rooms. ([\#9935](matrix-org/synapse#9935)) - Add a config option to allow you to prevent device display names from being shared over federation. Contributed by @aaronraimist. ([\#9945](matrix-org/synapse#9945)) - Update support for [MSC2946](matrix-org/matrix-spec-proposals#2946): Spaces Summary. ([\#9947](matrix-org/synapse#9947), [\#9954](matrix-org/synapse#9954)) Bugfixes -------- - Fix a bug introduced in v1.32.0 where the associated connection was improperly logged for SQL logging statements. ([\#9895](matrix-org/synapse#9895)) - Correct the type hint for the `user_may_create_room_alias` method of spam checkers. It is provided a `RoomAlias`, not a `str`. ([\#9896](matrix-org/synapse#9896)) - Fix bug where user directory could get out of sync if room visibility and membership changed in quick succession. ([\#9910](matrix-org/synapse#9910)) - Include the `origin_server_ts` property in the experimental [MSC2946](matrix-org/matrix-spec-proposals#2946) support to allow clients to properly sort rooms. ([\#9928](matrix-org/synapse#9928)) - Fix bugs introduced in v1.23.0 which made the PostgreSQL port script fail when run with a newly-created SQLite database. ([\#9930](matrix-org/synapse#9930)) - Fix a bug introduced in Synapse 1.29.0 which caused `m.room_key_request` to-device messages sent from one user to another to be dropped. ([\#9961](matrix-org/synapse#9961), [\#9965](matrix-org/synapse#9965)) - Fix a bug introduced in v1.27.0 preventing users and appservices exempt from ratelimiting from creating rooms with many invitees. ([\#9968](matrix-org/synapse#9968)) Updates to the Docker image --------------------------- - Add `startup_delay` to docker healthcheck to reduce waiting time for coming online and update the documentation with extra options. Contributed by @maquis196. ([\#9913](matrix-org/synapse#9913)) Improved Documentation ---------------------- - Add `port` argument to the Postgres database sample config section. ([\#9911](matrix-org/synapse#9911)) Deprecations and Removals ------------------------- - Mark as deprecated `POST /_synapse/admin/v1/rooms/<room_id>/delete`. ([\#9889](matrix-org/synapse#9889)) Internal Changes ---------------- - Reduce the length of Synapse's access tokens. ([\#5588](matrix-org/synapse#5588)) - Export jemalloc stats to Prometheus if it is being used. ([\#9882](matrix-org/synapse#9882)) - Add type hints to presence handler. ([\#9885](matrix-org/synapse#9885)) - Reduce memory usage of the LRU caches. ([\#9886](matrix-org/synapse#9886)) - Add type hints to the `synapse.handlers` module. ([\#9896](matrix-org/synapse#9896)) - Time response time for external cache requests. ([\#9904](matrix-org/synapse#9904)) - Minor fixes to the `make_full_schema.sh` script. ([\#9931](matrix-org/synapse#9931)) - Move database schema files into a common directory. ([\#9932](matrix-org/synapse#9932)) - Add debug logging for lost/delayed to-device messages. ([\#9959](matrix-org/synapse#9959))

erikjohnston added 5 commits April 29, 2021 17:35

Use get_current_users_in_room from store and not StateHandler

996c0ce

Newsfile

0c8cd62

Use correct name

6640fb4

Increase perf of handling presence when joining large rooms.

0ed608c

We ended up doing a *lot* of duplicate work, and e.g. ended up doing n^2 worth of `is_mine_id(..)` checks across all joined users.

Newsfile

68b6106

erikjohnston requested a review from a team April 30, 2021 13:07

richvdh approved these changes Apr 30, 2021

View reviewed changes

erikjohnston added 2 commits May 4, 2021 13:16

Process state deltas in presence by room

48cf260

Use lists instead of sets where appropriate

4caa84b

erikjohnston requested a review from richvdh May 4, 2021 13:01

richvdh approved these changes May 4, 2021

View reviewed changes

erikjohnston added 2 commits May 5, 2021 16:51

Merge remote-tracking branch 'origin/develop' into erikj/efficient_pr…

43b8f20

…esence_join

Fiddle with changelogs

9b109fd

erikjohnston merged commit 37623e3 into develop May 5, 2021

erikjohnston deleted the erikj/efficient_presence_join branch May 5, 2021 16:27

erikjohnston mentioned this pull request Jul 26, 2021

Presence is increasingly heavy #3971

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Increase perf of handling presence when joining large rooms.#9916

Increase perf of handling presence when joining large rooms.#9916
erikjohnston merged 9 commits intodevelopfrom
erikj/efficient_presence_join

erikjohnston commented Apr 30, 2021

Uh oh!

richvdh left a comment

Uh oh!

richvdh Apr 30, 2021

Uh oh!

erikjohnston May 4, 2021

Uh oh!

Uh oh!

richvdh Apr 30, 2021

Uh oh!

erikjohnston May 4, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

erikjohnston commented Apr 30, 2021

Uh oh!

richvdh left a comment

Choose a reason for hiding this comment

Uh oh!

richvdh Apr 30, 2021

Choose a reason for hiding this comment

Uh oh!

erikjohnston May 4, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

richvdh Apr 30, 2021

Choose a reason for hiding this comment

Uh oh!

erikjohnston May 4, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants