This repository was archived by the owner on Apr 26, 2024. It is now read-only.
Add membership column to current_state_events table#5706
Merged
erikjohnston merged 7 commits intodevelopfrom Jul 19, 2019
Merged
Add membership column to current_state_events table#5706erikjohnston merged 7 commits intodevelopfrom
erikjohnston merged 7 commits intodevelopfrom
Conversation
Codecov Report
@@ Coverage Diff @@
## develop #5706 +/- ##
===========================================
- Coverage 63.42% 63.34% -0.09%
===========================================
Files 331 331
Lines 36163 36176 +13
Branches 5960 5960
===========================================
- Hits 22938 22915 -23
- Misses 11597 11627 +30
- Partials 1628 1634 +6 |
9f75cdc to
eabf0a4
Compare
It turns out that doing a join is surprisingly expensive for the DB to do when room_membership table is larger than the disk cache.
eabf0a4 to
89c8859
Compare
richvdh
reviewed
Jul 19, 2019
Member
richvdh
left a comment
There was a problem hiding this comment.
couple of nits, generally lgtm
next we're going to need an index which includes membership?
| # Remember to update this number every time a change is made to database | ||
| # schema files, so the users will be informed on server restarts. | ||
| SCHEMA_VERSION = 55 | ||
| SCHEMA_VERSION = 56 |
Member
There was a problem hiding this comment.
what's the thinking behind the schema bump?
Member
Author
There was a problem hiding this comment.
Mainly that this stops people from rolling back, which will cause membership column to not be correctly updated.
| if pending_update: | ||
| self._clock.call_later( | ||
| 15.0, | ||
| run_as_background_process, |
Its a bit disingenuousto give LoggingTransaction lists to append callbacks to if we're not going to run the callbacks.
Member
Author
Maybe, though I don't think its the end of the world for postgres to scan all membership events in current state tbh. |
Member
Author
|
Although I guess we have the index: which might be good to add a |
This was referenced Jul 23, 2019
babolivier
added a commit
that referenced
this pull request
Aug 15, 2019
Synapse 1.3.0 (2019-08-15) ========================== Bugfixes -------- - Fix 500 Internal Server Error on `publicRooms` when the public room list was cached. ([\#5851](#5851)) Synapse 1.3.0rc1 (2019-08-13) ========================== Features -------- - Use `M_USER_DEACTIVATED` instead of `M_UNKNOWN` for errcode when a deactivated user attempts to login. ([\#5686](#5686)) - Add sd_notify hooks to ease systemd integration and allows usage of Type=Notify. ([\#5732](#5732)) - Synapse will no longer serve any media repo admin endpoints when `enable_media_repo` is set to False in the configuration. If a media repo worker is used, the admin APIs relating to the media repo will be served from it instead. ([\#5754](#5754), [\#5848](#5848)) - Synapse can now be configured to not join remote rooms of a given "complexity" (currently, state events) over federation. This option can be used to prevent adverse performance on resource-constrained homeservers. ([\#5783](#5783)) - Allow defining HTML templates to serve the user on account renewal attempt when using the account validity feature. ([\#5807](#5807)) Bugfixes -------- - Fix UISIs during homeserver outage. ([\#5693](#5693), [\#5789](#5789)) - Fix stack overflow in server key lookup code. ([\#5724](#5724)) - start.sh no longer uses deprecated cli option. ([\#5725](#5725)) - Log when we receive an event receipt from an unexpected origin. ([\#5743](#5743)) - Fix debian packaging scripts to correctly build sid packages. ([\#5775](#5775)) - Correctly handle redactions of redactions. ([\#5788](#5788)) - Return 404 instead of 403 when accessing /rooms/{roomId}/event/{eventId} for an event without the appropriate permissions. ([\#5798](#5798)) - Fix check that tombstone is a state event in push rules. ([\#5804](#5804)) - Fix error when trying to login as a deactivated user when using a worker to handle login. ([\#5806](#5806)) - Fix bug where user `/sync` stream could get wedged in rare circumstances. ([\#5825](#5825)) - The purge_remote_media.sh script was fixed. ([\#5839](#5839)) Deprecations and Removals ------------------------- - Synapse now no longer accepts the `-v`/`--verbose`, `-f`/`--log-file`, or `--log-config` command line flags, and removes the deprecated `verbose` and `log_file` configuration file options. Users of these options should migrate their options into the dedicated log configuration. ([\#5678](#5678), [\#5729](#5729)) - Remove non-functional 'expire_access_token' setting. ([\#5782](#5782)) Internal Changes ---------------- - Make Jaeger fully configurable. ([\#5694](#5694)) - Add precautionary measures to prevent future abuse of `window.opener` in default welcome page. ([\#5695](#5695)) - Reduce database IO usage by optimising queries for current membership. ([\#5706](#5706), [\#5738](#5738), [\#5746](#5746), [\#5752](#5752), [\#5770](#5770), [\#5774](#5774), [\#5792](#5792), [\#5793](#5793)) - Improve caching when fetching `get_filtered_current_state_ids`. ([\#5713](#5713)) - Don't accept opentracing data from clients. ([\#5715](#5715)) - Speed up PostgreSQL unit tests in CI. ([\#5717](#5717)) - Update the coding style document. ([\#5719](#5719)) - Improve database query performance when recording retry intervals for remote hosts. ([\#5720](#5720)) - Add a set of opentracing utils. ([\#5722](#5722)) - Cache result of get_version_string to reduce overhead of `/version` federation requests. ([\#5730](#5730)) - Return 'user_type' in admin API user endpoints results. ([\#5731](#5731)) - Don't package the sytest test blacklist file. ([\#5733](#5733)) - Replace uses of returnValue with plain return, as returnValue is not needed on Python 3. ([\#5736](#5736)) - Blacklist some flakey tests in worker mode. ([\#5740](#5740)) - Fix some error cases in the caching layer. ([\#5749](#5749)) - Add a prometheus metric for pending cache lookups. ([\#5750](#5750)) - Stop trying to fetch events with event_id=None. ([\#5753](#5753)) - Convert RedactionTestCase to modern test style. ([\#5768](#5768)) - Allow looping calls to be given arguments. ([\#5780](#5780)) - Set the logs emitted when checking typing and presence timeouts to DEBUG level, not INFO. ([\#5785](#5785)) - Remove DelayedCall debugging from the test suite, as it is no longer required in the vast majority of Synapse's tests. ([\#5787](#5787)) - Remove some spurious exceptions from the logs where we failed to talk to a remote server. ([\#5790](#5790)) - Improve performance when making `.well-known` requests by sharing the SSL options between requests. ([\#5794](#5794)) - Disable codecov GitHub comments on PRs. ([\#5796](#5796)) - Don't allow clients to send tombstone events that reference the room it's sent in. ([\#5801](#5801)) - Deny redactions of events sent in a different room. ([\#5802](#5802)) - Deny sending well known state types as non-state events. ([\#5805](#5805)) - Handle incorrectly encoded query params correctly by returning a 400. ([\#5808](#5808)) - Handle pusher being deleted during processing rather than logging an exception. ([\#5809](#5809)) - Return 502 not 500 when failing to reach any remote server. ([\#5810](#5810)) - Reduce global pauses in the events stream caused by expensive state resolution during persistence. ([\#5826](#5826)) - Add a lower bound to well-known lookup cache time to avoid repeated lookups. ([\#5836](#5836)) - Whitelist history visbility sytests in worker mode tests. ([\#5843](#5843))
anoadragon453
added a commit
that referenced
this pull request
Feb 19, 2020
richvdh
added a commit
that referenced
this pull request
May 1, 2020
phil-flex
pushed a commit
to phil-flex/synapse
that referenced
this pull request
Jun 16, 2020
…s_rooms (matrix-org#7387) populate_stats_process_rooms was added in matrix-org#5971 / v1.4.0; current_state_events_membership was added in matrix-org#5706 / v1.3.0. Fixes matrix-org#7380.
4 tasks
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Staring at
pg_stat_statementson matrix.org's synapse shows a lot of IO goes on queries that are of the formcurrent_state_events INNER JOIN room_memberships. This PR folds in themembershipcolumn into thecurrent_state_eventstable so that we don't have to do the join. This should hopefully mean a lot less IO, sincecurrent_state_eventstable can fit in the disk cache.The commits should be separately reviewable.