This repository was archived by the owner on Apr 26, 2024. It is now read-only.
Draft: Add support for MSC2716 marker events#10420
Closed
MadLittleMods wants to merge 45 commits intomadlittlemods/2716-backfill-historical-events-for-federationfrom
Closed
Draft: Add support for MSC2716 marker events#10420MadLittleMods wants to merge 45 commits intomadlittlemods/2716-backfill-historical-events-for-federationfrom
MadLittleMods wants to merge 45 commits intomadlittlemods/2716-backfill-historical-events-for-federationfrom
Conversation
MadLittleMods
commented
Jul 16, 2021
| state = None | ||
|
|
||
| # Get missing pdus if necessary. | ||
| # We don't need to worry about outliers because TODO! |
Contributor
Author
There was a problem hiding this comment.
Unrelated to the PR but just want to better clarify why we skip outliers in this step. Any insight here?
- Outliers are outside of the normal DAG
- We don't have state for outliers
- We trust their auth events over calculating them
MadLittleMods
commented
Jul 17, 2021
MadLittleMods
commented
Jul 17, 2021
But running into the `sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group` error
MadLittleMods
commented
Jul 17, 2021
| for e in events: | ||
| if not is_sender_erased(e, erased_senders): | ||
| erased = is_sender_erased(e, erased_senders) | ||
| if not erased: |
Contributor
Author
There was a problem hiding this comment.
TODO: can revert
MadLittleMods
commented
Jul 17, 2021
| SELECT event_id FROM insertion_events | ||
| WHERE next_chunk_id = ? | ||
| ) | ||
| """ |
Contributor
Author
There was a problem hiding this comment.
I'm unsure if this query is correct 🎯
# Doesn't seem to be supported in postgres
sql = """
DELETE extrem FROM insertion_event_extremities as extrem
INNER JOIN insertion_events AS i USING (event_id)
WHERE i.next_chunk_id = ?
"""
sql = """
DELETE FROM insertion_event_extremities WHERE event_id IN (
SELECT event_id FROM insertion_events
WHERE next_chunk_id = ?
)
"""
12 tasks
…ation' into madlittlemods/2716-marker-events Conflicts: synapse/storage/databases/main/events.py
```
2021-07-13 02:27:57,810 - synapse.handlers.federation - 1248 - ERROR - GET-4 - Failed to backfill from hs1 because NOT NULL constraint failed: event_to_state_groups.state_group
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1216, in try_backfill
await self.backfill(
File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1035, in backfill
await self._auth_and_persist_event(dest, event, context, backfilled=True)
File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2222, in _auth_and_persist_event
await self._run_push_actions_and_persist_event(event, context, backfilled)
File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2244, in _run_push_actions_and_persist_event
await self.persist_events_and_notify(
File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 3290, in persist_events_and_notify
events, max_stream_token = await self.storage.persistence.persist_events(
File "/usr/local/lib/python3.8/site-packages/synapse/logging/opentracing.py", line 774, in _trace_inner
return await func(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 320, in persist_events
ret_vals = await yieldable_gather_results(enqueue, partitioned.items())
File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 237, in handle_queue_loop
ret = await self._per_item_callback(
File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 577, in _persist_event_batch
await self.persist_events_store._persist_events_and_state_updates(
File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 176, in _persist_events_and_state_updates
await self.db_pool.runInteraction(
File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 681, in runInteraction
result = await self.runWithConnection(
File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 770, in runWithConnection
return await make_deferred_yieldable(
File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 238, in inContext
result = inContext.theWork() # type: ignore[attr-defined]
File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 254, in <lambda>
inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 83, in callWithContext
return func(*args, **kw)
File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection
compat.reraise(excValue, excTraceback)
File "/usr/local/lib/python3.8/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction
return function(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/twisted/python/compat.py", line 403, in reraise
raise exception.with_traceback(traceback)
File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection
result = func(conn, *args, **kw)
File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 765, in inner_func
return func(db_conn, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 549, in new_transaction
r = func(cursor, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/synapse/logging/utils.py", line 69, in wrapped
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 385, in _persist_events_txn
self._store_event_state_mappings_txn(txn, events_and_contexts)
File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 2065, in _store_event_state_mappings_txn
self.db_pool.simple_insert_many_txn(
File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 923, in simple_insert_many_txn
txn.execute_batch(sql, vals)
File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 280, in execute_batch
self.executemany(sql, args)
File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 300, in executemany
self._do_execute(self.txn.executemany, sql, *args)
File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 330, in _do_execute
return func(sql, *args)
sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group
```
…nts" This reverts commit 187ab28.
As per comment Fixes #10456 See also element-hq/element-android#3725
…ation' into madlittlemods/2716-marker-events Conflicts: scripts-dev/complement.sh synapse/storage/databases/main/events.py
Synapse 1.39.0rc2 (2021-07-22) ============================== Bugfixes -------- - Always include `device_one_time_keys_count` key in `/sync` response to work around a bug in Element Android that broke encryption for new devices. ([\#10457](#10457)) Internal Changes ---------------- - Move docker image build to Github Actions. ([\#10416](#10416))
* Fix no-access-token bug in deactivation tests * Support MSC2033: Device ID on whoami * Test for appservices too MSC: matrix-org/matrix-spec-proposals#2033 The MSC has passed FCP, which means stable endpoints can be used.
IE11 doesn't support Content-Security-Policy but it has support for a non-standard X-Content-Security-Policy header, which only supports the sandbox directive. This prevents script execution, so it at least offers some protection against media repo-based attacks. Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
Updates CI and the helper script t ensures all tests are run (in parallel).
…10407) Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
Fix a circular import, which was causing exceptions on boot if SAML was configured.
tighten up some of the typing in CachedCall, which is going to be needed when Twisted 21.7 brings better typing on Deferred.
Mostly this involves decorating a few Deferred declarations with extra type hints. We wrap the types in quotes to avoid runtime errors when running against older versions of Twisted that don't have generics on Deferred.
Synapse 1.39.0rc3 (2021-07-28) ============================== Bugfixes -------- - Fix a bug introduced in Synapse 1.38 which caused an exception at startup when SAML authentication was enabled. ([\#10477](#10477)) - Fix a long-standing bug where Synapse would not inform clients that a device had exhausted its one-time-key pool, potentially causing problems decrypting events. ([\#10485](#10485)) - Fix reporting old R30 stats as R30v2 stats. Introduced in v1.39.0rc1. ([\#10486](#10486)) Internal Changes ---------------- - Fix an error which prevented the Github Actions workflow to build the docker images from running. ([\#10461](#10461)) - Fix release script to correctly version debian changelog when doing RCs. ([\#10465](#10465))
… any scrollback history (MSC2716) (#10245) * Make historical messages available to federated servers Part of MSC2716: matrix-org/matrix-spec-proposals#2716 Follow-up to #9247 * Debug message not available on federation * Add base starting insertion point when no chunk ID is provided * Fix messages from multiple senders in historical chunk Follow-up to #9247 Part of MSC2716: matrix-org/matrix-spec-proposals#2716 --- Previously, Synapse would throw a 403, `Cannot force another user to join.`, because we were trying to use `?user_id` from a single virtual user which did not match with messages from other users in the chunk. * Remove debug lines * Messing with selecting insertion event extremeties * Move db schema change to new version * Add more better comments * Make a fake requester with just what we need See #10276 (comment) * Store insertion events in table * Make base insertion event float off on its own See #10250 (comment) Conflicts: synapse/rest/client/v1/room.py * Validate that the app service can actually control the given user See #10276 (comment) Conflicts: synapse/rest/client/v1/room.py * Add some better comments on what we're trying to check for * Continue debugging * Share validation logic * Add inserted historical messages to /backfill response * Remove debug sql queries * Some marker event implemntation trials * Clean up PR * Rename insertion_event_id to just event_id * Add some better sql comments * More accurate description * Add changelog * Make it clear what MSC the change is part of * Add more detail on which insertion event came through * Address review and improve sql queries * Only use event_id as unique constraint * Fix test case where insertion event is already in the normal DAG * Remove debug changes * Switch to chunk events so we can auth via power_levels Previously, we were using `content.chunk_id` to connect one chunk to another. But these events can be from any `sender` and we can't tell who should be able to send historical events. We know we only want the application service to do it but these events have the sender of a real historical message, not the application service user ID as the sender. Other federated homeservers also have no indicator which senders are an application service on the originating homeserver. So we want to auth all of the MSC2716 events via power_levels and have them be sent by the application service with proper PL levels in the room. * Switch to chunk events for federation * Add unstable room version to support new historical PL * Fix federated events being rejected for no state_groups Add fix from #10439 until it merges. * Only connect base insertion event to prev_event_ids Per discussion with @erikjohnston, https://matrix.to/#/!UytJQHLQYfvYWsGrGY:jki.re/$12bTUiObDFdHLAYtT7E-BvYRp3k_xv8w0dUQHibasJk?via=jki.re&via=matrix.org * Make it possible to get the room_version with txn * Allow but ignore historical events in unsupported room version See #10245 (comment) We can't reject historical events on unsupported room versions because homeservers without knowledge of MSC2716 or the new room version don't reject historical events either. Since we can't rely on the auth check here to stop historical events on unsupported room versions, I've added some additional checks in the processing/persisting code (`synapse/storage/databases/main/events.py` -> `_handle_insertion_event` and `_handle_chunk_event`). I've had to do some refactoring so there is method to fetch the room version by `txn`. * Move to unique index syntax See #10245 (comment) * High-level document how the insertion->chunk lookup works * Remove create_event fallback for room_versions See https://github.com/matrix-org/synapse/pull/10245/files#r677641879 * Use updated method name
An error occurred while trying to automatically change base from
madlittlemods/2716-backfill-historical-events-for-federation
to
develop
July 28, 2021 15:46
Now that `Deferred` is a generic class, let's update `ObeservableDeferred` to follow suit.
Conflicts: scripts-dev/complement.sh synapse/event_auth.py synapse/storage/databases/main/event_federation.py synapse/storage/databases/main/events.py synapse/storage/schema/main/delta/61/01insertion_event_lookups.sql
Contributor
Author
|
Closing this PR in favor of #10498 because GitHub failed to change the base: "An error occurred while trying to automatically change base from madlittlemods/2716-backfill-historical-events-for-federation to develop" and messed up the history. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
13 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
..
Dev notes
event_backward_extremitieshs2 asks for the list of
/messagesCall stacks for backfilling
From the remote server
RoomMessageListRestServlet.on_GET:/rooms/(?P<room_id>[^/]*)/messages$(hs2asks for more messages)get_messagesmaybe_backfill_maybe_backfill_innerget_oldest_events_with_depth_in_roomtry_backfillbackfillfederation_client.backfilltransport_layer.backfill-> calls/backfillFrom the origin server
FederationBackfillServlet.on_GET:/backfill/(?P<room_id>[^/]*)/?on_backfill_requestget_backfill_eventsWhere to fetch the missing insertion event?
_persist_events_and_state_updates->_persist_events_txn->_update_metadata_tables_txn(synapse/storage/databases/main/events.py#L1442)async/awaitstuff and probably not best to call federation code from thereon_receive_pdusynapse/handlers/federation.py#L193How to fetch the missing insertion event?
federation_client.backfill(synapse/federation/federation_client.py#L203): Not for fetching a specific event. Only goes off of extremities and goes backwards from there._get_events_and_persist:synapse/handlers/federation.py#L1303federation_client.get_pdu:synapse/federation/federation_client.py#L241federation_client.transport_layer.get_event:synapse/federation/transport/client.py#L76Pull Request Checklist
EventStoretoEventWorkerStore.".code blocks.