This repository was archived by the owner on Apr 26, 2024. It is now read-only.
This repository was archived by the owner on Apr 26, 2024. It is now read-only.
Port script cannot be run twice on the same DB #8619
Description
The port script should be able to run multiple times on the same DB, to allow incremental porting.
This fails with:
(env) ~/temp/synapse >>> synapse_port_db --sqlite-database matrix.db.snapshot --postgres-config homeserver-postgres.yaml [130]
Preparing sqlite3...
2020-10-21 18:04:54,927 - synapse.storage.prepare_database - 102 - INFO - ['main', 'state']: Checking existing schema version
2020-10-21 18:04:54,928 - synapse.storage.prepare_database - 107 - INFO - ['main', 'state']: Existing schema is 58 (+20 deltas)
2020-10-21 18:04:54,929 - synapse.storage.databases.main - 601 - INFO - Checking database for consistency with configuration...
2020-10-21 18:04:54,929 - synapse.storage.prepare_database - 379 - INFO - Applying schema deltas for v58
2020-10-21 18:04:54,929 - synapse.storage.prepare_database - 503 - INFO - Schema now up to date
2020-10-21 18:04:54,931 - synapse.storage.util.id_generators - 57 - INFO - initialising stream generator for events(stream_ordering)
2020-10-21 18:04:54,931 - synapse.storage.util.id_generators - 57 - INFO - initialising stream generator for events(stream_ordering)
2020-10-21 18:04:54,931 - synapse.storage.util.id_generators - 57 - INFO - initialising stream generator for ex_outlier_stream(event_stream_ordering)
Preparing psycopg2...
2020-10-21 18:04:54,946 - synapse.storage.prepare_database - 102 - INFO - ['main', 'state']: Checking existing schema version
2020-10-21 18:04:54,948 - synapse.storage.prepare_database - 107 - INFO - ['main', 'state']: Existing schema is 58 (+22 deltas)
2020-10-21 18:04:54,948 - synapse.storage.databases.main - 601 - INFO - Checking database for consistency with configuration...
2020-10-21 18:04:54,949 - synapse.storage.prepare_database - 379 - INFO - Applying schema deltas for v58
2020-10-21 18:04:54,949 - synapse.storage.prepare_database - 503 - INFO - Schema now up to date
--- Logging error ---
Traceback (most recent call last):
File "/usr/lib/python3.8/logging/__init__.py", line 1081, in emit
msg = self.format(record)
File "/usr/lib/python3.8/logging/__init__.py", line 925, in format
return fmt.format(record)
File "/usr/lib/python3.8/logging/__init__.py", line 664, in format
record.message = record.getMessage()
File "/usr/lib/python3.8/logging/__init__.py", line 369, in getMessage
msg = msg % self.args
TypeError: not enough arguments for format string
Call stack:
File "/home/twilight/temp/synapse/env/bin/synapse_port_db", line 1097, in <module>
start()
File "/home/twilight/temp/synapse/env/bin/synapse_port_db", line 1092, in start
reactor.run()
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/twisted/internet/base.py", line 1283, in run
self.mainLoop()
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/twisted/internet/base.py", line 1292, in mainLoop
self.runUntilCurrent()
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/metrics/__init__.py", line 538, in f
ret = func(*args, **kwargs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/twisted/internet/base.py", line 886, in runUntilCurrent
f(*a, **kw)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 460, in callback
self._startRunCallbacks(result)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
self._runCallbacks()
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1475, in gotResult
_inlineCallbacks(r, g, status)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
result = g.send(result)
File "/home/twilight/temp/synapse/env/bin/synapse_port_db", line 547, in run
self.postgres_store = self.build_db_store(
File "/home/twilight/temp/synapse/env/bin/synapse_port_db", line 497, in build_db_store
store = Store(DatabasePool(hs, db_config, engine), db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/client_ips.py", line 34, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/deviceinbox.py", line 286, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/devices.py", line 704, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/events_bg_updates.py", line 32, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/media_repository.py", line 27, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/registration.py", line 784, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/registration.py", line 39, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/room.py", line 866, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/roommember.py", line 826, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/search.py", line 92, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/state/bg_updates.py", line 184, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/state.py", line 323, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/roommember.py", line 57, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/events_worker.py", line 83, in __init__
self._stream_id_gen = MultiWriterIdGenerator(
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/util/id_generators.py", line 263, in __init__
self._sequence_gen.check_consistency(
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/util/sequence.py", line 118, in check_consistency
logger.warning(
Message: 'Postgres sequence %s is behind table %s: %d < %d'
Arguments: (1, 41)
2020-10-21 18:04:54,960 - synapse_port_db - 639 - ERROR -
Traceback (most recent call last):
File "/home/twilight/temp/synapse/env/bin/synapse_port_db", line 547, in run
self.postgres_store = self.build_db_store(
File "/home/twilight/temp/synapse/env/bin/synapse_port_db", line 497, in build_db_store
store = Store(DatabasePool(hs, db_config, engine), db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/client_ips.py", line 34, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/deviceinbox.py", line 286, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/devices.py", line 704, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/events_bg_updates.py", line 32, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/media_repository.py", line 27, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/registration.py", line 784, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/registration.py", line 39, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/room.py", line 866, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/roommember.py", line 826, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/search.py", line 92, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/state/bg_updates.py", line 184, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/state.py", line 323, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/roommember.py", line 57, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/events_worker.py", line 83, in __init__
self._stream_id_gen = MultiWriterIdGenerator(
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/util/id_generators.py", line 263, in __init__
self._sequence_gen.check_consistency(
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/util/sequence.py", line 123, in check_consistency
raise IncorrectDatabaseSetup(
synapse.storage.engines._base.IncorrectDatabaseSetup:
Postgres sequence 'events_stream_seq' is inconsistent with associated
table 'events'. This can happen if Synapse has been downgraded and
then upgraded again, or due to a bad migration.
To fix this error, shut down Synapse (including any and all workers)
and run the following SQL:
SELECT setval('events_stream_seq', (
SELECT GREATEST(MAX(stream_ordering), 0) FROM events
));
See docs/postgres.md for more information.
Traceback (most recent call last):
File "/home/twilight/temp/synapse/env/bin/synapse_port_db", line 547, in run
self.postgres_store = self.build_db_store(
File "/home/twilight/temp/synapse/env/bin/synapse_port_db", line 497, in build_db_store
store = Store(DatabasePool(hs, db_config, engine), db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/client_ips.py", line 34, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/deviceinbox.py", line 286, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/devices.py", line 704, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/events_bg_updates.py", line 32, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/media_repository.py", line 27, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/registration.py", line 784, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/registration.py", line 39, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/room.py", line 866, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/roommember.py", line 826, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/search.py", line 92, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/state/bg_updates.py", line 184, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/state.py", line 323, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/roommember.py", line 57, in __init__
super().__init__(database, db_conn, hs)
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/databases/main/events_worker.py", line 83, in __init__
self._stream_id_gen = MultiWriterIdGenerator(
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/util/id_generators.py", line 263, in __init__
self._sequence_gen.check_consistency(
File "/home/twilight/temp/synapse/env/lib/python3.8/site-packages/synapse/storage/util/sequence.py", line 123, in check_consistency
raise IncorrectDatabaseSetup(
synapse.storage.engines._base.IncorrectDatabaseSetup:
Postgres sequence 'events_stream_seq' is inconsistent with associated
table 'events'. This can happen if Synapse has been downgraded and
then upgraded again, or due to a bad migration.
To fix this error, shut down Synapse (including any and all workers)
and run the following SQL:
SELECT setval('events_stream_seq', (
SELECT GREATEST(MAX(stream_ordering), 0) FROM events
));
See docs/postgres.md for more information.
Traceback (most recent call last):
File "/home/twilight/temp/synapse/env/bin/synapse_port_db", line 1104, in <module>
sys.stderr.write(end_error)
TypeError: write() argument must be str, not IncorrectDatabaseSetup