-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Replicated database: check metadata consistency on server startup #24880
Description
The problem
Suppose we have a database (e.g. db) with Engine = Replicated on the cluster. Let's make manually delete on one of the hosts
rm -R /var/lib/clickhouse/store/<db_uuid_prefix>/<db_uuid>/*
(or even better)
rm -R /var/lib/clickhouse/store
After restarting this host there are no tables restored at db
Feature
Comparing to existing replication mechanizm of ReplicatedMergeTrees:
ReplicatedMergeTree has data, and this data represented as a parts. When manually deleting parts
rm -r -R /var/lib/clickhouse/<table_uuid_prefix>/<table_uuid>/*
after a while Clickhouse will detect missing parts and restore them from the replica.
In case of Replicated database, table's sql-files can be treated as database data, and similar replication mechanizm expected
This makes process of restoring the lost data/metadata on the replicas almost fully-automated
Use case
We are mounting metadata folder to separate disk (something very reliable, like RAIDs). E.g. when disk with store/ is corrupted, we have to manually copy metadata from replica and restore symlinks at metadata/ folder, (for Atomic databases).
In case of using Replicated Database with improvements described above, when disk with store/ is corrupted, data and metadata will be restored automatically without manual intervention