Skip to content

Replicated database: check metadata consistency on server startup #24880

@Blackmorse

Description

@Blackmorse

The problem
Suppose we have a database (e.g. db) with Engine = Replicated on the cluster. Let's make manually delete on one of the hosts
rm -R /var/lib/clickhouse/store/<db_uuid_prefix>/<db_uuid>/*
(or even better)
rm -R /var/lib/clickhouse/store
After restarting this host there are no tables restored at db

Feature
Comparing to existing replication mechanizm of ReplicatedMergeTrees:
ReplicatedMergeTree has data, and this data represented as a parts. When manually deleting parts
rm -r -R /var/lib/clickhouse/<table_uuid_prefix>/<table_uuid>/*
after a while Clickhouse will detect missing parts and restore them from the replica.

In case of Replicated database, table's sql-files can be treated as database data, and similar replication mechanizm expected

This makes process of restoring the lost data/metadata on the replicas almost fully-automated

Use case
We are mounting metadata folder to separate disk (something very reliable, like RAIDs). E.g. when disk with store/ is corrupted, we have to manually copy metadata from replica and restore symlinks at metadata/ folder, (for Atomic databases).
In case of using Replicated Database with improvements described above, when disk with store/ is corrupted, data and metadata will be restored automatically without manual intervention

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions