Inconsistent metadata in ReplicatedVersionedCollapsingMergeTree after upgrading to v20.5 #12093
Description
We have upgraded to v20.5.2.7 from v20.1.11.73
And now CH won't start up and we're seeing error where local PK metadata differ from ZK metadata.
Existing table metadata in ZooKeeper differs in primary key. Stored in ZooKeeper: mac, port_id, topology_type, first_seen_at, last_seen_at, local: mac, port_id, topology_type, first_seen_at, last_seen_at, version
- ZK: mac, port_id, topology_type, first_seen_at, last_seen_at
+ local mac, port_id, topology_type, first_seen_at, last_seen_at, versionAs you can see, CH somehow added version field to the PK in table with ReplicatedVersionedCollapsingMergeTree engine
How to resolve this?
Stacktrace:
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x95c923d in /usr/bin/clickhouse
2. ? @ 0xe5d7da6 in /usr/bin/clickhouse
3. DB::ReplicatedMergeTreeTableMetadata::checkEquals(DB::ReplicatedMergeTreeTableMetadata const&, DB::ColumnsDescription const&, DB::Context const&) const @ 0xe5d5c49 in /usr/bin/clickhouse
4. DB::StorageReplicatedMergeTree::checkTableStructure(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) @ 0xe27d184 in /usr/bin/clickhouse
5. DB::StorageReplicatedMergeTree::StorageReplicatedMergeTree(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, DB::StorageID const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::StorageInMemoryMetadata const&, DB::Context&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::MergeTreeData::MergingParams const&, std::__1::unique_ptr<DB::MergeTreeSettings, std::__1::default_delete<DB::MergeTreeSettings> >, bool) @ 0xe2a4e79 in /usr/bin/clickhouse
6. ? @ 0xe5e382d in /usr/bin/clickhouse
7. std::__1::__function::__func<std::__1::shared_ptr<DB::IStorage> (*)(DB::StorageFactory::Arguments const&), std::__1::allocator<std::__1::shared_ptr<DB::IStorage> (*)(DB::StorageFactory::Arguments const&)>, std::__1::shared_ptr<DB::IStorage> (DB::StorageFactory::Arguments const&)>::operator()(DB::StorageFactory::Arguments const&) @ 0xe5e7a37 in /usr/bin/clickhouse
8. DB::StorageFactory::get(DB::ASTCreateQuery const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context&, DB::Context&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const @ 0xe1e7d2d in /usr/bin/clickhouse
9. DB::createTableFromAST(DB::ASTCreateQuery, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context&, bool) @ 0xdba5749 in /usr/bin/clickhouse
10. ? @ 0xdb98528 in /usr/bin/clickhouse
11. ? @ 0xdb98ee2 in /usr/bin/clickhouse
12. ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0x95f7987 in /usr/bin/clickhouse
13. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'()::operator()() const @ 0x95f80fa in /usr/bin/clickhouse
14. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x95f6e97 in /usr/bin/clickhouse
15. ? @ 0x95f5383 in /usr/bin/clickhouse
16. start_thread @ 0x74a4 in /lib/x86_64-linux-gnu/libpthread-2.24.so
17. __clone @ 0xe8d0f in /lib/x86_64-linux-gnu/libc-2.24.so
(version 20.5.2.7 (official build))
2020.07.02 16:40:02.369840 [ 4862 ] {} <Error> Application: DB::Exception: Existing table metadata in ZooKeeper differs in primary key. Stored in ZooKeeper: mac, port_id, topology_type, first_seen_at, last_seen_at, local: mac, port_id, topology_type, first_seen_at, last_seen_at, version: Cannot attach table `default`.`map___mac__port` from metadata file /var/lib/clickhouse/metadata/default/map___mac__port.sql from query ATTACH TABLE map___mac__port (`first_seen_at` DateTime, `last_seen_at` DateTime, `mac` UInt64, `port_id` UUID, `topology_type` Enum8('Edge' = 1, 'Infrastructure' = 2), `sign` Int8, `version` UInt32) ENGINE = ReplicatedVersionedCollapsingMergeTree('/clickhouse/tables/{shard}/map___mac__port', '{replica}', sign, version) PARTITION BY toDate(first_seen_at) ORDER BY (mac, port_id, topology_type, first_seen_at, last_seen_at) SETTINGS index_granularity = 8192```