-
Notifications
You must be signed in to change notification settings - Fork 8.3k
"Part [...] is broken and needs manual correction" in 02443_detach_attach_partition.sh #72817
Copy link
Copy link
Closed
Labels
potential bugTo be reviewed by developers and confirmed/rejected.To be reviewed by developers and confirmed/rejected.
Description
The test does many DETACH/ATTACH PARTITION queries in multiple threads. In this test run, 1/7 of the ATTACHes printed error Part [...] is broken and needs manual correction. Some of the errors say No columns.txt, others say The specified key does not exist. This error happened for S3 disk.
% grep -aP '02443_detach_attach_partition\.sh.*alter_table0|alter_table0.*is broken and needs manual correction' clickhouse-server.log
2024.12.04 18:58:18.393461 [ 27333 ] {da519325-1446-4d02-bcdc-46a4904db88b} <Debug> executeQuery: (from [::1]:51446) (comment: 02443_detach_attach_partition.sh) DROP TABLE IF EXISTS alter_table0; (stage: Complete)
2024.12.04 18:58:18.405742 [ 27333 ] {c3cdfdcd-5d73-4f5e-b142-909405d42175} <Debug> executeQuery: (from [::1]:51446) (comment: 02443_detach_attach_partition.sh) CREATE TABLE alter_table0 (`a` UInt8, `b` Int16) ENGINE = ReplicatedMergeTree('/clickhouse/tables/02443_detach_attach_partition_test_yemh4i2j/alter_table', 'r1') ORDER BY a SETTINGS index_granularity = 46666, min_bytes_for_wide_part = 192054060, ratio_of_defaults_for_sparse_serialization = 1., replace_long_file_name_to_hash = true, max_file_name_length = 128, min_bytes_for_full_part_storage = 0, compact_parts_max_bytes_to_buffer = 20790197, compact_parts_max_granules_to_buffer = 210, compact_parts_merge_max_bytes_to_prefetch_part = 5724822, merge_max_block_size = 706, old_parts_lifetime = 480., prefer_fetch_merged_part_size_threshold = 1, vertical_merge_algorithm_min_rows_to_activate = 1, vertical_merge_algorithm_min_columns_to_activate = 100, min_merge_bytes_to_use_direct_io = 7649538519, index_granularity_bytes = 5728697, use_const_adaptive_granularity = true, enable_index_granularity_compression = false, concurrent_part_removal_threshold = 0, allow_vertical_merges_from_compact_to_wide_parts = true, cache_populated_by_fetch = true, marks_compress_block_size = 51142, primary_key_compress_block_size = 73388, use_primary_key_cache = true, prewarm_primary_key_cache = true, prewarm_mark_cache = false (stage: Complete)
2024.12.04 18:58:19.156920 [ 27932 ] {3746793b-3063-41dd-8b20-d334e0bfd21b} <Debug> executeQuery: (from [::1]:51500) (comment: 02443_detach_attach_partition.sh) SELECT engine FROM system.tables WHERE database=currentDatabase() AND table='alter_table0' (stage: Complete)
2024.12.04 18:58:21.350751 [ 28463 ] {12648b19-7c54-43cb-94a9-df9700df1952} <Debug> executeQuery: (from [::1]:51696) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 DETACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:21.668184 [ 27333 ] {5c8a2112-b9f4-45e3-8d46-4d7f1654a0e4} <Debug> executeQuery: (from [::1]:51752) (comment: 02443_detach_attach_partition.sh) INSERT INTO alter_table0 SELECT 2, 2 (stage: Complete)
2024.12.04 18:58:22.350890 [ 27932 ] {da8221f4-3ade-49c4-89ee-52a24f7176b6} <Debug> executeQuery: (from [::1]:51834) (comment: 02443_detach_attach_partition.sh) INSERT INTO alter_table0 SELECT 3, 3 (stage: Complete)
2024.12.04 18:58:23.045487 [ 27546 ] {4c29d4a4-0a2e-40cc-b263-6f6005fca231} <Debug> executeQuery: (from [::1]:51898) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 ATTACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:23.161969 [ 27333 ] {3a507f92-0b07-4804-8b20-d6791085702a} <Debug> executeQuery: (from [::1]:51926) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 DETACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:23.267646 [ 11188 ] {2af3daca-62ef-4821-91c6-d3f2ea6107f2} <Debug> executeQuery: (from [::1]:51960) (comment: 02443_detach_attach_partition.sh) INSERT INTO alter_table0 SELECT 4, 4 (stage: Complete)
2024.12.04 18:58:23.781696 [ 28468 ] {f430ad1c-3530-4d27-8197-f6d9986b3abd} <Debug> executeQuery: (from [::1]:51994) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 ATTACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:24.000030 [ 28472 ] {53ccceb5-9edd-49cf-9287-687d28e3192b} <Debug> executeQuery: (from [::1]:52038) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 DETACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:24.010479 [ 27932 ] {ff5b4ca1-3b11-4529-b0ec-608a5d47be20} <Debug> executeQuery: (from [::1]:52042) (comment: 02443_detach_attach_partition.sh) INSERT INTO alter_table0 SELECT 5, 5 (stage: Complete)
2024.12.04 18:58:24.410538 [ 27546 ] {43efb634-9bd4-4a96-9177-c7b092295bbf} <Debug> executeQuery: (from [::1]:52066) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 DETACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:24.805317 [ 27333 ] {cdf05059-26c4-48d4-9d1e-338ebb0a40a6} <Debug> executeQuery: (from [::1]:52128) (comment: 02443_detach_attach_partition.sh) INSERT INTO alter_table0 SELECT 6, 6 (stage: Complete)
2024.12.04 18:58:24.910773 [ 11264 ] {e5cffecf-784d-4ab9-ae2d-c4547e710a2b} <Debug> executeQuery: (from [::1]:52154) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 ATTACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:25.274270 [ 1143 ] {} <Error> test_yemh4i2j.alter_table0 (8be88c6f-26ff-4cad-b7d4-f99dd0bfebe0): Part /var/lib/clickhouse/disks/s3/store/8be/8be88c6f-26ff-4cad-b7d4-f99dd0bfebe0/detached/all_13_13_0/ is broken and needs manual correction. Reason: Code: 226. DB::Exception: No checksums.txt in part all_19_19_0. (NO_FILE_IN_DATA_PART), Stack trace (when copying this message, always include the lines below):
2024.12.04 18:58:25.555837 [ 444 ] {34385049-7142-418f-9662-ca94bdeb000d} <Debug> executeQuery: (from [::1]:52210) (comment: 02443_detach_attach_partition.sh) INSERT INTO alter_table0 SELECT 7, 7 (stage: Complete)
2024.12.04 18:58:26.290680 [ 26952 ] {af44e97c-c58c-4047-98a1-8a94f098e65c} <Debug> executeQuery: (from [::1]:52264) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 ATTACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:26.719975 [ 28463 ] {989c643c-8e94-4f0a-b491-d2199d229bea} <Debug> executeQuery: (from [::1]:35806) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 ATTACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:26.920587 [ 27546 ] {ee674a08-0d81-4451-8e40-98c17b066615} <Debug> executeQuery: (from [::1]:35824) (comment: 02443_detach_attach_partition.sh) INSERT INTO alter_table0 SELECT 9, 9 (stage: Complete)
2024.12.04 18:58:27.025550 [ 444 ] {3d3ff9f2-1349-4873-8fe2-711f31c50869} <Debug> executeQuery: (from [::1]:35838) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 DETACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:27.590785 [ 27546 ] {78c30bea-063b-406a-a7e8-3c3786e117c1} <Debug> executeQuery: (from [::1]:35890) (comment: 02443_detach_attach_partition.sh) INSERT INTO alter_table0 SELECT 10, 10 (stage: Complete)
2024.12.04 18:58:28.033821 [ 28468 ] {c782d87f-da36-4d8d-93e3-b038da2bb413} <Debug> executeQuery: (from [::1]:35942) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 ATTACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:28.978038 [ 2178 ] {769e2ca7-0640-43d3-b9b2-c7929c9957f5} <Debug> executeQuery: (from [::1]:36012) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 ATTACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:30.678479 [ 27334 ] {c7a6ac2d-3b35-45dc-a3b8-f3739d35ca75} <Debug> executeQuery: (from [::1]:36116) (comment: 02443_detach_attach_partition.sh) INSERT INTO alter_table0 SELECT 14, 14 (stage: Complete)
2024.12.04 18:58:30.964501 [ 27931 ] {e5775483-da82-428b-8f57-b1807360287a} <Debug> executeQuery: (from [::1]:36142) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 ATTACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:31.304116 [ 2177 ] {1c6b481c-9933-4fb8-9c64-18a40ad7f8f7} <Debug> executeQuery: (from [::1]:36162) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 ATTACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:31.432521 [ 27334 ] {7b8e91fc-e31f-4060-b1c4-8362226a37c0} <Debug> executeQuery: (from [::1]:36164) (comment: 02443_detach_attach_partition.sh) INSERT INTO alter_table0 SELECT 15, 15 (stage: Complete)
2024.12.04 18:58:33.110820 [ 27931 ] {9d78d431-f075-426e-81ca-43baa8a0e3e7} <Debug> executeQuery: (from [::1]:36276) (comment: 02443_detach_attach_partition.sh) INSERT INTO alter_table0 SELECT 17, 17 (stage: Complete)
2024.12.04 18:58:33.938770 [ 1138 ] {} <Error> test_yemh4i2j.alter_table0 (8be88c6f-26ff-4cad-b7d4-f99dd0bfebe0): Part /var/lib/clickhouse/disks/s3/store/8be/8be88c6f-26ff-4cad-b7d4-f99dd0bfebe0/detached/all_82_82_0/ is broken and needs manual correction. Reason: Code: 226. DB::Exception: No checksums.txt in part all_97_97_0. (NO_FILE_IN_DATA_PART), Stack trace (when copying this message, always include the lines below):
2024.12.04 18:58:34.319008 [ 27546 ] {7b791f86-ce9d-4acf-a25e-8dd9c03da8d6} <Debug> executeQuery: (from [::1]:36360) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 ATTACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:34.625751 [ 28468 ] {37d73773-0139-47d6-aaf1-db62be6477be} <Debug> executeQuery: (from [::1]:36364) (comment: 02443_detach_attach_partition.sh) INSERT INTO alter_table0 SELECT 19, 19 (stage: Complete)
2024.12.04 18:58:34.714539 [ 11188 ] {e6a4a6f1-f3eb-46e8-8d04-cbbcf0958e01} <Debug> executeQuery: (from [::1]:36366) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 DETACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:35.292474 [ 26952 ] {9cb9ee99-6933-498f-9bd7-640b5d6a67b5} <Debug> executeQuery: (from [::1]:36374) (comment: 02443_detach_attach_partition.sh) INSERT INTO alter_table0 SELECT 20, 20 (stage: Complete)
2024.12.04 18:58:35.399586 [ 28468 ] {634ab2d9-416a-48d2-b061-49039d64856c} <Debug> executeQuery: (from [::1]:36388) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 ATTACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:35.806230 [ 28472 ] {b74b1902-005b-446a-bb0f-b9213974a2eb} <Debug> executeQuery: (from [::1]:36398) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 DETACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:36.335833 [ 26952 ] {70f3fa49-dbdc-49e0-bcff-58858d4497b8} <Debug> executeQuery: (from [::1]:58144) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 DETACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:36.775338 [ 27546 ] {dd2256d1-05f7-4940-8964-a557a6af885e} <Debug> executeQuery: (from [::1]:58196) (comment: 02443_detach_attach_partition.sh) INSERT INTO alter_table0 SELECT 22, 22 (stage: Complete)
2024.12.04 18:58:37.540444 [ 27932 ] {5414f8cb-802b-4c05-80cf-e8f215e23451} <Debug> executeQuery: (from [::1]:58228) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 DETACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:37.592260 [ 27931 ] {af35fe9b-f3eb-4e21-8437-79abebc71d1f} <Debug> executeQuery: (from [::1]:58234) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 ATTACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:38.050126 [ 11188 ] {d7ed4345-6263-4e2b-83d1-28bed57a63f9} <Debug> executeQuery: (from [::1]:58264) (comment: 02443_detach_attach_partition.sh) INSERT INTO alter_table0 SELECT 24, 24 (stage: Complete)
2024.12.04 18:58:38.201182 [ 28472 ] {853c0008-ae49-483a-8888-41e73945c66d} <Debug> executeQuery: (from [::1]:58270) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 ATTACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:38.623452 [ 2177 ] {f7783333-bc2d-42e3-8584-ef89e22a2c16} <Debug> executeQuery: (from [::1]:58276) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 ATTACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:38.888405 [ 2178 ] {36cfb06f-892a-4c64-b59c-cf99a4c0efab} <Debug> executeQuery: (from [::1]:58306) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 DETACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:39.781349 [ 27334 ] {9e9bde14-9326-465f-a64b-2c6e5e9c7c17} <Debug> executeQuery: (from [::1]:58348) (comment: 02443_detach_attach_partition.sh) INSERT INTO alter_table0 SELECT 26, 26 (stage: Complete)
2024.12.04 18:58:40.519612 [ 28463 ] {04662de4-10ef-456d-89c1-0e65bc56cef7} <Debug> executeQuery: (from [::1]:58376) (comment: 02443_detach_attach_partition.sh) INSERT INTO alter_table0 SELECT 27, 27 (stage: Complete)
2024.12.04 18:58:40.684333 [ 27932 ] {7afcf863-7963-4555-9229-9a0b0cd575ed} <Debug> executeQuery: (from [::1]:58384) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 DETACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:40.768769 [ 2177 ] {f2d841d5-0ec9-45ac-ad1e-4d9189ef6a52} <Debug> executeQuery: (from [::1]:58394) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 DETACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:41.739079 [ 1135 ] {} <Error> test_yemh4i2j.alter_table0 (8be88c6f-26ff-4cad-b7d4-f99dd0bfebe0): Part /var/lib/clickhouse/disks/s3/store/8be/8be88c6f-26ff-4cad-b7d4-f99dd0bfebe0/detached/all_148_148_0/ is broken and needs manual correction. Reason: Code: 226. DB::Exception: No count.txt in part all_155_155_0. (NO_FILE_IN_DATA_PART), Stack trace (when copying this message, always include the lines below):
2024.12.04 18:58:41.864133 [ 11188 ] {f110228f-e67e-46f7-a9cf-ef5bfca7d8c4} <Debug> executeQuery: (from [::1]:58436) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 ATTACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:41.922814 [ 26952 ] {c701e5ef-9336-4266-98a9-00e5b1b53685} <Debug> executeQuery: (from [::1]:58444) (comment: 02443_detach_attach_partition.sh) INSERT INTO alter_table0 SELECT 29, 29 (stage: Complete)
2024.12.04 18:58:42.254674 [ 27334 ] {d1ad58f6-5b01-4d0f-a57d-e00fa5bf2b04} <Debug> executeQuery: (from [::1]:58452) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 DETACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:42.284101 [ 27932 ] {06d2cb8c-718c-4ab8-b1c2-29fad3626f26} <Debug> executeQuery: (from [::1]:58462) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 DETACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:43.122086 [ 27546 ] {28bdba38-271c-4df7-8897-fcab2fd4cd1f} <Debug> executeQuery: (from [::1]:58514) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 DETACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:43.361075 [ 28463 ] {60cf8c2d-41b1-488e-aa95-ef078077b1fb} <Debug> executeQuery: (from [::1]:58536) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 ATTACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:43.726172 [ 27334 ] {22055277-5f06-49ec-87b4-970947b01a37} <Debug> executeQuery: (from [::1]:58586) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 DETACH PARTITION ID 'all'; (stage: Complete)
2024.12.04 18:58:46.276052 [ 28472 ] {e6afd7b7-d9cf-444d-ab91-931caaac018c} <Debug> executeQuery: (from [::1]:58692) (comment: 02443_detach_attach_partition.sh) SYSTEM SYNC REPLICA alter_table0 (stage: Complete)
2024.12.04 18:58:47.309640 [ 2178 ] {efadab2c-daa5-4b14-8327-a0120bd01ca5} <Debug> executeQuery: (from [::1]:46354) (comment: 02443_detach_attach_partition.sh) ALTER TABLE alter_table0 ATTACH PARTITION ID 'all' (stage: Complete)
2024.12.04 18:58:49.750130 [ 27932 ] {3538b5ba-3a01-4406-a1fe-638cd5dbdf3c} <Debug> executeQuery: (from [::1]:46458) (comment: 02443_detach_attach_partition.sh) SYSTEM SYNC REPLICA alter_table0 (stage: Complete)
2024.12.04 18:58:50.769309 [ 2178 ] {ce0e0b27-338f-4939-a901-ce1411df5700} <Debug> executeQuery: (from [::1]:46526) (comment: 02443_detach_attach_partition.sh) OPTIMIZE TABLE alter_table0 FINAL DEDUPLICATE (stage: Complete)
2024.12.04 18:58:51.984787 [ 27546 ] {dbccbfef-3eaf-4a6a-a74d-cbc1b7174cc1} <Debug> executeQuery: (from [::1]:46602) (comment: 02443_detach_attach_partition.sh) SELECT count(), sum(b) FROM alter_table0 (stage: Complete)
2024.12.04 18:58:53.136522 [ 28472 ] {b7e60d67-bcef-4a0d-8226-4aa2f6de2944} <Debug> executeQuery: (from [::1]:46676) (comment: 02443_detach_attach_partition.sh) DROP TABLE alter_table0 (stage: Complete)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
potential bugTo be reviewed by developers and confirmed/rejected.To be reviewed by developers and confirmed/rejected.