Fast tests hung because of the DatabaseCatalog hiding unused dirs feature #83719
Description
Describe the bug
We hang waiting there:
6400060:2025.07.14 12:50:27.696690 [ 203035 ] {9e8b1b5b-9211-4f54-ac05-37853d87d781} <Debug> executeQuery: (from [::ffff:127.0.0.1]:50580) (comment: 03541_table_without_insertable_columns.sql) DROP DATABASE IF EXISTS test_9ndmpbdw SYNC (stage: Complete)
6400061:2025.07.14 12:50:27.697502 [ 203035 ] {9e8b1b5b-9211-4f54-ac05-37853d87d781} <Debug> DatabaseCatalog: Waiting for table 65435e22-ee03-48d8-bc0b-04fc63e1e952 to be finally dropped
6400064:2025.07.14 12:50:27.697679 [ 203035 ] {9e8b1b5b-9211-4f54-ac05-37853d87d781} <Debug> DatabaseCatalog: Done waiting for the table 65435e22-ee03-48d8-bc0b-04fc63e1e952 to be dropped. The outcome: table dropped successfully
6400065:2025.07.14 12:50:27.697683 [ 203035 ] {9e8b1b5b-9211-4f54-ac05-37853d87d781} <Debug> DatabaseCatalog: Waiting for table 00000000-0000-0000-0000-000000000001 to be finally dropped
There are a ton of these messages:
6400066:2025.07.14 12:50:27.697747 [ 272557 ] {} <Information> DatabaseCatalog: Removing data directory store/000/00000000-0000-0000-0000-000000000001/ of dropped table test_9ndmpbdw.no_insertable (00000000-0000-0000-0000-000000000001) from disk default
6400067:2025.07.14 12:50:27.703561 [ 272557 ] {} <Error> DatabaseCatalog: Cannot drop table test_9ndmpbdw.no_insertable (00000000-0000-0000-0000-000000000001). Will retry later.: std::exception. Code: 1001, type: std::__1::filesystem::filesystem_error, e.what() = filesystem error: in remove_all: Permission denied ["/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ci/tmp/run_r0/store/000/00000000-0000-0000-0000-000000000001/"], Stack trace (when copying this message, always include the lines below):
It seems that permissions were revoked here:
6399643:2025.07.14 12:50:27.370824 [ 169708 ] {BgSchPool::6d0e2f55-b0de-4144-8a4d-594b1e3dfe91} <Information> DatabaseCatalog: Removing access rights for unused directory store/000/00000000-0000-0000-0000-000000000001/ from disk default (will remove it when timeout exceed)
And it also seems that test_9ndmpbdw.no_insertable has completely different location under store:
6399997:2025.07.14 12:50:27.611402 [ 203035 ] {32e4bc22-3a43-4971-9268-b3a640e2b51a} <Debug> executeQuery: (from [::ffff:127.0.0.1]:35706) (comment: 03541_table_without_insertable_columns.sql) (query 4, line 9) ATTACH TABLE no_insertable UUID '00000000-0000-0000-0000-000000000001' (a Int MATERIALIZED 1) Engine=Memory; (stage: Complete)
6399998:2025.07.14 12:50:27.611438 [ 203035 ] {32e4bc22-3a43-4971-9268-b3a640e2b51a} <Warning> InterpreterCreateQuery: ATTACH TABLE query with full table definition is not recommended: use either ATTACH TABLE no_insertable; to attach existing table or CREATE TABLE no_insertable <table definition>; to create new table or ATTACH TABLE no_insertable FROM '/path/to/data/' <table definition>; to create new table and attach data.
6399999:2025.07.14 12:50:27.611728 [ 203035 ] {32e4bc22-3a43-4971-9268-b3a640e2b51a} <Debug> DatabaseAtomic (test_9ndmpbdw): Creating directory symlink, path_to_metadata_symlink: metadata/test_9ndmpbdw, metadata_path: store/e3f/e3ff14e4-5467-451e-9e73-4fc8cd5383a1/
6400000:2025.07.14 12:50:27.611922 [ 203035 ] {32e4bc22-3a43-4971-9268-b3a640e2b51a} <Debug> TCPHandler: Processed in 0.000841054 sec.
How to reproduce
No response