Skip to content

Fix race between DROP DATABASE and cleanup thread#83733

Closed
azat wants to merge 1 commit intoClickHouse:masterfrom
azat:drop-database-cleanup-race
Closed

Fix race between DROP DATABASE and cleanup thread#83733
azat wants to merge 1 commit intoClickHouse:masterfrom
azat:drop-database-cleanup-race

Conversation

@azat
Copy link
Copy Markdown
Member

@azat azat commented Jul 14, 2025

In case of database_catalog_unused_dir_hide_timeout_sec=0 it is possible that cleanup thread will set the permissions to 0, and unlink will fail, like here 1:

2025.07.14 18:54:19.462907 [ 504837 ] {} <Information> DatabaseCatalog: Removing data directory store/000/00000000-0000-0000-0000-000000000001/ of dropped table test_l14n5q09.no_insertable (00000000-0000-0000-0000-000000000001) from disk default
2025.07.14 18:54:19.462975 [ 472719 ] {6654b89c-041d-4254-89fb-e1ad1ae031c5} <Debug> DatabaseCatalog: Waiting for table 00000000-0000-0000-0000-000000000001 to be finally dropped
2025.07.14 18:54:19.468710 [ 504837 ] {} <Error> DatabaseCatalog: Cannot drop table test_l14n5q09.no_insertable (00000000-0000-0000-0000-000000000001). Will retry later.: std::exception. Code: 1001, type: std::__1::filesystem::filesystem_error, e.what() = filesystem error: in remove_all: Permission denied ["/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ci/tmp/run_r0/store/000/00000000-0000-0000-0000-000000000001/"], Stack trace (when copying this message, always include the lines below):

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

In case of database_catalog_unused_dir_hide_timeout_sec=0 it is possible
that cleanup thread will set the permissions to 0, and unlink will fail,
like here [1]:

    2025.07.14 18:54:19.462907 [ 504837 ] {} <Information> DatabaseCatalog: Removing data directory store/000/00000000-0000-0000-0000-000000000001/ of dropped table test_l14n5q09.no_insertable (00000000-0000-0000-0000-000000000001) from disk default
    2025.07.14 18:54:19.462975 [ 472719 ] {6654b89c-041d-4254-89fb-e1ad1ae031c5} <Debug> DatabaseCatalog: Waiting for table 00000000-0000-0000-0000-000000000001 to be finally dropped
    2025.07.14 18:54:19.468710 [ 504837 ] {} <Error> DatabaseCatalog: Cannot drop table test_l14n5q09.no_insertable (00000000-0000-0000-0000-000000000001). Will retry later.: std::exception. Code: 1001, type: std::__1::filesystem::filesystem_error, e.what() = filesystem error: in remove_all: Permission denied ["/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/ci/tmp/run_r0/store/000/00000000-0000-0000-0000-000000000001/"], Stack trace (when copying this message, always include the lines below):

  [1]: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=83727&sha=f3e974985d16b8abbd8d8843729c8f43b8355d4e&name_0=PR&name_1=Fast%20test
@azat azat added the 🍃 green ci 🌿 Fixing flaky tests in CI label Jul 14, 2025
@clickhouse-gh
Copy link
Copy Markdown
Contributor

clickhouse-gh bot commented Jul 14, 2025

Workflow [PR], commit [b86dc3c]

@clickhouse-gh clickhouse-gh bot added the pr-not-for-changelog This PR should not be mentioned in the changelog label Jul 14, 2025
@evillique evillique self-assigned this Jul 14, 2025
@Algunenano
Copy link
Copy Markdown
Member

This one looks related to #83725 and #83719, right?

@azat
Copy link
Copy Markdown
Member Author

azat commented Jul 14, 2025

Yes, I will leave a comment there.

@azat
Copy link
Copy Markdown
Member Author

azat commented Jul 14, 2025

Actually I think @tavplubix is right there, and it is enough to avoid UUID collision, I don't like this try/catch anyway, closing.

@azat azat closed this Jul 14, 2025
@azat azat deleted the drop-database-cleanup-race branch July 15, 2025 07:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

🍃 green ci 🌿 Fixing flaky tests in CI pr-not-for-changelog This PR should not be mentioned in the changelog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants