Keep track of the number of async tables loading jobs. If there are some running jobs, do not update tail_ptr in TransactionLog::removeOldEntries#82824
Merged
tuanpach merged 1 commit intoClickHouse:masterfrom Jul 9, 2025
Conversation
Contributor
14123dd to
e441e0e
Compare
tavplubix
approved these changes
Jul 2, 2025
src/Interpreters/TransactionLog.cpp
Outdated
Comment on lines
+653
to
+663
| void TransactionLog::increaseAsyncTablesLoadingJobNumber() | ||
| { | ||
| async_tables_loading_job_number.fetch_add(1, std::memory_order_relaxed); | ||
| } | ||
| void TransactionLog::decreaseAsyncTablesLoadingJobNumber() | ||
| { | ||
| async_tables_loading_job_number.fetch_sub(1, std::memory_order_relaxed); | ||
| } | ||
| Int64 TransactionLog::asyncTablesLoadingJobNumber() | ||
| { | ||
| return async_tables_loading_job_number.load(std::memory_order_relaxed); |
Member
Author
There was a problem hiding this comment.
I removed memory_order_relaxed.
I thought it might help to optimize the implementation but it seems to have some problems.
In DatabaseOrdinary::loadTableFromMetadataAsync:
LoadTaskPtr DatabaseOrdinary::loadTableFromMetadataAsync(
AsyncLoader & async_loader,
LoadJobSet load_after,
ContextMutablePtr local_context,
const String & file_path,
const QualifiedTableName & name,
const ASTPtr & ast,
LoadingStrictnessLevel mode)
{
TransactionLog::increaseAsyncTablesLoadingJobNumber();
std::scoped_lock lock(mutex);
auto job = makeLoadJob(
std::move(load_after),
TablesLoaderBackgroundLoadPoolId,
fmt::format("load table {}", name.getFullName()),
[this, local_context, file_path, name, ast, mode](AsyncLoader &, const LoadJobPtr &)
{
SCOPE_EXIT(TransactionLog::decreaseAsyncTablesLoadingJobNumber(););
loadTableFromMetadata(local_context, file_path, name, ast, mode);
});
Even though TransactionLog::increaseAsyncTablesLoadingJobNumber() is called before std::scoped_lock lock(mutex);, it might be called after makeLoadJob.
ad2af29 to
19002cc
Compare
…ome running jobs, do not update `tail_ptr` in `TransactionLog::removeOldEntries`
Merged
via the queue into
ClickHouse:master
with commit Jul 9, 2025
a6587e2
118 of 122 checks passed
Member
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Keep track of the number of async tables loading jobs. If there are some running jobs, do not update
tail_ptrinTransactionLog::removeOldEntriesBecause
loadTableFromMetadataAsyncis running asynchronously, it is possible that the outdated parts are loading while thetail_ptris updated here. It might trigger assertion when the partcreate_csnis lower thantail_ptr. Refer: #60406We keep track of
asyncTablesLoadingJobNumber, and not updatetail_ptrif there are running jobs.Closes #60406
Closes #82718
Documentation entry for user-facing changes