Check and mark the interserver IO address active in DDL worker by tuanpach · Pull Request #92339 · ClickHouse/ClickHouse

tuanpach · 2025-12-17T03:05:34Z

Changelog category (leave one):

Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Related issue #https://github.com/ClickHouse/support-escalation/issues/6365

Previously, when marking replica active, we don't check the interserver IO address. This address is used for cluster created by Replicated DBs.

In this PR:

It checks and marks the interserver IO address as active in DDLWorker::markReplicasActive
Notify DDLWorker when host IDs are updated when cluster config updated. This is a separate fix to let DDLWorker runs markReplicaActive again when the host IDs are updated in remoter_servers config.

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

clickhouse-gh · 2025-12-17T03:06:03Z

Workflow [PR], commit [61b2d55]

Summary: ❌

job_name	test_name	status	info
Stateless tests (arm_binary, parallel)		failure
	03221_merge_profile_events	FAIL	cidb
Integration tests (arm_binary, distributed plan, 1/4)		failure
	test_merges_memory_limit/test.py::test_memory_limit_success	FAIL	cidb
BuzzHouse (amd_debug)		failure
	Logical error: 'Inconsistent AST formatting in Function_and: the query: (STID: 1941-1bfa)	FAIL	cidb

…o mark replicas active.

tavplubix · 2025-12-22T17:02:35Z

What happens when the cluster gets updated through DatabaseReplicated::setCluster?

tuanpach · 2025-12-23T02:52:06Z

What happens when the cluster gets updated through DatabaseReplicated::setCluster?

I updated the logic, it will notify shared_ddl_worker that host IDs were updated.

tavplubix · 2026-01-20T17:19:12Z

src/Interpreters/DDLWorker.cpp

+    // Add interserver IO host IDs for Replicated DBs
+    try
+    {
+        auto host_port = context->getInterserverIOAddress();
+        HostID interserver_io_host_id = {host_port.first, port};
+        all_host_ids.emplace(interserver_io_host_id.toString());
+        LOG_INFO(log, "Add interserver IO host ID {}", interserver_io_host_id.toString());


So this is the main part that will fix the issue in the cloud

The problem was that context->getClusters() does not return Replicated DB clusters, so the list of hosts was empty in the cloud. However, we don't need to check all hosts in Replicated DB clusters because it's enough to simply use getInterserverIOAddress which is our host for sure

And for remote_servers config, we notify DDLWorker on config changes, but it's a separate fix

Yes. Originally, I thought the IP of Replicated DBs was also changeable. But it is not.

I will update the PR description and title.

tuanpach · 2026-01-22T06:30:19Z

03221_merge_profile_events

Stateless test 03221_merge_profile_events is flaky #85891

test_merges_memory_limit/test.py::test_memory_limit_success

test_merges_memory_limit/test.py::test_memory_limit_success is flaky #93441

Cherry pick #92339 to 25.11: Check and mark the interserver IO address active in DDL worker

…ctive in DDL worker

Cherry pick #92339 to 25.12: Check and mark the interserver IO address active in DDL worker

…ctive in DDL worker

Backport #92339 to 25.12: Check and mark the interserver IO address active in DDL worker

…very iteration Replica dirs in ZK are created in enqueueQueryAttempt() when the first DDL is enqueued. At worker init getChildren(replicas_dir) was still empty, so markReplicasActive() never created replicas_dir/<host_id>/active for those host_ids. The worker requires that active node (and for loopback, this node's UUID) before executing a task, so the task was skipped with "loopback not claimed" and the initiator saw timeouts (e.g. HTTP 503). Call markReplicasActive(reinitialized) on every main-loop iteration, before scheduleTasks(), so new replica dirs get their active node before we schedule tasks. Future backport ClickHouse#92339 It checks and marks the interserver IO address as active in DDLWorker::markReplicasActive. Notify DDLWorker when host IDs are updated when cluster config updated. This is a separate fix to let DDLWorker runs markReplicaActive again when the host IDs are updated in remoter_servers config.

…licas-active-on-new-host-ids Check and mark the interserver IO address active in DDL worker

25.8.16 Stable backport of ClickHouse#92339: Check and mark the interserver IO address active in DDL worker

Backport #92339 to 25.11: Check and mark the interserver IO address active in DDL worker

Cherry pick #92339 to 25.8: Check and mark the interserver IO address active in DDL worker

…tive in DDL worker

Backport #92339 to 25.8: Check and mark the interserver IO address active in DDL worker

tuanpach added the can be tested Allows running workflows for external contributors label Dec 17, 2025

clickhouse-gh bot added the pr-bugfix Pull request with bugfix, not backported by default label Dec 17, 2025

tuanpach force-pushed the ddl-worker-mark-replicas-active-on-new-host-ids branch from 208c812 to f299bf6 Compare December 17, 2025 03:08

clickhouse-gh bot added the submodule changed At least one submodule changed in this PR. label Dec 17, 2025

tuanpach force-pushed the ddl-worker-mark-replicas-active-on-new-host-ids branch 2 times, most recently from 0929de7 to 833f4b7 Compare December 17, 2025 12:34

tuanpach removed the submodule changed At least one submodule changed in this PR. label Dec 17, 2025

Notify DDLWorker when host IDs are updated. DDLWorker will schedule t…

357dfbe

…o mark replicas active.

tuanpach force-pushed the ddl-worker-mark-replicas-active-on-new-host-ids branch from 833f4b7 to 357dfbe Compare December 17, 2025 23:51

alesapin self-assigned this Dec 22, 2025

tuanpach mentioned this pull request Dec 23, 2025

In DDLWorker, mark replicas as active periodically #92023

Closed

1 task

Notify host IDs updated in DatabaseReplicated::setCluster

cdd52c7

Using interserver IO host ID for Replicated DBs

61b2d55

tavplubix approved these changes Jan 20, 2026

View reviewed changes

tuanpach changed the title ~~Notify DDLWorker when host IDs are updated to re-mark replicas active~~ Check and mark the interserver IO address active in DDL worker Jan 21, 2026

tuanpach added this pull request to the merge queue Jan 22, 2026

Merged via the queue into ClickHouse:master with commit 6f99054 Jan 22, 2026
127 of 131 checks passed

tuanpach deleted the ddl-worker-mark-replicas-active-on-new-host-ids branch January 22, 2026 06:47

robot-ch-test-poll added the pr-synced-to-cloud The PR is synced to the cloud repo label Jan 22, 2026

tuanpach added the pr-must-backport Pull request should be backported intentionally. Use this label with great care! label Jan 22, 2026

robot-ch-test-poll2 added the pr-must-backport-synced The `*-must-backport` labels are synced into the cloud Sync PR label Jan 22, 2026

robot-ch-test-poll1 added a commit that referenced this pull request Jan 22, 2026

Merge pull request #94802 from ClickHouse/cherrypick/25.11/92339

b11dc96

Cherry pick #92339 to 25.11: Check and mark the interserver IO address active in DDL worker

robot-clickhouse added a commit that referenced this pull request Jan 22, 2026

Backport #92339 to 25.11: Check and mark the interserver IO address a…

13ba08e

…ctive in DDL worker

This was referenced Jan 22, 2026

Backport #92339 to 25.11: Check and mark the interserver IO address active in DDL worker #94803

Merged

Cherry pick #92339 to 25.12: Check and mark the interserver IO address active in DDL worker #94804

Merged

robot-ch-test-poll1 added a commit that referenced this pull request Jan 22, 2026

Merge pull request #94804 from ClickHouse/cherrypick/25.12/92339

721f8ca

Cherry pick #92339 to 25.12: Check and mark the interserver IO address active in DDL worker

robot-clickhouse added a commit that referenced this pull request Jan 22, 2026

Backport #92339 to 25.12: Check and mark the interserver IO address a…

b9180f9

…ctive in DDL worker

robot-ch-test-poll1 mentioned this pull request Jan 22, 2026

Backport #92339 to 25.12: Check and mark the interserver IO address active in DDL worker #94805

Merged

clickhouse-gh bot added a commit that referenced this pull request Jan 22, 2026

Merge pull request #94805 from ClickHouse/backport/25.12/92339

66d38c3

Backport #92339 to 25.12: Check and mark the interserver IO address active in DDL worker

ianton-ru mentioned this pull request Jan 29, 2026

Fix DDLWorker initialization with delayed DNS records. #92040

Closed

1 task

robot-ch-test-poll mentioned this pull request Jan 31, 2026

Cherry pick #92339 to 24.3: Check and mark the interserver IO address active in DDL worker #95717

Closed

zvonand pushed a commit to Altinity/ClickHouse that referenced this pull request Feb 14, 2026

Merge pull request ClickHouse#92339 from tuanpach/ddl-worker-mark-rep…

91627b0

…licas-active-on-new-host-ids Check and mark the interserver IO address active in DDL worker

zvonand mentioned this pull request Feb 14, 2026

25.8.16 Stable backport of #92339: Check and mark the interserver IO address active in DDL worker Altinity/ClickHouse#1403

Merged

25 tasks

alex-zaitsev mentioned this pull request Feb 16, 2026

Race condition: deploying CHI after Keeper leaves CHI unconnected to Keeper/replication Altinity/clickhouse-operator#1913

Closed

zvonand added a commit to Altinity/ClickHouse that referenced this pull request Feb 16, 2026

Merge pull request #1403 from Altinity/backports/releases/25.8.16/92339

9e7610c

25.8.16 Stable backport of ClickHouse#92339: Check and mark the interserver IO address active in DDL worker

alex-zaitsev mentioned this pull request Feb 19, 2026

DDLWorker Fails for ON CLUSTER Statements in Multiple ClickHouse Versions (Including 25.10.2.65 and 25.11.1) Altinity/clickhouse-operator#1883

Closed

tuanpach added a commit that referenced this pull request Feb 24, 2026

Merge pull request #94803 from ClickHouse/backport/25.11/92339

49efb18

Backport #92339 to 25.11: Check and mark the interserver IO address active in DDL worker

robot-ch-test-poll2 added a commit that referenced this pull request Feb 24, 2026

Merge pull request #94800 from ClickHouse/cherrypick/25.8/92339

d2f88f8

Cherry pick #92339 to 25.8: Check and mark the interserver IO address active in DDL worker

robot-clickhouse added a commit that referenced this pull request Feb 24, 2026

Backport #92339 to 25.8: Check and mark the interserver IO address ac…

6283000

…tive in DDL worker

robot-ch-test-poll2 mentioned this pull request Feb 24, 2026

Backport #92339 to 25.8: Check and mark the interserver IO address active in DDL worker #97815

Merged

robot-ch-test-poll1 added the pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore label Feb 24, 2026

tuanpach added a commit that referenced this pull request Feb 24, 2026

Merge pull request #97815 from ClickHouse/backport/25.8/92339

8e5550e

Backport #92339 to 25.8: Check and mark the interserver IO address active in DDL worker

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Check and mark the interserver IO address active in DDL worker#92339

Check and mark the interserver IO address active in DDL worker#92339
tuanpach merged 3 commits intoClickHouse:masterfrom
tuanpach:ddl-worker-mark-replicas-active-on-new-host-ids

tuanpach commented Dec 17, 2025 •

edited

Loading

Uh oh!

clickhouse-gh bot commented Dec 17, 2025 •

edited

Loading

Uh oh!

tavplubix commented Dec 22, 2025

Uh oh!

tuanpach commented Dec 23, 2025

Uh oh!

tavplubix Jan 20, 2026

Uh oh!

tuanpach Jan 21, 2026

Uh oh!

tuanpach commented Jan 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

tuanpach commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Documentation entry for user-facing changes

Uh oh!

clickhouse-gh bot commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tavplubix commented Dec 22, 2025

Uh oh!

tuanpach commented Dec 23, 2025

Uh oh!

tavplubix Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

tuanpach Jan 21, 2026

Choose a reason for hiding this comment

Uh oh!

tuanpach commented Jan 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

tuanpach commented Dec 17, 2025 •

edited

Loading

clickhouse-gh bot commented Dec 17, 2025 •

edited

Loading