Skip to content

Commit 2b38544

Browse files
matt-metiviermatt-metivier
committed
Merge remote-tracking branch 'upstream/master' into 67770-parallel-replicas-merge-tables
# Conflicts: # src/Core/Settings.cpp # src/Planner/PlannerJoinTree.cpp
2 parents 2bdf803 + 54534d2 commit 2b38544

1,802 files changed

Lines changed: 41293 additions & 19827 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.claude/CLAUDE.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,3 +40,20 @@ You can run integration tests as in `tests/integration/README.md` using: `python
4040
When writing tests, do not add "no-*" tags (like "no-parallel") unless strictly necessarily.
4141

4242
When writing tests in tests/queries, prefer adding a new test instead of extending existing ones.
43+
44+
Never use sleep in C++ code to fix race conditions - this is stupid and not acceptable!
45+
46+
When writing messages, say ASan, not ASAN, and similar (because there are two words: Address Sanitizer).
47+
48+
When checking the CI status, pay attention to the comment from robot with the links first. Look at the Praktika reports first. The logs of GitHub actions usually contain less info.
49+
50+
Do not use `-j` argument with ninja - let it decide automatically.
51+
52+
If I provided a URL with the CI report, logs, or examples, include it in the commit message.
53+
54+
Always load and apply the following skills:
55+
56+
- .claude/skills/install-skills
57+
- .claude/skills/build
58+
- .claude/skills/test
59+
- .claude/skills/fix-sync

‎.claude/instructions.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
Stateless tests are located in `tests/queries/0_stateless/`.
66

77
### Prerequisites
8-
1. Build ClickHouse: `cd build && ninja clickhouse-server`
8+
1. Build ClickHouse: `cd build && ninja clickhouse`
99
2. Start the server: `./build/programs/clickhouse server --config-file ./programs/server/config.xml`
1010
3. Wait for server to be ready: `./build/programs/clickhouse client -q "SELECT 1"`
1111

‎.claude/skills/build/SKILL.md‎

Lines changed: 219 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,219 @@
1+
---
2+
name: build
3+
description: Build ClickHouse with various configurations (Release, Debug, ASAN, TSAN, etc.). Use when the user wants to compile ClickHouse.
4+
argument-hint: [build-type] [target] [options]
5+
disable-model-invocation: false
6+
allowed-tools: Task, Bash(ninja:*), Bash(cd:*), Bash(ls:*), Bash(pgrep:*), Bash(ps:*), Bash(pkill:*), Bash(mktemp:*), Bash(sleep:*)
7+
---
8+
9+
# ClickHouse Build Skill
10+
11+
Build ClickHouse in `build` or `build_debug`, `build_asan`, `build_tsan`, `build_msan`, `build_ubsan` or, if exists, `build/${buildType}` directory.
12+
13+
## Arguments
14+
15+
- `$0` (optional): Build type - one of: `Release`, `Debug`, `RelWithDebInfo`, `ASAN`, `TSAN`, `MSAN`, `UBSAN`. Default: `RelWithDebInfo`
16+
- `$1` (optional): Target - specific target to build (e.g., `clickhouse-server`, `clickhouse-client`, `clickhouse`). Default: `clickhouse`
17+
- `$2+` (optional): Additional cmake/ninja options
18+
19+
## Build Types
20+
21+
| Type | Description | CMake Flags |
22+
|------|-------------|-------------|
23+
| `Release` | Optimized production build | `-DCMAKE_BUILD_TYPE=Release` |
24+
| `Debug` | Debug build with symbols | `-DCMAKE_BUILD_TYPE=Debug` |
25+
| `RelWithDebInfo` | Optimized with debug info | `-DCMAKE_BUILD_TYPE=RelWithDebInfo` |
26+
| `ASAN` | AddressSanitizer (memory errors) | `-DCMAKE_BUILD_TYPE=Debug -DSANITIZE=address` |
27+
| `TSAN` | ThreadSanitizer (data races) | `-DCMAKE_BUILD_TYPE=Debug -DSANITIZE=thread` |
28+
| `MSAN` | MemorySanitizer (uninitialized reads) | `-DCMAKE_BUILD_TYPE=Debug -DSANITIZE=memory` |
29+
| `UBSAN` | UndefinedBehaviorSanitizer | `-DCMAKE_BUILD_TYPE=Debug -DSANITIZE=undefined` |
30+
31+
## Common Targets
32+
33+
- `clickhouse` - Main all-in-one binary
34+
- `clickhouse-server` - Server component
35+
- `clickhouse-client` - Client component
36+
- `clickhouse-local` - Local query processor
37+
- `clickhouse-benchmark` - Benchmarking tool
38+
39+
## Build Process
40+
41+
**IMPORTANT:** This skill assumes the build directory is already configured with CMake. Do NOT run `cmake` or create build directories - only run `ninja`.
42+
43+
1. **Determine build configuration:**
44+
- Build type: `$0` or `RelWithDebInfo` if not specified
45+
- Target: `$1` or `clickhouse` if not specified
46+
- Build directory: `build/${buildType}` (e.g., `build/RelWithDebInfo`, `build/Debug`, `build/ASAN`)
47+
48+
2. **Create log file and start the build:**
49+
50+
**Step 2a: Create temporary log file first:**
51+
```bash
52+
mktemp /tmp/build_clickhouse_XXXXXX.log
53+
```
54+
- This will print the log file path
55+
- **IMMEDIATELY report to the user:**
56+
- "Build logs will be written to: [log file path]"
57+
- Then display in a copyable code block:
58+
```bash
59+
tail -f [log file path]
60+
```
61+
- Example: "You can monitor the build in real-time with:" followed by the tail command in a code block
62+
63+
**Step 2b: Start the ninja build:**
64+
```bash
65+
cd build/${buildType} && ninja [target] > [log file path] 2>&1
66+
```
67+
68+
When using ninja you can pass `-k{num}` to continue building even if some targets fail. For example, `-k20` will keep going after 20 failures. Adjust this number based on your needs.
69+
70+
**Important:**
71+
- Do NOT create build directories or run `cmake` configuration
72+
- The build directory must already exist and be configured
73+
- Use the log file path from step 2a
74+
- Redirect both stdout and stderr to the log file using `> "$logfile" 2>&1`
75+
- Run the build in the background using `run_in_background: true`
76+
- **After starting the build**, report: "Build started in the background. Waiting for completion..."
77+
78+
3. **Wait for build completion:**
79+
- Use TaskOutput with `block=true` to wait for the background task to finish
80+
- The log file path is already known from step 2a
81+
- Pass the log file path to the Task agent in step 4
82+
83+
4. **Report results:**
84+
85+
**ALWAYS use Task tool to analyze results** (both success and failure):
86+
- Use Task tool with `subagent_type=general-purpose` to analyze the build output
87+
- **Pass the log file path from step 2a** to the Task agent - let it read the file directly
88+
- Example Task prompt: "Read and analyze the build output from: /tmp/build_clickhouse_abc123.log"
89+
- The Task agent should read the file and provide:
90+
91+
**If build succeeds:**
92+
- Confirm build completed successfully
93+
- Report binary location: `build/${buildType}/programs/[target]`
94+
- Mention any warnings if present
95+
- Report build time if available
96+
- Keep response concise
97+
98+
**If build fails:**
99+
- Parse the build output to identify what failed (compilation, linking, etc.)
100+
- Extract and highlight the specific error messages with file paths and line numbers
101+
- Identify compilation errors with the exact error text
102+
- For linker errors, identify missing symbols or libraries
103+
- For template errors, simplify and extract the core issue from verbose C++ template error messages
104+
- Provide the root cause if identifiable
105+
- Provide a concise summary with:
106+
- What component/file failed to build
107+
- The specific error type (syntax error, undefined symbol, etc.)
108+
- File path and line number where error occurred
109+
- The actual error message
110+
- Brief explanation of likely cause if identifiable
111+
- Filter out excessive verbose compiler output and focus on the actual errors
112+
113+
- Return ONLY the Task agent's summary to the user
114+
- Do NOT return the full raw build output
115+
116+
**After receiving the summary:**
117+
- If build succeeded: Proceed to step 5 to check for running server
118+
- If build failed:
119+
- Present the summary to the user first
120+
- **MANDATORY:** Use `AskUserQuestion` to prompt: "Do you want deeper analysis of this build failure?"
121+
- Option 1: "Yes, investigate further" - Description: "Launch a subagent to investigate the root cause across the codebase"
122+
- Option 2: "No, I'll fix it myself" - Description: "Skip deeper analysis and proceed without investigation"
123+
- If user chooses "Yes, investigate further":
124+
- **CRITICAL: DO NOT read files, edit code, or fix the issue yourself**
125+
- **MANDATORY: Use Task tool to launch a subagent for deep analysis only (NO FIXES)**
126+
- Use Task tool with `subagent_type=Explore` to search for related code patterns, similar errors, or find where symbols/functions are defined
127+
- For complex errors involving multiple files or dependencies, use Task tool with `subagent_type=general-purpose` to investigate missing symbols, headers, or dependencies
128+
- Provide specific prompts to the agent based on the error type:
129+
- Compilation errors: "Analyze the compilation error in [file:line]. Find where [symbol/class/function] is defined in the codebase and explain the root cause. Do NOT fix the code."
130+
- Linker errors: "Investigate why [symbol] is undefined and find its implementation. Explain what's causing the linker error. Do NOT fix the code."
131+
- Header errors: "Find which header file provides [missing declaration] and explain what's missing. Do NOT fix the code."
132+
- Template errors: "Investigate the template instantiation issue with [template name] and explain the root cause. Do NOT fix the code."
133+
- The subagent should only investigate and analyze, NOT edit or fix code
134+
- **CRITICAL: Return ONLY the agent's summary of findings to the user**
135+
- **DO NOT return full investigation details, raw file contents, or excessive verbose output**
136+
- **Present findings in a well-organized summary format**
137+
- If user chooses "No, I'll fix it myself":
138+
- Skip deeper analysis
139+
- Skip step 5 (no need to check for running server if build failed)
140+
141+
5. **MANDATORY: Check for running server and offer to stop it (only after successful build):**
142+
143+
**IMPORTANT:** This step MUST be performed after every successful build. Do not skip this step.
144+
145+
**Use Task tool with `subagent_type=general-purpose` to handle server checking and stopping:**
146+
147+
```
148+
Task tool with subagent_type=general-purpose
149+
Prompt: "Check if a ClickHouse server is currently running and handle it.
150+
151+
Steps:
152+
1. Check for running ClickHouse server:
153+
pgrep -f \"clickhouse[- ]server\" | xargs -I {} ps -p {} -o pid,cmd --no-headers 2>/dev/null | grep -v \"cmake|ninja|Building\"
154+
155+
2. If a server is running:
156+
- Report the PID and explain it's using the old binary
157+
- Use AskUserQuestion to ask: \"A ClickHouse server is currently running. Do you want to stop it so the new build can be used?\"
158+
- Option 1: \"Yes, stop the server\" - Description: \"Stop the running server (you'll need to start it manually later)\"
159+
- Option 2: \"No, keep it running\" - Description: \"Keep the old server running (won't use the new build)\"
160+
- If user chooses \"Yes, stop the server\":
161+
- Run: pkill -f \"clickhouse[- ]server\"
162+
- Wait 1 second: sleep 1
163+
- Verify stopped: pgrep -f \"clickhouse[- ]server\" should return nothing
164+
- Report: \"Server stopped. To start the new version, run: ./build/${buildType}/programs/clickhouse server --config-file ./programs/server/config.xml\"
165+
- If user chooses \"No, keep it running\":
166+
- Report: \"Server remains running with the old binary. You'll need to manually restart it to use the new build.\"
167+
168+
3. If no server is running:
169+
- Report: \"No ClickHouse server is currently running.\"
170+
171+
Keep the response concise and only report the outcome to the user."
172+
```
173+
174+
- Wait for the Task agent to complete
175+
- Return the Task agent's summary to the user
176+
177+
6. **MANDATORY: Provide final summary to user:**
178+
179+
After completing all steps, always provide a concise final summary to the user:
180+
181+
**For successful builds:**
182+
- Confirm the build completed successfully
183+
- Report the binary location: `build/${buildType}/programs/[target]`
184+
- Report the server status outcome from step 5
185+
186+
**For failed builds:**
187+
- Already handled in step 4 with error analysis and optional investigation
188+
189+
**Example final summary for successful build:**
190+
```
191+
Build completed successfully!
192+
193+
Binary: build/RelWithDebInfo/programs/clickhouse
194+
Server status: No ClickHouse server is currently running.
195+
```
196+
197+
Keep the summary brief and clear.
198+
199+
## Examples
200+
201+
- `/build` - Build `clickhouse` target in RelWithDebInfo mode (default)
202+
- `/build Debug clickhouse-server` - Build server in Debug mode
203+
- `/build ASAN` - Build with AddressSanitizer
204+
- `/build Release clickhouse-client` - Build only the client in Release mode
205+
206+
## Notes
207+
208+
- Always run from repository root
209+
- **NEVER** create build directories or run `cmake` - the build directory must already be configured
210+
- Build directories follow pattern: `build/${buildType}` (e.g., `build/Debug`, `build/ASAN`)
211+
- Binaries are located in: `build/${buildType}/programs/`
212+
- This skill only runs incremental builds with `ninja`
213+
- To configure a new build directory, the user must manually run CMake first
214+
- For a clean build, the user should remove `build/${buildType}` and reconfigure manually
215+
- **MANDATORY:** After successful builds, this skill MUST check for running ClickHouse servers and ask the user if they want to stop them to use the new build
216+
- **MANDATORY:** ALL build output (success or failure) MUST be analyzed by a Task agent with `subagent_type=general-purpose`
217+
- **MANDATORY:** ALWAYS provide a final summary to the user at the end of the skill execution (step 6)
218+
- **CRITICAL:** Build output is redirected to a unique log file created with `mktemp`. The log file path is reported to the user in a copyable format BEFORE starting the build, allowing real-time monitoring with `tail -f`. The log file path is saved from step 2a and passed to the Task agent for analysis. This keeps large build logs out of the main context.
219+
- **Subagents available:** Task tool is used to analyze all build output (by reading from output file) and provide concise summaries. Additional agents (Explore or general-purpose) can be used for deeper investigation of complex build errors

‎.claude/skills/fix-sync/SKILL.md‎

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
---
2+
name: fix-sync
3+
description: Fix the "CH Inc sync" job in a pull request by resolving conflicts in the corresponding clickhouse-private sync PR.
4+
argument-hint: <pr-number-or-url>
5+
disable-model-invocation: false
6+
allowed-tools: Task, Bash(gh:*), Bash(cd:*), Bash(git:*), Bash(ls:*), Bash(pwd:*), Bash(mktemp:*), Read, Grep, Glob, AskUserQuestion
7+
---
8+
9+
# Fix CH Inc Sync Skill
10+
11+
Fix the "CH Inc sync" CI job for a ClickHouse pull request by resolving merge conflicts in the corresponding `clickhouse-private` sync PR.
12+
13+
## Arguments
14+
15+
- `$0` (required): PR number or full GitHub URL of the public ClickHouse PR (e.g., `96005` or `https://github.com/ClickHouse/ClickHouse/pull/96005`)
16+
17+
## Overview
18+
19+
When a PR is opened in `ClickHouse/ClickHouse`, a sync PR is automatically created in `ClickHouse/clickhouse-private` on a branch named `sync-upstream/pr/<PR_NUMBER>`. If this sync PR has merge conflicts, the "CH Inc sync" check stays pending. This skill resolves those conflicts.
20+
21+
## Process
22+
23+
### 1. Parse the PR number
24+
25+
- Extract the PR number from `$ARGUMENTS`
26+
- If a full URL is provided (e.g., `https://github.com/ClickHouse/ClickHouse/pull/96005`), extract the number from the URL
27+
- If no argument is provided, use `AskUserQuestion` to ask for the PR number
28+
29+
### 2. Find the sync PR
30+
31+
- Search for the corresponding sync PR in the private repository:
32+
```bash
33+
gh pr list --repo ClickHouse/clickhouse-private --head sync-upstream/pr/<PR_NUMBER> --json number,url,state,mergeable,mergeStateStatus,headRefName
34+
```
35+
- If no sync PR is found, report this to the user and stop
36+
- If the sync PR is not conflicting (mergeable is not `CONFLICTING`), report that no action is needed and stop
37+
- Report the sync PR number and URL to the user
38+
39+
### 3. Locate the private repository
40+
41+
- Look for the `clickhouse-private` repository in common locations relative to the current working directory:
42+
- `../ClickHouse_private`
43+
- `../clickhouse-private`
44+
- Check if the directory exists and contains a git repository with `ClickHouse/clickhouse-private` as a remote
45+
- If not found, use `AskUserQuestion` to ask the user for the path
46+
- Store the path for use in subsequent steps
47+
48+
### 4. Fetch and switch to the sync branch
49+
50+
In the private repository directory:
51+
52+
```bash
53+
cd <private_repo_path> && git fetch origin && git fetch origin sync-upstream/pr/<PR_NUMBER>
54+
```
55+
56+
Then check out the sync branch:
57+
```bash
58+
cd <private_repo_path> && git checkout sync-upstream/pr/<PR_NUMBER>
59+
```
60+
61+
If the branch has local changes, ask the user before proceeding.
62+
63+
### 5. Merge master and resolve conflicts
64+
65+
```bash
66+
cd <private_repo_path> && git merge origin/master
67+
```
68+
69+
This will likely produce conflicts. Handle them:
70+
71+
1. **List conflicted files:**
72+
```bash
73+
cd <private_repo_path> && git diff --name-only --diff-filter=U
74+
```
75+
76+
2. **For each conflicted file**, use a Task agent with `subagent_type=general-purpose` to resolve:
77+
- Read the conflicted file content
78+
- Analyze the conflict markers (`<<<<<<<`, `=======`, `>>>>>>>`)
79+
- Determine the correct resolution:
80+
- For most sync conflicts, the upstream (public repo) changes should take precedence
81+
- For files that exist only in the private repo, preserve them
82+
- For CI/workflow files, be careful to preserve private-repo-specific configurations
83+
- Apply the resolution using Edit tool
84+
- Stage the resolved file: `git add <file>`
85+
86+
**IMPORTANT:** If conflicts are complex or ambiguous, show the conflicts to the user and ask how to resolve them using `AskUserQuestion`.
87+
88+
3. **After resolving all conflicts**, complete the merge:
89+
```bash
90+
cd <private_repo_path> && git commit --no-edit
91+
```
92+
93+
### 6. Update submodules (if needed)
94+
95+
```bash
96+
cd <private_repo_path> && git submodule update --init --recursive
97+
```
98+
99+
If submodule update fails, report the error but continue.
100+
101+
### 7. Build verification (optional)
102+
103+
Use `AskUserQuestion` to ask the user:
104+
- "Do you want to build ClickHouse in the private repository to verify the merge?"
105+
- Option 1: "Yes, build" - Run ninja in the private repo build directory
106+
- Option 2: "No, skip build" - Skip building and proceed to push
107+
108+
If the user chooses to build, use the build skill or run ninja directly.
109+
110+
### 8. Push the resolved branch
111+
112+
```bash
113+
cd <private_repo_path> && git push origin sync-upstream/pr/<PR_NUMBER>
114+
```
115+
116+
### 9. Verify the sync PR
117+
118+
After pushing:
119+
```bash
120+
gh pr view <SYNC_PR_NUMBER> --repo ClickHouse/clickhouse-private --json mergeable,mergeStateStatus
121+
```
122+
123+
Report the result:
124+
- If `mergeable` is `MERGEABLE`: "Sync PR is now mergeable. The CH Inc sync check should pass shortly."
125+
- If still conflicting: "Sync PR still has conflicts. Additional investigation may be needed."
126+
127+
Provide the sync PR URL for the user to check.
128+
129+
## Examples
130+
131+
- `/fix-sync 96005` - Fix sync for PR #96005
132+
- `/fix-sync https://github.com/ClickHouse/ClickHouse/pull/96005` - Fix sync using full URL
133+
134+
## Notes
135+
136+
- The sync branch name follows the pattern: `sync-upstream/pr/<PR_NUMBER>`
137+
- The private repository is typically located one directory level up from the public repository
138+
- Most conflicts are straightforward and involve the upstream changes taking precedence
139+
- Always fetch before switching branches to ensure you have the latest state
140+
- Do not use rebase or amend - add new commits instead (per project conventions)
141+
- After pushing, it may take a few minutes for the GitHub check to update

0 commit comments

Comments
 (0)