Skip to content

Add runner --debug option to enable ClickHouse root folder mount for debugging#84252

Merged
serxa merged 4 commits intomasterfrom
integration-tests-debugging
Jul 25, 2025
Merged

Add runner --debug option to enable ClickHouse root folder mount for debugging#84252
serxa merged 4 commits intomasterfrom
integration-tests-debugging

Conversation

@serxa
Copy link
Copy Markdown
Member

@serxa serxa commented Jul 22, 2025 •
edited
Loading

To be able to debug clickhouse server inside integration test container it is useful to have access to the source code. This PR add an option --debug that enables this.

Here is how you run a debugger for the server under integration test

  1. Put the statically linked binary of your debugger into the ClickHouse root folder of your repo. The nnd debugger is ideal for this purpose (and for ClickHouse debugging in general).
  2. Go to the integration test test.py file and add a new line with a single breakpoint() command to debug specific point in your test (to make the server work forever and not die after the test).
  3. Run the integration test with runner script as usual but add --debug option. It will produce helper shell command in stdout and start testing:
   # =====> DEBUG MODE <=====
    # ClickHouse root folder will be read-only mounted into /debug in all containers.
    # Tip: place a `breakpoint()` somewhere in your integration test python code before `runner`.
    # Open another shell and include:
        source /path/to/ClickHouse/tests/integration/runner-env.sh
    # =====> DEBUG MODE <=====
  1. When the test hits a breakpoint, it will show the Python debugger prompt, which is useful by itself for debugging purposes. For example:
test_throttling/test.py::test_write_throttling[user_remote_throttling]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /ClickHouse/tests/integration/test_throttling/test.py(493)test_write_throttling()
-> assert_took(took, should_take)
(Pdb)
  1. Open another shell and run the command there to set up the environment. Then run:
❯ source /path/to/ClickHouse/tests/integration/runner-env.sh

USAGE:
   runner-client - Run clickhouse client inside an integration test
   runner-bash   - Open shell on a node inside an integration test
   runner-nnd    - Attach nnd debugger to a clickhouse server on a node inside an integration test
  1. You can use either Container ID or Container Name to log into the required node:
❯ runner-client
CONTAINER ID   IMAGE                                      COMMAND                  CREATED          STATUS          PORTS                                                            NAMES
867c67cc9957   clickhouse/integration-test:latest         "bash -c 'trap 'pkil…"   2 seconds ago    Up 1 second                                                                      rootteststoragedelta-node1-1
64b8744cb71a   clickhouse/integration-test:latest         "bash -c 'trap 'pkil…"   2 seconds ago    Up 1 second                                                                      rootteststoragedelta-node2-1
414187056a06   clickhouse/clickhouse-server:25.3.3.42     "bash -c 'trap 'pkil…"   2 seconds ago    Up 1 second     8123/tcp, 9000/tcp, 9009/tcp                                     rootteststoragedelta-node_old-1
f5c9aef75a04   clickhouse/integration-test:latest         "clickhouse server -…"   2 seconds ago    Up 1 second                                                                      rootteststoragedelta-node_with_environment_credentials-1
c7eae7f0c29d   mcr.microsoft.com/azure-storage/azurite    "docker-entrypoint.s…"   21 seconds ago   Up 20 seconds   10000-10002/tcp, 0.0.0.0:30000->30000/tcp, :::30000->30000/tcp   rootteststoragedelta-azurite1-1
8a81755f9358   minio/minio:RELEASE.2024-07-31T05-46-26Z   "/usr/bin/docker-ent…"   22 seconds ago   Up 21 seconds   9000-9001/tcp                                                    rootteststoragedelta-minio1-1
6f11a018f16c   clickhouse/python-bottle:latest            "python3"                22 seconds ago   Up 21 seconds   8080/tcp                                                         rootteststoragedelta-resolver-1
bf180f6b2f7d   clickhouse/s3-proxy                        "/docker-entrypoint.…"   22 seconds ago   Up 22 seconds   80/tcp, 443/tcp, 8080/tcp                                        rootteststoragedelta-proxy1-1
0c03e8d780e5   clickhouse/s3-proxy                        "/docker-entrypoint.…"   22 seconds ago   Up 22 seconds   80/tcp, 443/tcp, 8080/tcp                                        rootteststoragedelta-proxy2-1
1ef790b3251e   clickhouse/integration-test:latest         "clickhouse keeper -…"   28 seconds ago   Up 27 seconds                                                                    rootteststoragedelta-zoo2-1
a65b504a1e60   clickhouse/integration-test:latest         "clickhouse keeper -…"   28 seconds ago   Up 27 seconds                                                                    rootteststoragedelta-zoo3-1
51d20a4f40c4   clickhouse/integration-test:latest         "clickhouse keeper -…"   28 seconds ago   Up 28 seconds                                                                    rootteststoragedelta-zoo1-1
Enter ClickHouse Node CONTAINER ID or NAME (default: 867c67cc9957):
ClickHouse client version 25.7.1.1.
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 25.7.1.
node1 :) select 1

SELECT 1

Query id: 7222639b-6d33-4237-85f4-ad4ee04e8691

   ┌─1─┐
1. │ 1 │
   └───┘

1 row in set. Elapsed: 0.001 sec.

node1 :) Bye.
  1. Use runner-nnd to start debugging:
Screenshot 2025-07-24 at 22 25 35

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

@clickhouse-gh
Copy link
Copy Markdown
Contributor

clickhouse-gh bot commented Jul 22, 2025 •
edited
Loading

Workflow [PR], commit [38f250c]

Summary: ✅

@clickhouse-gh clickhouse-gh bot added the pr-not-for-changelog This PR should not be mentioned in the changelog label Jul 22, 2025
@serxa serxa mentioned this pull request Jul 23, 2025
Merged
@serxa
Copy link
Copy Markdown
Member Author

serxa commented Jul 24, 2025 •
edited
Loading

The failed tests look a bit suspicious. Complaining about a read-only mount. I've added one ro mount, but it should be overridden by another rw mount, and it is only under --debug

E               Exception: Command [docker exec rootteststoragedelta-gw0-node1-1 bash -c mkdir -p $(dirname /ClickHouse/tests/integration/test_storage_delta/_instances-0-gw0/node1/database/user_files/test_checkpointlHzqpHKQqs/part-00000-6b703aa8-2103-4f68-b475-7d97e6b42f5c-c000.parquet) && echo UEFSMRUAFRwVHBXEu8....PgDAABQQVIx | base64 --decode > /ClickHouse/tests/integration/test_storage_delta/_instances-0-gw0/node1/database/user_files/test_checkpointlHzqpHKQqs/part-00000-6b703aa8-2103-4f68-b475-7d97e6b42f5c-c000.parquet] return non-zero code 1: bash: line 1: /ClickHouse/tests/integration/test_storage_delta/_instances-0-gw0/node1/database/user_files/test_checkpointlHzqpHKQqs/part-00000-6b703aa8-2103-4f68-b475-7d97e6b42f5c-c000.parquet: Read-only file system

UPD. Fixed. Now I mount it to /debug instead of /ClickHouse to avoid nested mounts

@serxa serxa changed the title Add runner --debug option to enable /ClickHouse root mount Add runner --debug option to enable ClickHouse root folder mount for debugging Jul 24, 2025
Felixoid
Felixoid reviewed Jul 25, 2025
View reviewed changes
Felixoid
Felixoid approved these changes Jul 25, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@Felixoid Felixoid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Was questions regarding mount ClickHouse root to debug,

@serxa @Felixoid
Update README.md
38f250c 
Co-authored-by: Mikhail f. Shiryaev <felixoid@clickhouse.com>
@serxa serxa enabled auto-merge July 25, 2025 10:02
@serxa serxa added this pull request to the merge queue Jul 25, 2025
Merged via the queue into master with commit 13f9d33 Jul 25, 2025
125 checks passed
@serxa serxa deleted the integration-tests-debugging branch July 25, 2025 13:26
@robot-ch-test-poll3 robot-ch-test-poll3 added the pr-synced-to-cloud The PR is synced to the cloud repo label Jul 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Felixoid Felixoid Felixoid approved these changes

Assignees

No one assigned

Labels

pr-not-for-changelog This PR should not be mentioned in the changelog pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@serxa @Felixoid @robot-ch-test-poll3