Skip to content

Run upstream ClickHouse SQL test suite against @clickhouse/client#707

Closed
Copilot wants to merge 3 commits into
mainfrom
copilot/add-shallow-clone-clickhouse-tests
Closed

Run upstream ClickHouse SQL test suite against @clickhouse/client#707
Copilot wants to merge 3 commits into
mainfrom
copilot/add-shallow-clone-clickhouse-tests

Conversation

Copilot AI commented May 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #706. The wrapper CLI was merged but orchestration was still manual ("download the test files, splice bin/ into PATH, run by hand"). This PR makes that flow reproducible locally and exercised in CI, without forcing a multi-GB clone of ClickHouse/ClickHouse.

  • tests/clickhouse-test-runner/scripts/run-upstream-tests.sh — driver that builds the wrapper if needed, reads upstream-allowlist.txt, exports PATH/log path/impl, then execs python3 tests/clickhouse-test "${tests[@]}" "$@" from the upstream checkout. Knobs: UPSTREAM_CLICKHOUSE_DIR, UPSTREAM_TEST_LIST, CLICKHOUSE_CLIENT_CLI_IMPL, CLICKHOUSE_CLIENT_CLI_LOG. Extra args are forwarded.
  • tests/clickhouse-test-runner/upstream-allowlist.txt — curated test list, seeded with 01428_hash_set_nan_key. One name per line, # for comments, whitespace trimmed.
  • .github/workflows/upstream-sql-tests.yml — standalone workflow (not in the success aggregator). Uses a second actions/checkout with repository: ClickHouse/ClickHouse, fetch-depth: 1, and sparse-checkout: limited to tests/clickhouse-test, tests/queries, tests/config, tests/ci, tests/performance, docker/test/util — no full clone. Matrix impl: [client, http] × clickhouse: [head, latest]. Triggers: workflow_dispatch (with upstream_ref/clickhouse_version inputs), nightly cron, and push/PR scoped to tests/clickhouse-test-runner/**. Run log + *.stdout/*.stderr/*.diff always uploaded as artifacts.
  • Docs — README replaces the manual snippet with sparse-clone + helper usage, documents the new env vars, and adds "Extending the allowlist" / "CI" subsections. CONTRIBUTING.md and AGENTS.md get pointer entries. .gitignore excludes .upstream/.

Local usage:

git clone --depth 1 --filter=blob:none --sparse https://github.com/ClickHouse/ClickHouse.git
(cd ClickHouse && git sparse-checkout set tests/clickhouse-test tests/queries tests/config tests/ci)

UPSTREAM_CLICKHOUSE_DIR=$PWD/ClickHouse \
  tests/clickhouse-test-runner/scripts/run-upstream-tests.sh --print-time

The allowlist is intentionally tiny on day one — it grows as we triage which upstream tests pass cleanly through client and http backends. The nightly run against master is what surfaces upstream drift (new tests/clickhouse-test flags, extract-from-config keys, or settings classification changes that the wrapper needs to keep up with).

Checklist

  • Unit and integration tests covering the common scenarios were added

Copilot AI and others added 3 commits May 8, 2026 13:48
Agent-Logs-Url: https://github.com/ClickHouse/clickhouse-js/sessions/082ff17d-204a-4d87-8da8-e9a080a4a54b

Co-authored-by: peter-leonov-ch <209667683+peter-leonov-ch@users.noreply.github.com>
Agent-Logs-Url: https://github.com/ClickHouse/clickhouse-js/sessions/082ff17d-204a-4d87-8da8-e9a080a4a54b

Co-authored-by: peter-leonov-ch <209667683+peter-leonov-ch@users.noreply.github.com>
Agent-Logs-Url: https://github.com/ClickHouse/clickhouse-js/sessions/082ff17d-204a-4d87-8da8-e9a080a4a54b

Co-authored-by: peter-leonov-ch <209667683+peter-leonov-ch@users.noreply.github.com>
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Comment thread .github/workflows/upstream-sql-tests.yml Dismissed
Comment thread .github/workflows/upstream-sql-tests.yml Dismissed
Comment thread tests/clickhouse-test-runner/scripts/run-upstream-tests.sh Dismissed
@codecov

codecov Bot commented May 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@peter-leonov-ch

Copy link
Copy Markdown
Collaborator

@peter-leonov-ch peter-leonov-ch deleted the copilot/add-shallow-clone-clickhouse-tests branch May 8, 2026 18:51
peter-leonov-ch added a commit that referenced this pull request May 9, 2026
…#708)

Fix CI errors from #707.

- [x] Remove `--print-time` from workflow + README example (it isn't a
valid `tests/clickhouse-test` flag)
- [x] Remove unused `clickhouse_version` `workflow_dispatch` input from
`upstream-sql-tests.yml`
- [x] Remove unused `UPSTREAM_REF` env var from `upstream-sql-tests.yml`
- [x] Install `jinja2` in `upstream-sql-tests.yml`
- [x] Fix `MaxListenersExceededWarning` in `client` backend by replacing
`pipeline(result.stream, process.stdout, { end: false })` with `for
await (const chunk of result.stream)` so no error/close listeners
accumulate on stdout
- [x] Keep `--trace-warnings` enabled in
`tests/clickhouse-test-runner/bin/clickhouse` so any future Node
warnings include their stack traces in CI logs

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: peter-leonov-ch <209667683+peter-leonov-ch@users.noreply.github.com>
Co-authored-by: Peter Leonov <peter.leonov@clickhouse.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants