Postgres wire protocol: add copy command by scanhex12 · Pull Request #74344 · ClickHouse/ClickHouse

scanhex12 · 2025-01-08T23:45:21Z

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

PostgreSQL protocol COPY command support

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

CI Settings (Only check the boxes if you know what you are doing)

All builds in Builds_1 and Builds_2 stages are always mandatory
and will run independently of the checks below:

Allow: All Required Checks
Allow: Stateless tests
Allow: Stateful tests
Allow: Integration Tests
Allow: Performance tests
Allow: All Builds
Allow: batch 1, 2 for multi-batch jobs
Allow: batch 3, 4, 5, 6 for multi-batch jobs

Run only fuzzers related jobs (libFuzzer fuzzers, AST fuzzers, BuzzHouse, etc.)
Exclude: AST fuzzers

robot-ch-test-poll1 · 2025-01-09T19:35:30Z

This is an automated comment for commit f701035 with description of existing statuses. It's updated for the latest CI running

❌ Click here to open a full report in a separate page

Check name	Description	Status
Builds	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	⏳ pending
Flaky tests	Checks if new added or modified tests are flaky by running them repeatedly, in parallel, with more randomization. Functional tests are run 100 times with address sanitizer, and additional randomization of thread scheduling. Integration tests are run up to 10 times. If at least once a new test has failed, or was too long, this check will be red. We don't allow flaky tests, read the doc	❌ failure
Integration tests	The integration tests report. In parenthesis the package type is given, and in square brackets are the optional part/total tests	❌ failure

Successful checks

Check name	Description	Status
Docs check	Builds and tests the documentation	✅ success
Fast test	Normally this is the first check that is ran for a PR. It builds ClickHouse and runs most of stateless functional tests, omitting some. If it fails, further checks are not started until it is fixed. Look at the report to see which tests fail, then reproduce the failure locally as described here	✅ success
Install packages	Checks that the built packages are installable in a clear environment	✅ success
Stateless tests	Runs stateless functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	✅ success
Style check	Runs a set of checks to keep the code style clean. If some of tests failed, see the related log from the report	✅ success
Unit tests	Runs the unit tests for different release types	✅ success

scanhex12 · 2025-01-13T18:38:07Z

@kssenii can you please review this MR? failed tests are not relevant to my changes

src/Parsers/ASTCopyQuery.h

tests/integration/test_postgresql_protocol/test.py

rschu1ze · 2025-02-09T17:47:45Z

@scanhex12 Thanks for the PR. Please help the reviewers a bit: What does this PR implement, how does it implement it (in particular is it modeled after some spec document or information), what are implications (e.g. in terms of backward compatibility, documentation, required tests, etc.)? Please be as verbose as possible. Sorry for asking, there are zero comments in the code, no information in the PR message and the Git commit history is list of "fix" commits.

clickhouse-gh · 2025-02-09T22:41:04Z

Workflow [PR], commit [4b42c7d]

pamarcos · 2025-02-17T10:42:06Z

@scanhex12 thanks for your contribution. After all the effort with the source code, please don't forget to address what Robert asked at #74344 (comment) to help us conduct the review.

Seems there are a couple of conflicts you'll need to resolve as well.

scanhex12 · 2025-03-03T05:56:56Z

@pamarcos Sorry for the delay in the review, I rebased the branch and wrote a couple of comments in the code, I hope it’s clearer. If not, let me know, I'll try to describe it in more detail.

pamarcos · 2025-03-03T07:48:58Z

@pamarcos Sorry for the delay in the review, I rebased the branch and wrote a couple of comments in the code, I hope it’s clearer. If not, let me know, I'll try to describe it in more detail.

No worries. I think Robert's ask and my own was that we're missing a proper description on the PR itself, not comments on the code. Please check out Robert's comment 🙏

clickhouse-gh · 2025-04-08T13:16:10Z

Dear @pamarcos, this PR hasn't been updated for a while. You will be unassigned. Will you continue working on it? If so, please feel free to reassign yourself.

src/Server/PostgreSQLHandler.h

src/Server/PostgreSQLHandler.cpp

src/Parsers/ParserCopyQuery.cpp

clickhouse-gh · 2025-06-16T21:41:33Z

Workflow [PR], commit [9cf66c5]

Summary: ⏳

job_name	test_name	status	info	comment
Stateless tests (amd_binary, old analyzer, s3 storage, DatabaseReplicated, parallel)		failure
	02435_rollback_cancelled_queries	FAIL

src/Parsers/ParserCopyQuery.cpp

src/Server/PostgreSQLHandler.cpp

GrigoryPervakov

Overall LGTM
I've also noticed in the documentation that during the COPY FROM query, a client is allowed to send a batch in the middle of the row. It can break in such a case.
But for now, we can just leave a notice about that and fix it later if needed

GrigoryPervakov · 2025-08-12T11:47:36Z

src/Server/PostgreSQLHandler.cpp

+        String columns_to_insert;
+        if (!copy_query->column_names.empty())
+        {
+            for (const auto & column_name : copy_query->column_names)


nit: could use boost::algorithm::join(copy_query->column_names, ", ");

GrigoryPervakov · 2025-08-12T15:25:42Z

src/Server/PostgreSQLHandler.cpp

            format_ptr->flush();
            output_buffer.finalize();
-            message_transport->send(PostgreSQLProtocol::Messaging::CopyOutData(std::move(result_buf)));
+            message_transport->send(PostgreSQLProtocol::Messaging::CopyOutData(result_buf));


Maybe it's better to move it and then explicitly initialize it after.
Extra empty vector allocation looks better than a full buffer copy before send

rschu1ze · 2025-08-12T16:12:18Z

@scanhex12 Kindly add a proper changelog message to the PR, thanks.

scanhex12 marked this pull request as draft January 8, 2025 23:45

alexey-milovidov added the can be tested Allows running workflows for external contributors label Jan 9, 2025

kssenii self-assigned this Jan 9, 2025

scanhex12 force-pushed the copy_command branch from 7891749 to 2b8d161 Compare January 9, 2025 19:20

scanhex12 marked this pull request as ready for review January 9, 2025 19:20

scanhex12 changed the title ~~Draft: Postgres wire protocol : add copy command~~ Postgres wire protocol : add copy command Jan 9, 2025

robot-ch-test-poll1 added the pr-feature Pull request with new product feature label Jan 9, 2025

robot-ch-test-poll added pr-improvement Pull request with some product improvements and removed pr-feature Pull request with new product feature labels Jan 9, 2025

rschu1ze reviewed Jan 16, 2025

View reviewed changes

src/Parsers/ASTCopyQuery.h Outdated Show resolved Hide resolved

tests/integration/test_postgresql_protocol/test.py Outdated Show resolved Hide resolved

kssenii removed their assignment Feb 4, 2025

scanhex12 requested a review from rschu1ze February 8, 2025 20:47

nikitamikhaylov assigned pamarcos Feb 10, 2025

pamarcos self-requested a review February 10, 2025 14:42

clickhouse-ci bot added the manual approve Manual approve required to run CI label Mar 2, 2025

scanhex12 force-pushed the copy_command branch from 48fc71f to 2a064ae Compare March 2, 2025 22:18

clickhouse-gh bot unassigned pamarcos Apr 8, 2025

alexey-milovidov changed the title ~~Postgres wire protocol : add copy command~~ Postgres wire protocol: add copy command Jun 1, 2025

GrigoryPervakov self-assigned this Jun 2, 2025

GrigoryPervakov requested changes Jun 2, 2025

View reviewed changes

scanhex12 added 2 commits June 16, 2025 12:38

init

7b18b7a

fix tests

d048032

fix

4959424

scanhex12 force-pushed the copy_command branch from 4b42c7d to 4959424 Compare June 16, 2025 21:41

scanhex12 requested a review from GrigoryPervakov June 16, 2025 21:44

scanhex12 added 3 commits June 16, 2025 21:53

style issues

f0a81b4

fix tests

f9d0140

fix

ae8ce46

pamarcos removed their request for review June 17, 2025 14:44

GrigoryPervakov reviewed Jul 14, 2025

View reviewed changes

src/Parsers/ParserCopyQuery.cpp Outdated Show resolved Hide resolved

GrigoryPervakov reviewed Jul 14, 2025

View reviewed changes

src/Parsers/ParserCopyQuery.cpp Show resolved Hide resolved

scanhex12 and others added 3 commits July 23, 2025 17:01

Merge branch 'master' into copy_command

0229e23

review fixes

1f61ea9

fix style

d04a1a4

scanhex12 requested a review from GrigoryPervakov July 28, 2025 22:06

robot-clickhouse-ci-1 added the pr-synced-to-cloud The PR is synced to the cloud repo label Jul 28, 2025

GrigoryPervakov reviewed Aug 1, 2025

View reviewed changes

scanhex12 added 2 commits August 12, 2025 11:27

fix

b0e94a6

maybe fix build

1929d47

GrigoryPervakov approved these changes Aug 12, 2025

View reviewed changes

scanhex12 enabled auto-merge August 12, 2025 12:08

fix build

9cf66c5

GrigoryPervakov reviewed Aug 12, 2025

View reviewed changes

scanhex12 added this pull request to the merge queue Aug 12, 2025

Merged via the queue into ClickHouse:master with commit 04304c3 Aug 12, 2025
113 of 122 checks passed

scanhex12 deleted the copy_command branch August 12, 2025 16:00

scanhex12 added pr-feature Pull request with new product feature and removed pr-improvement Pull request with some product improvements labels Aug 13, 2025

rschu1ze mentioned this pull request Aug 31, 2025

Intern Tasks 2024/2025 #71175

Closed

Conversation

scanhex12 commented Jan 8, 2025 • edited by fm4v Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

CI Settings (Only check the boxes if you know what you are doing)

Uh oh!

robot-ch-test-poll1 commented Jan 9, 2025 • edited by robot-clickhouse-ci-2 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

scanhex12 commented Jan 13, 2025

Uh oh!

Uh oh!

Uh oh!

rschu1ze commented Feb 9, 2025

Uh oh!

clickhouse-gh bot commented Feb 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pamarcos commented Feb 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

scanhex12 commented Mar 3, 2025

Uh oh!

pamarcos commented Mar 3, 2025

Uh oh!

clickhouse-gh bot commented Apr 8, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

clickhouse-gh bot commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GrigoryPervakov left a comment

Choose a reason for hiding this comment

Uh oh!

GrigoryPervakov Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

GrigoryPervakov Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rschu1ze commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

scanhex12 commented Jan 8, 2025 •

edited by fm4v

Loading

robot-ch-test-poll1 commented Jan 9, 2025 •

edited by robot-clickhouse-ci-2

Loading

clickhouse-gh bot commented Feb 9, 2025 •

edited

Loading

pamarcos commented Feb 17, 2025 •

edited

Loading

clickhouse-gh bot commented Jun 16, 2025 •

edited

Loading

rschu1ze commented Aug 12, 2025 •

edited

Loading