Add ProfileEvents for number of rows and bytes passing filter transform by onkar · Pull Request #76662 · ClickHouse/ClickHouse

onkar · 2025-02-24T05:45:30Z

This PR adds profile events to calculate the number of rows and number of bytes in filter transform of a query. It resolves #67288.

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

FilterTransformPassedRows and FilterTransformPassedBytes profile events will show the number of rows and number of bytes filtered during the query execution.

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

CLAassistant · 2025-02-24T05:45:36Z

All committers have signed the CLA.

antaljanosbenjamin

We also need tests for this.

src/Common/ProfileEvents.cpp

src/Processors/Transforms/FilterTransform.cpp

clickhouse-gh · 2025-02-24T12:25:50Z

Workflow [PR], commit [af52977]

onkar · 2025-02-25T20:41:18Z

Hi @antaljanosbenjamin, thank you for reviewing the PR. I addressed your comments, please take a look and let me know if anything else needs to be changed.

The following tests are failing:

test_odbc_interaction/test.py::test_postgres_odbc_hashed_dictionary_no_tty_pipe_overflow but my change has not touched any ODBC logic.
Stress test (ubsan) but the stack trace of this failure does not seem to be related to my change.

Please advise on the next steps to deal with these test failures.

src/Processors/Transforms/FilterTransform.cpp

tests/queries/0_stateless/03362_filter_transform_profile_events.sql

antaljanosbenjamin · 2025-03-04T09:46:37Z

Sorry for the late answer, I was busy. The PR looks good. Could you please resolve the conflicts?

The test failures seems to be unrelated.

onkar · 2025-03-04T23:23:47Z

Sorry for the late answer, I was busy. The PR looks good. Could you please resolve the conflicts?

The test failures seems to be unrelated.

Thanks for the re-review @antaljanosbenjamin. I have resolved the conflicts. Please take a look and merge if everything looks good.

antaljanosbenjamin · 2025-03-05T10:50:42Z

CH Inc sync failed for unrelated reason, merging this.

Fix rebase issue: - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

Fix rebase issue: - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

Add ProfileEvents for number of rows and bytes passing WHERE clause

22650a3

antaljanosbenjamin self-assigned this Feb 24, 2025

antaljanosbenjamin reviewed Feb 24, 2025

View reviewed changes

src/Common/ProfileEvents.cpp Outdated Show resolved Hide resolved

src/Processors/Transforms/FilterTransform.cpp Outdated Show resolved Hide resolved

antaljanosbenjamin added the can be tested Allows running workflows for external contributors label Feb 24, 2025

clickhouse-gh bot added the pr-feature Pull request with new product feature label Feb 24, 2025

onkar added 2 commits February 25, 2025 00:59

Address review comments and add test

568632b

Check style fix

51c4939

onkar changed the title ~~Add ProfileEvents for number of rows and bytes passing WHERE clause~~ Add ProfileEvents for number of rows and bytes passing filter transform Feb 25, 2025

More style fixes

d8bbe15

antaljanosbenjamin approved these changes Feb 26, 2025

View reviewed changes

src/Processors/Transforms/FilterTransform.cpp Outdated Show resolved Hide resolved

src/Processors/Transforms/FilterTransform.cpp Show resolved Hide resolved

tests/queries/0_stateless/03362_filter_transform_profile_events.sql Show resolved Hide resolved

Address remaining review feedback

8ec98d8

onkar force-pushed the onkar/profile-event branch from 4af99ac to 8ec98d8 Compare February 26, 2025 19:11

Fix test failures

ba168d0

onkar force-pushed the onkar/profile-event branch from 112a814 to ba168d0 Compare February 27, 2025 01:57

onkar added 3 commits February 27, 2025 05:29

Stabilize stateless test

d80b1c1

Add test with deterministic output

f8f30a5

Add reference file

ce8f522

antaljanosbenjamin approved these changes Mar 4, 2025

View reviewed changes

Merge branch 'master' into onkar/profile-event

af52977

antaljanosbenjamin added this pull request to the merge queue Mar 5, 2025

Merged via the queue into ClickHouse:master with commit bee4ac9 Mar 5, 2025
122 of 125 checks passed

robot-clickhouse-ci-1 added the pr-synced-to-cloud The PR is synced to the cloud repo label Mar 5, 2025

baibaichen pushed a commit to Kyligence/ClickHouse that referenced this pull request Mar 6, 2025

Support new native parquet reader (#516)

1c14a89

Fix rebase issue: - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Mar 31, 2025

Support new native parquet reader (#516)

cc698f3

Fix rebase issue: - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 1, 2025

Support new native parquet reader (#516)

a4c090b

Fix rebase issue: - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 2, 2025

Support new native parquet reader (#516)

29caf7d

Fix rebase issue: - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 3, 2025

Support new native parquet reader (#516)

e021055

Fix rebase issue: - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 4, 2025

Support new native parquet reader (#516)

f092c2d

Fix rebase issue: - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 5, 2025

Support new native parquet reader (#516)

0485e83

Fix rebase issue: - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 6, 2025

Support new native parquet reader (#516)

cd0d6d1

Fix rebase issue: - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 7, 2025

Support new native parquet reader (#516)

f669220

Fix rebase issue: - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

baibaichen pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 9, 2025

Support new native parquet reader (#516)

d962c68

Fix rebase issue: - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 9, 2025

Support new native parquet reader (#516)

709dc48

Fix rebase issue: - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 10, 2025

Support new native parquet reader (#516)

dd776e0

Fix rebase issue: - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 11, 2025

Support new native parquet reader (#516)

637373b

Fix rebase issue: - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 12, 2025

Support new native parquet reader (#516)

bbfea35

Fix rebase issue: - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 13, 2025

Support new native parquet reader (#516)

112036f

Fix rebase issue: - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 14, 2025

Support new native parquet reader (#516)

4099265

Fix rebase issue: - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

baibaichen pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 16, 2025

Support new native parquet reader (#516)

c2aa673

Fix rebase issue: - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 16, 2025

Support new native parquet reader (#516)

6da112b

Fix rebase issue: - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 17, 2025

Support new native parquet reader (#516)

a362445

Fix rebase issue: - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 18, 2025

Support new native parquet reader (#516)

be48320

Fix rebase issue: - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 19, 2025

Support new native parquet reader (#516)

7e05928

Fix rebase issue: - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

baibaichen pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 21, 2025

Support new native parquet reader (#516)

14ce457

Fix rebase issue: - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 21, 2025

Support new native parquet reader (#516)

209bef7

Fix rebase issue: - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 22, 2025

Support new native parquet reader (#516)

dfd82ad

Fix rebase issue: - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 23, 2025

Support new native parquet reader (#516)

50a891c

Fix rebase issue: - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 24, 2025

Support new native parquet reader (#516)

0f5a035

Fix rebase issue: - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 25, 2025

Support new native parquet reader (#516)

d45385b

Fix rebase issue: - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 26, 2025

Support new native parquet reader (#516)

e7916e7

Fix rebase issue: - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 27, 2025

Support new native parquet reader (#516)

d0cf0a9

Fix rebase issue: - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 28, 2025

Support new native parquet reader (#516)

718d30b

Fix rebase issue: - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

kyligence-git pushed a commit to Kyligence/ClickHouse that referenced this pull request Apr 29, 2025

Support new native parquet reader (#516)

692047b

Fix rebase issue: - 20250416 ClickHouse#78485 - 20250306 ClickHouse#76662 Co-authored-by: liuneng1994 <neng.liu@kyligence.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ProfileEvents for number of rows and bytes passing filter transform#76662

Add ProfileEvents for number of rows and bytes passing filter transform#76662
antaljanosbenjamin merged 10 commits intoClickHouse:masterfrom
onkar:onkar/profile-event

onkar commented Feb 24, 2025 •

edited

Loading

Uh oh!

CLAassistant commented Feb 24, 2025 •

edited

Loading

Uh oh!

antaljanosbenjamin left a comment

Uh oh!

Uh oh!

Uh oh!

clickhouse-gh bot commented Feb 24, 2025 •

edited

Loading

Uh oh!

onkar commented Feb 25, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

antaljanosbenjamin commented Mar 4, 2025

Uh oh!

onkar commented Mar 4, 2025

Uh oh!

antaljanosbenjamin commented Mar 5, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

onkar commented Feb 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

Uh oh!

CLAassistant commented Feb 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

antaljanosbenjamin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

clickhouse-gh bot commented Feb 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

onkar commented Feb 25, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

antaljanosbenjamin commented Mar 4, 2025

Uh oh!

onkar commented Mar 4, 2025

Uh oh!

antaljanosbenjamin commented Mar 5, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

onkar commented Feb 24, 2025 •

edited

Loading

CLAassistant commented Feb 24, 2025 •

edited

Loading

clickhouse-gh bot commented Feb 24, 2025 •

edited

Loading