Skip to content

fix virtual column not found when remote read happens#9920

Merged
ti-chi-bot[bot] merged 6 commits intopingcap:masterfrom
guo-shaoge:fix_virtual_col_not_found
Feb 27, 2025
Merged

fix virtual column not found when remote read happens#9920
ti-chi-bot[bot] merged 6 commits intopingcap:masterfrom
guo-shaoge:fix_virtual_col_not_found

Conversation

@guo-shaoge
Copy link
Contributor

@guo-shaoge guo-shaoge commented Feb 26, 2025

What problem does this PR solve?

Issue Number: close #9561

Problem Summary:

select col, gen_col from t;

gen_col is a virtual generated column. The current approach is that TiFlash generates an empty column and places it in the block. Then, it passes the block to TiDB, allowing TiDB to calculate and populate the column.

I’ve named this column generated_xxx.

The information about this column is marked in tipb_table_scan.columns with a flag. When TiFlash sees this flag, it automatically adds a generated_xxx column.

However, this situation has an issue when a remote read occurs: the block returned from the remote read, when parsed, assigns names to each column based on RemoteRequest.schema. For example, if TiFlash-1 receives a block from TiFlash-2, it will parse it as <col, gen_col>.

But TiFlash-1, in the table scan's projection, expects the block to be <col, generated_xxx>. So, when projecting the block from the remote read, an error occurs because it cannot find the generated_xxx column.

What is changed and how it works?

make the column name of local read and remote read be same


Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

fix virtual column not found when remote read happens

@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Feb 26, 2025
@guo-shaoge guo-shaoge force-pushed the fix_virtual_col_not_found branch from e0e1759 to 2924572 Compare February 26, 2025 12:37
Signed-off-by: guo-shaoge <shaoge1994@163.com>
@guo-shaoge guo-shaoge force-pushed the fix_virtual_col_not_found branch from 2924572 to 3ae266b Compare February 27, 2025 02:52
@ti-chi-bot ti-chi-bot bot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Feb 27, 2025
Signed-off-by: guo-shaoge <shaoge1994@163.com>
@guo-shaoge guo-shaoge force-pushed the fix_virtual_col_not_found branch from d1ba195 to 30a4e3d Compare February 27, 2025 05:19
Signed-off-by: guo-shaoge <shaoge1994@163.com>
Signed-off-by: guo-shaoge <shaoge1994@163.com>
@guo-shaoge guo-shaoge force-pushed the fix_virtual_col_not_found branch from ac77a2e to 633f195 Compare February 27, 2025 06:24
Signed-off-by: guo-shaoge <shaoge1994@163.com>
@guo-shaoge guo-shaoge force-pushed the fix_virtual_col_not_found branch from 74e1daf to f9d85ec Compare February 27, 2025 07:46
@guo-shaoge guo-shaoge requested a review from solotzg February 27, 2025 07:54
Copy link
Contributor

@solotzg solotzg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@ti-chi-bot ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Feb 27, 2025
Copy link
Contributor

@windtalker windtalker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Feb 27, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: solotzg, windtalker

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added needs-cherry-pick-release-8.1 Should cherry pick this PR to release-8.1 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. needs-cherry-pick-release-6.1 Should cherry pick this PR to release-6.1 branch. needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. labels Feb 27, 2025
@ti-chi-bot ti-chi-bot bot merged commit 27cfea2 into pingcap:master Feb 27, 2025
5 checks passed
ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Feb 27, 2025
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-6.1: #9923.
But this PR has conflicts, please resolve them!

@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-6.5: #9924.
But this PR has conflicts, please resolve them!

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Feb 27, 2025
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Feb 27, 2025
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-7.1: #9925.
But this PR has conflicts, please resolve them!

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Feb 27, 2025
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-7.5: #9926.
But this PR has conflicts, please resolve them!

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Feb 27, 2025
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-8.1: #9927.
But this PR has conflicts, please resolve them!

@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-8.5: #9928.
But this PR has conflicts, please resolve them!

ti-chi-bot pushed a commit to ti-chi-bot/tiflash that referenced this pull request Feb 27, 2025
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot bot pushed a commit that referenced this pull request Feb 27, 2025
close #9561

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: guo-shaoge <shaoge1994@163.com>

Co-authored-by: guo-shaoge <shaoge1994@163.com>
ti-chi-bot bot pushed a commit that referenced this pull request Feb 28, 2025
close #9561

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: guo-shaoge <shaoge1994@163.com>

Co-authored-by: guo-shaoge <shaoge1994@163.com>
ti-chi-bot bot pushed a commit that referenced this pull request Feb 28, 2025
close #9561

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: guo-shaoge <shaoge1994@163.com>

Co-authored-by: guo-shaoge <shaoge1994@163.com>
ti-chi-bot bot pushed a commit that referenced this pull request Feb 28, 2025
close #9561

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: guo-shaoge <shaoge1994@163.com>

Co-authored-by: guo-shaoge <shaoge1994@163.com>
ti-chi-bot bot pushed a commit that referenced this pull request Feb 28, 2025
close #9561

Signed-off-by: guo-shaoge <shaoge1994@163.com>

Co-authored-by: guo-shaoge <shaoge1994@163.com>
ti-chi-bot bot pushed a commit that referenced this pull request Mar 3, 2025
close #9561

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: guo-shaoge <shaoge1994@163.com>

Co-authored-by: guo-shaoge <shaoge1994@163.com>
yongman pushed a commit to yongman/tiflash that referenced this pull request Jun 18, 2025
…ingcap#9928)

close pingcap#9561

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: guo-shaoge <shaoge1994@163.com>

Co-authored-by: guo-shaoge <shaoge1994@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved lgtm needs-cherry-pick-release-6.1 Should cherry pick this PR to release-6.1 branch. needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. needs-cherry-pick-release-8.1 Should cherry pick this PR to release-8.1 branch. needs-cherry-pick-release-8.5 Should cherry pick this PR to release-8.5 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Not found column generated_column_xxx in block when remote read is triggered

4 participants