ddl: don't reuse the chunk until underlying memory is not referenced by tangenta · Pull Request #39382 · pingcap/tidb

tangenta · 2022-11-24T18:00:33Z

What problem does this PR solve?

Issue Number: close #39308

Problem Summary:

The issue could be reproduced easily when the data set is large enough.

diff --git a/ddl/index.go b/ddl/index.go
index c3c3ffd37c..f5ed601d2b 100644
--- a/ddl/index.go
+++ b/ddl/index.go
@@ -1548,6 +1548,13 @@ func (w *addIndexWorker) BackfillDataInTxn(handleRange reorgBackfillTask) (taskC
                        } else { // The lightning environment is ready.
                                vars := w.sessCtx.GetSessionVars()
                                sCtx, writeBufs := vars.StmtCtx, vars.GetWriteStmtBufs()
+                               logutil.BgLogger().Info("write", zap.String("handle", idxRecord.handle.String()),
+                                       zap.String("idxVal", idxRecord.vals[0].GetString()),
+                                       zap.String("addr", fmt.Sprintf("%p", idxRecords)),
+                                       zap.String("idxAddr", fmt.Sprintf("%p", idxRecord)),
+                                       zap.String("idxDtAddr", fmt.Sprintf("%p", idxRecord.vals)),
+                                       zap.String("idxDt0Addr", fmt.Sprintf("%p", &idxRecord.vals[0])),
+                                       zap.String("d.b", fmt.Sprintf("%p", idxRecord.vals[0].GetBytes())))
                                key, distinct, err := w.index.GenIndexKey(sCtx, idxRecord.vals, idxRecord.handle, writeBufs.IndexKeyBuf)
@@ -291,10 +295,17 @@ func (c *copContext) fetchTableScanResult(ctx context.Context, result distsql.Se
                        return nil, false, errors.Trace(err)
                }
                rsData := tables.TryGetHandleRestoredDataWrapper(c.tblInfo, hdDt, nil, c.idxInfo)
-               buf = append(buf, &indexRecord{handle: handle, key: nil, vals: idxDt, rsData: rsData, skip: false})
+               idxRec := &indexRecord{handle: handle, key: nil, vals: idxDt, rsData: rsData, skip: false}
+               buf = append(buf, idxRec)
+               logutil.BgLogger().Info("rows", zap.String("val", row.ToString(c.fieldTps)),
+                       zap.String("handle", handle.String()),
+                       zap.String("addr", fmt.Sprintf("%p", buf)),
+                       zap.String("idxAddr", fmt.Sprintf("%p", idxRec)),
+                       zap.String("idxDtAddr", fmt.Sprintf("%p", idxRec.vals)),
+                       zap.String("idxDt0Addr", fmt.Sprintf("%p", &idxRec.vals[0])),
+                       zap.String("d.b", fmt.Sprintf("%p", idxRec.vals[0].GetBytes())))
        }

[2022/11/25 01:27:51.107 +08:00] [INFO] [index_cop.go:300] [rows] [val="32644160646-97332178411-20202217618-32930568818-43901782951-91030404202-17535639422-27973945537-69415721036-64437459094, 2561"] [handle=2561] [addr=0x1400182e000] [idxAddr=0x14003662c60] [idxDtAddr=0x14003b562d0] [idxDt0Addr=0x14003b562d0] [d.b=0x14001e60000]

[2022/11/25 01:27:51.107 +08:00] [INFO] [index.go:1551] [write] [handle=1] [idxVal=32644160646-97332178411-20202217618-32930568818-43901782951-91030404202-17535639422-27973945537-69415721036-64437459094] [addr=0x14000e57000] [idxAddr=0x1400069cde0] [idxDtAddr=0x140015381b0] [idxDt0Addr=0x140015381b0] [d.b=0x14001e60000]

The actual data underlying types.Datum is referenced by two different goroutines, because when we initialize the datum from the chunk, we use GetString() for varchar columns.

// GetString gets string value.
func (d *Datum) GetString() string {
	return string(hack.String(d.b))
}

The string is not copied here!

What is changed and how it works?

Similar to idxBufPool, this PR adds a chunk sync.Pool to manage & reuse the memory as soon as possible.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
It is too hard to write a suitable unit test, because it needs a large amount of data and the concurrency. I test it with sysbench, no data inconsistency is found anymore.
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

ti-chi-bot · 2022-11-24T18:00:34Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

Benjamin2037
zimulala

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

Benjamin2037

LGTM

zimulala

LGTM

tangenta · 2022-11-25T03:08:46Z

/merge

ti-chi-bot · 2022-11-25T03:08:49Z

This pull request has been accepted and is ready to merge.

Details

Commit hash: 009058d

hawkingrei

LGTM

FYI, We have the reuse-chunk feature by #38607. we maybe able to use it.

hawkingrei · 2022-11-25T03:34:28Z

/run-all-tests

hawkingrei · 2022-11-25T04:00:51Z

/run-all-tests

Benjamin2037 · 2022-11-25T05:03:59Z

/run-all-tests

tangenta · 2022-11-25T05:27:49Z

/merge

ti-chi-bot · 2022-11-25T05:27:52Z

This pull request has been accepted and is ready to merge.

Details

Commit hash: 1399840

hawkingrei · 2022-11-25T06:00:02Z

@tangenta PTAL

        	Error Trace:	/home/jenkins/.tidb/tmp/63a9840cd0739f2c243bb46478607469/sandbox/linux-sandbox/6874/execroot/__main__/bazel-out/k8-fastbuild/bin/ddl/ddl_test_/ddl_test.runfiles/__main__/ddl/index_cop_test.go:48
        	            				/home/jenkins/.tidb/tmp/63a9840cd0739f2c243bb46478607469/sandbox/linux-sandbox/6874/execroot/__main__/bazel-out/k8-fastbuild/bin/ddl/ddl_test_/ddl_test.runfiles/__main__/ddl/index_cop_test.go:66
        	Error:      	Should be true
        	Test:       	TestAddIndexFetchRowsFromCoprocessor

tangenta · 2022-11-25T06:08:52Z

/merge

ti-chi-bot · 2022-11-25T06:08:55Z

This pull request has been accepted and is ready to merge.

Details

Commit hash: 2642946

sre-bot · 2022-11-25T07:06:56Z

TiDB MergeCI notify

✅ Well Done! New fixed [1] after this pr merged.

CI Name	Result	Duration	Compare with Parent commit
idc-jenkins-ci-tidb/mybatis-test	🔴 failed 1, success 0, total 1	11 min	Existing failure
idc-jenkins-ci-tidb/integration-ddl-test	✅ all 6 tests passed	4 min 48 sec	Fixed
idc-jenkins-ci-tidb/sqllogic-test-2	🟢 all 28 tests passed	32 min	Existing passed
idc-jenkins-ci/integration-cdc-test	🟢 all 39 tests passed	21 min	Existing passed
idc-jenkins-ci-tidb/common-test	🟢 all 11 tests passed	18 min	Existing passed
idc-jenkins-ci-tidb/integration-common-test	🟢 all 17 tests passed	17 min	Existing passed
idc-jenkins-ci-tidb/tics-test	🟢 all 1 tests passed	6 min 36 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-1	🟢 all 26 tests passed	4 min 47 sec	Existing passed
idc-jenkins-ci-tidb/integration-compatibility-test	🟢 all 1 tests passed	2 min 45 sec	Existing passed
idc-jenkins-ci-tidb/plugin-test	🟢 build success, plugin test success	4min	Existing passed

ddl: don't reuse the chunk until underlying memory is not referenced

009058d

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Nov 24, 2022

Benjamin2037 approved these changes Nov 25, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Nov 25, 2022

zimulala approved these changes Nov 25, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Nov 25, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Nov 25, 2022

hawkingrei approved these changes Nov 25, 2022

View reviewed changes

prevent concurrent map access

1399840

ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Nov 25, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Nov 25, 2022

fix TestAddIndexFetchRowsFromCoprocessor

2642946

ti-chi-bot removed the status/can-merge Indicates a PR has been approved by a committer. label Nov 25, 2022

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Nov 25, 2022

ti-chi-bot merged commit b285ef8 into pingcap:master Nov 25, 2022

tangenta mentioned this pull request Nov 29, 2022

ddl: reuse chunk for copr-read and check context done #39473

Merged

12 tasks

Conversation

tangenta commented Nov 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

Uh oh!

ti-chi-bot commented Nov 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Benjamin2037 left a comment

Choose a reason for hiding this comment

Uh oh!

zimulala left a comment

Choose a reason for hiding this comment

Uh oh!

tangenta commented Nov 25, 2022

Uh oh!

ti-chi-bot commented Nov 25, 2022

Uh oh!

hawkingrei left a comment

Choose a reason for hiding this comment

Uh oh!

hawkingrei commented Nov 25, 2022

Uh oh!

hawkingrei commented Nov 25, 2022

Uh oh!

Benjamin2037 commented Nov 25, 2022

Uh oh!

tangenta commented Nov 25, 2022

Uh oh!

ti-chi-bot commented Nov 25, 2022

Uh oh!

hawkingrei commented Nov 25, 2022

Uh oh!

tangenta commented Nov 25, 2022

Uh oh!

ti-chi-bot commented Nov 25, 2022

Uh oh!

sre-bot commented Nov 25, 2022

TiDB MergeCI notify

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

tangenta commented Nov 24, 2022 •

edited

Loading

ti-chi-bot commented Nov 24, 2022 •

edited

Loading