Skip to content

session,server: refactor the execute statement processing#17555

Merged
tiancaiamao merged 13 commits intopingcap:masterfrom
tiancaiamao:refactor-execute-stmt
Jun 5, 2020
Merged

session,server: refactor the execute statement processing#17555
tiancaiamao merged 13 commits intopingcap:masterfrom
tiancaiamao:refactor-execute-stmt

Conversation

@tiancaiamao
Copy link
Contributor

@tiancaiamao tiancaiamao commented Jun 1, 2020

What problem does this PR solve?

Issue Number: close #16817

Problem Summary:

The root cause of the panic in #16817 is the data race.

And the data race exists because we close the transaction immediately in ExecuteStmt.
When using the RecordSet returned by ExecuteStmt, executor.Next() is called and it maybe still refering the txn, the txn is closed and set to nil however.

if err := se.CommitTxn(ctx); err != nil {

rs :=    ExecuteStmt --->
                runStmt --->
                    finishStmt --->
                        CommitTxn --->
                            txn set to nil !!

rs.Next() -->
    executor.Next --->
        PointGet.Next --->
            txn.Valid() ???

What is changed and how it works?

We should not commit a txn until the executor stop refering it.
The correct order is:

  1. ExecuteStmt return a RecordSet rs
  2. Call rs.Next(), drain the chunk data and sent result to the mysql client
  3. Call rs.Close()
  4. Then finishStmt should be called and txn maybe commit here.

What's Changed:

  • Move finishStmt out of runStmt.
  • Wrap a new execStmtResult based on RecordSet, override the Close method,
    and call finishStmt in execStmtResult.Close()

How it Works:

Txn is not commited until the executor close the RecordSet, so there aren't be any concurrent visits of txn between finishStmt and recordset.Next, the data race is fixed.

Related changes

The refactor in #16056 avoid the multiple return value []RecordSet. It makes a solid foundantion towards better code for the session processing.

And this commit is a furthur step after that. This commit solve the problem finally:

The problem is, the recordset is lazy delivered, and it references the state of the session which is always mutable.

This commit guarantee the session won't be modify until the chunk data are drained and recordset is closed.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

runStmtWrap is a copy of runStmt, that's because even a tiny modify would meet numerous CI error, I've tried.

Keep this commit as simple as possible: just move finishStmt to RecordSet.Close
And after it's merged, I'll clean old Execute method (which uses runStmt).

Release note

  • No release note

The executor maybe still refering `txn` inside the Next() function.
Do not call txn.Commit() until the executor close.
@tiancaiamao tiancaiamao marked this pull request as ready for review June 2, 2020 06:27
@tiancaiamao tiancaiamao added component/session require-LGT3 Indicates that the PR requires three LGTM. and removed status/DNM labels Jun 2, 2020
@codecov
Copy link

codecov bot commented Jun 2, 2020

Codecov Report

Merging #17555 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #17555   +/-   ##
===========================================
  Coverage   79.6646%   79.6646%           
===========================================
  Files           524        524           
  Lines        142314     142314           
===========================================
  Hits         113374     113374           
  Misses        19892      19892           
  Partials       9048       9048           

@tiancaiamao
Copy link
Contributor Author

/run-all-tests

PTAL @coocood @jackysp @SunRunAway

@tiancaiamao
Copy link
Contributor Author

/run-all-tests

@tiancaiamao
Copy link
Contributor Author

/run-all-tests

@tiancaiamao
Copy link
Contributor Author

/rebuild

@tiancaiamao
Copy link
Contributor Author

/run-all-tests

@tiancaiamao
Copy link
Contributor Author

PTAL @coocood @jacksp @lysu

@lysu lysu requested review from SunRunAway, coocood and lysu June 5, 2020 03:19
@coocood
Copy link
Member

coocood commented Jun 5, 2020

LGTM

Copy link
Contributor

@lysu lysu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lysu lysu requested a review from jackysp June 5, 2020 05:23
// Execute the physical plan.
logStmt(stmtNode, s.sessionVars)
recordSet, err := runStmt(ctx, s, stmt)
recordSet, err := runStmtWrap(ctx, s, stmt)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the binary protocol (execute statement) will still use runStmt? Do you need to fix it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it will be handled in the following works.

Copy link
Contributor

@jackysp jackysp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jackysp
Copy link
Contributor

jackysp commented Jun 5, 2020

/merge

@sre-bot sre-bot added the status/can-merge Indicates a PR has been approved by a committer. label Jun 5, 2020
@sre-bot
Copy link
Contributor

sre-bot commented Jun 5, 2020

/run-all-tests

@sre-bot
Copy link
Contributor

sre-bot commented Jun 5, 2020

/run-all-tests

@sre-bot
Copy link
Contributor

sre-bot commented Jun 5, 2020

@tiancaiamao merge failed.

@tiancaiamao
Copy link
Contributor Author

/run-all-tests

2 similar comments
@tiancaiamao
Copy link
Contributor Author

/run-all-tests

@tiancaiamao
Copy link
Contributor Author

/run-all-tests

@tiancaiamao
Copy link
Contributor Author

/rebuild

@tiancaiamao
Copy link
Contributor Author

/run-mybatis-test

@tiancaiamao tiancaiamao merged commit 1887f49 into pingcap:master Jun 5, 2020
@tiancaiamao tiancaiamao deleted the refactor-execute-stmt branch June 5, 2020 11:20
@zz-jason
Copy link
Member

@tiancaiamao could you cherry-pick this PR to release 4.0?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/session require-LGT3 Indicates that the PR requires three LGTM. status/can-merge Indicates a PR has been approved by a committer.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

panic when point get is executed under index merge join

6 participants