opt: allow casts in initial CTE expression#62808
Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom Mar 30, 2021
Merged
opt: allow casts in initial CTE expression#62808craig[bot] merged 1 commit intocockroachdb:masterfrom
craig[bot] merged 1 commit intocockroachdb:masterfrom
Conversation
In cockroachdb#60560, we made the matching of types in UNIONs more strict. In the recursive CTE code, we don't allow adding casts to the initial expression, so the change caused us to regress in terms of supported queries. This change fixes this by allowing casts to the initial expression. Not sure why I didn't allow this from the get-go. Release note (bug fix): fixed "types cannot be matched for WITH RECURSIVE" error in cases where we can cast the type in the initial expression.
Member
mgartner
approved these changes
Mar 30, 2021
Contributor
mgartner
left a comment
There was a problem hiding this comment.
But a little nervous that February Radu might have had a good reason to disallow this. I can't think of any reason though.
Reviewed 3 of 3 files at r1.
Reviewable status:complete! 1 of 0 LGTMs obtained (waiting on @rytaft)
RaduBerinde
commented
Mar 30, 2021
Member
Author
RaduBerinde
left a comment
There was a problem hiding this comment.
TFTR! In that PR I followed the old code that was there which didn't allow casts on the initial expression (and it was that way just for simplicity's sake).
Red CI is a flake in version-upgrade.
bors r+
Reviewable status:
complete! 1 of 0 LGTMs obtained (waiting on @rytaft)
This was referenced Mar 30, 2021
Contributor
|
Build succeeded: |
msirek
pushed a commit
to msirek/cockroach
that referenced
this pull request
Jan 2, 2023
Fixes cockroachdb#93371 Given the query: ``` SET VECTORIZE=off; WITH RECURSIVE foo(i) AS (SELECT i FROM (VALUES(1),(2)) t(i) UNION ALL SELECT (i+1)::numeric(10,0) FROM foo WHERE i < 10) SELECT * FROM foo; ``` column `i` is CAST to the output type of the UNION ALL due to cockroachdb#62808 in order to support more recursive CTEs without explicit CASTS in seed (initial) expressions: https://github.com/cockroachdb/cockroach/blob/0eac48e70abbdd163d4617e50659321b99d678d4/pkg/sql/opt/optbuilder/with.go#L335-L337 Column `i` is assigned column id 3 in `WithScanPrivate.InCols` and the expression in `recursiveScope.cols[0].scalar` references it as follows: ``` recursiveScope.cols[0].scalar: result = {opt.ScalarExpr | *memo.CastExpr} Input = {opt.ScalarExpr | *memo.PlusExpr} Left = {opt.ScalarExpr | *memo.VariableExpr} Col = {opt.ColumnID} 3 Typ = {*types.T | 0x976b060} InternalType = {types.InternalType} Family = {types.Family} IntFamily (1) ... TypeMeta = {types.UserDefinedTypeMetadata} rank = {opt.ScalarRank} 5 Right = {opt.ScalarExpr | *memo.ConstExpr} Value = {tree.Datum | *tree.DInt} 1 ``` After the CAST is applied, column id 3 is initialized as: ``` initialScope.cols[0].scalar: result = {opt.ScalarExpr | *memo.CastExpr} Input = {opt.ScalarExpr | *memo.VariableExpr} Col = {opt.ColumnID} 1 Typ = {*types.T | 0x976b060} rank = {opt.ScalarRank} 12 Typ = {*types.T | 0xc0037f2780} InternalType = {types.InternalType} Family = {types.Family} DecimalFamily (3) Width = {int32} 0 Precision = {int32} 10 ``` During execution, an assertion fails where the input to the `PlusExpr` is expected to be an integer, but is a decimal: ``` tree.MustBeDInt (datum.go:722) github.com/cockroachdb/cockroach/pkg/sql/sem/tree eval.(*evaluator).EvalPlusIntOp (binary_op.go:1325) github.com/cockroachdb/cockroach/pkg/sql/sem/eval tree.(*PlusIntOp).Eval (eval_op_generated.go:761) github.com/cockroachdb/cockroach/pkg/sql/sem/tree eval.(*evaluator).EvalBinaryExpr (expr.go:120) github.com/cockroachdb/cockroach/pkg/sql/sem/eval tree.(*BinaryExpr).Eval (eval_expr_generated.go:86) github.com/cockroachdb/cockroach/pkg/sql/sem/tree eval.(*evaluator).EvalCastExpr (expr.go:180) github.com/cockroachdb/cockroach/pkg/sql/sem/eval tree.(*CastExpr).Eval (eval_expr_generated.go:96) github.com/cockroachdb/cockroach/pkg/sql/sem/tree eval.Expr (expr.go:26) github.com/cockroachdb/cockroach/pkg/sql/sem/eval execinfrapb.(*ExprHelper).Eval (expr.go:248) github.com/cockroachdb/cockroach/pkg/sql/execinfrapb execinfra.(*ProcOutputHelper).ProcessRow (processorsbase.go:275) github.com/cockroachdb/cockroach/pkg/sql/execinfra ... ``` The left of the Plus is referring to the `IndexedVar` column at ordinal 0: ``` result = {*tree.BinaryExpr | 0xc0022e27c0} Operator = {treebin.BinaryOperator} Symbol = {treebin.BinaryOperatorSymbol} Plus (3) IsExplicitOperator = {bool} false Left = {tree.Expr | *tree.IndexedVar} Idx = {int} 0 Used = {bool} true col = {tree.NodeFormatter | *colinfo.varFormatter} typeAnnotation = {tree.typeAnnotation} Right = {tree.Expr | *tree.DInt} 1 typeAnnotation = {tree.typeAnnotation} Op = {*tree.BinOp | 0xc0001817a0} ``` which has been CAST as a decimal, as indicated above. To fully support casting of seed expressions, that may require replacing all occurences of those expressions in `recursiveScope` with the new CASTed expressions, and possibly re-doing type checking to catch any illegal casts, etc. This issue is solved by reverting cockroachdb#62808 in order to maintain compatibility with Postgres, which errors out this query. Relaxing the requirement for explicit CASTs in recursive CTE seed expressions could possibly be supported in the future, but more work is required to substitute seed expressions present in the recursive branch with the implicitly CASTed expressions and make sure type checking is complete. Release note(bug fix): This patch fixes recursive CTE expressions which cause internal errors when explicit CASTs of initial expressions to output types are missing.
msirek
pushed a commit
to msirek/cockroach
that referenced
this pull request
Jan 4, 2023
Fixes cockroachdb#93371 Given the query: ``` SET VECTORIZE=off; WITH RECURSIVE foo(i) AS (SELECT i FROM (VALUES(1),(2)) t(i) UNION ALL SELECT (i+1)::numeric(10,0) FROM foo WHERE i < 10) SELECT * FROM foo; ``` column `i` is CAST to the output type of the UNION ALL due to cockroachdb#62808 in order to support more recursive CTEs without explicit CASTS in seed (initial) expressions: https://github.com/cockroachdb/cockroach/blob/0eac48e70abbdd163d4617e50659321b99d678d4/pkg/sql/opt/optbuilder/with.go#L335-L337 Column `i` is assigned column id 3 in `WithScanPrivate.InCols` and the expression in `recursiveScope.cols[0].scalar` references it as follows: ``` recursiveScope.cols[0].scalar: result = {opt.ScalarExpr | *memo.CastExpr} Input = {opt.ScalarExpr | *memo.PlusExpr} Left = {opt.ScalarExpr | *memo.VariableExpr} Col = {opt.ColumnID} 3 Typ = {*types.T | 0x976b060} InternalType = {types.InternalType} Family = {types.Family} IntFamily (1) ... TypeMeta = {types.UserDefinedTypeMetadata} rank = {opt.ScalarRank} 5 Right = {opt.ScalarExpr | *memo.ConstExpr} Value = {tree.Datum | *tree.DInt} 1 ``` After the CAST is applied, column id 3 is initialized as: ``` initialScope.cols[0].scalar: result = {opt.ScalarExpr | *memo.CastExpr} Input = {opt.ScalarExpr | *memo.VariableExpr} Col = {opt.ColumnID} 1 Typ = {*types.T | 0x976b060} rank = {opt.ScalarRank} 12 Typ = {*types.T | 0xc0037f2780} InternalType = {types.InternalType} Family = {types.Family} DecimalFamily (3) Width = {int32} 0 Precision = {int32} 10 ``` During execution, an assertion fails where the input to the `PlusExpr` is expected to be an integer, but is a decimal: ``` tree.MustBeDInt (datum.go:722) github.com/cockroachdb/cockroach/pkg/sql/sem/tree eval.(*evaluator).EvalPlusIntOp (binary_op.go:1325) github.com/cockroachdb/cockroach/pkg/sql/sem/eval tree.(*PlusIntOp).Eval (eval_op_generated.go:761) github.com/cockroachdb/cockroach/pkg/sql/sem/tree eval.(*evaluator).EvalBinaryExpr (expr.go:120) github.com/cockroachdb/cockroach/pkg/sql/sem/eval tree.(*BinaryExpr).Eval (eval_expr_generated.go:86) github.com/cockroachdb/cockroach/pkg/sql/sem/tree eval.(*evaluator).EvalCastExpr (expr.go:180) github.com/cockroachdb/cockroach/pkg/sql/sem/eval tree.(*CastExpr).Eval (eval_expr_generated.go:96) github.com/cockroachdb/cockroach/pkg/sql/sem/tree eval.Expr (expr.go:26) github.com/cockroachdb/cockroach/pkg/sql/sem/eval execinfrapb.(*ExprHelper).Eval (expr.go:248) github.com/cockroachdb/cockroach/pkg/sql/execinfrapb execinfra.(*ProcOutputHelper).ProcessRow (processorsbase.go:275) github.com/cockroachdb/cockroach/pkg/sql/execinfra ... ``` The left of the Plus is referring to the `IndexedVar` column at ordinal 0: ``` result = {*tree.BinaryExpr | 0xc0022e27c0} Operator = {treebin.BinaryOperator} Symbol = {treebin.BinaryOperatorSymbol} Plus (3) IsExplicitOperator = {bool} false Left = {tree.Expr | *tree.IndexedVar} Idx = {int} 0 Used = {bool} true col = {tree.NodeFormatter | *colinfo.varFormatter} typeAnnotation = {tree.typeAnnotation} Right = {tree.Expr | *tree.DInt} 1 typeAnnotation = {tree.typeAnnotation} Op = {*tree.BinOp | 0xc0001817a0} ``` which has been CAST as a decimal, as indicated above. To fully support casting of seed expressions, that may require replacing all occurences of those expressions in `recursiveScope` with the new CASTed expressions, and possibly re-doing type checking to catch any illegal casts, etc. This issue is solved by reverting cockroachdb#62808 in order to maintain compatibility with Postgres, which errors out this query. Relaxing the requirement for explicit CASTs in recursive CTE seed expressions could possibly be supported in the future, but more work is required to substitute seed expressions present in the recursive branch with the implicitly CASTed expressions and make sure type checking is complete. Release note (bug fix): This patch fixes recursive CTE expressions which cause internal errors when explicit CASTs of initial expressions to output types are missing.
craig bot
pushed a commit
that referenced
this pull request
Jan 4, 2023
94581: optbuilder: do not CAST recursive CTE seed columns to output types r=msirek a=msirek Fixes #93371 Given the query: ``` SET VECTORIZE=off; WITH RECURSIVE foo(i) AS (SELECT i FROM (VALUES(1),(2)) t(i) UNION ALL SELECT (i+1)::numeric(10,0) FROM foo WHERE i < 10) SELECT * FROM foo; ``` column `i` is CAST to the output type of the UNION ALL due to #62808 in order to support more recursive CTEs without explicit CASTS in seed (initial) expressions: https://github.com/cockroachdb/cockroach/blob/0eac48e70abbdd163d4617e50659321b99d678d4/pkg/sql/opt/optbuilder/with.go#L335-L337 Column `i` is assigned column id 3 in `WithScanPrivate.InCols` and the expression in `recursiveScope.cols[0].scalar` references it as follows: ``` recursiveScope.cols[0].scalar: result = {opt.ScalarExpr | *memo.CastExpr} Input = {opt.ScalarExpr | *memo.PlusExpr} Left = {opt.ScalarExpr | *memo.VariableExpr} Col = {opt.ColumnID} 3 Typ = {*types.T | 0x976b060} InternalType = {types.InternalType} Family = {types.Family} IntFamily (1) ... TypeMeta = {types.UserDefinedTypeMetadata} rank = {opt.ScalarRank} 5 Right = {opt.ScalarExpr | *memo.ConstExpr} Value = {tree.Datum | *tree.DInt} 1 ``` After the CAST is applied, column id 3 is initialized as: ``` initialScope.cols[0].scalar: result = {opt.ScalarExpr | *memo.CastExpr} Input = {opt.ScalarExpr | *memo.VariableExpr} Col = {opt.ColumnID} 1 Typ = {*types.T | 0x976b060} rank = {opt.ScalarRank} 12 Typ = {*types.T | 0xc0037f2780} InternalType = {types.InternalType} Family = {types.Family} DecimalFamily (3) Width = {int32} 0 Precision = {int32} 10 ``` During execution, an assertion fails where the input to the `PlusExpr` is expected to be an integer, but is a decimal: ``` tree.MustBeDInt (datum.go:722) github.com/cockroachdb/cockroach/pkg/sql/sem/tree eval.(*evaluator).EvalPlusIntOp (binary_op.go:1325) github.com/cockroachdb/cockroach/pkg/sql/sem/eval tree.(*PlusIntOp).Eval (eval_op_generated.go:761) github.com/cockroachdb/cockroach/pkg/sql/sem/tree eval.(*evaluator).EvalBinaryExpr (expr.go:120) github.com/cockroachdb/cockroach/pkg/sql/sem/eval tree.(*BinaryExpr).Eval (eval_expr_generated.go:86) github.com/cockroachdb/cockroach/pkg/sql/sem/tree eval.(*evaluator).EvalCastExpr (expr.go:180) github.com/cockroachdb/cockroach/pkg/sql/sem/eval tree.(*CastExpr).Eval (eval_expr_generated.go:96) github.com/cockroachdb/cockroach/pkg/sql/sem/tree eval.Expr (expr.go:26) github.com/cockroachdb/cockroach/pkg/sql/sem/eval execinfrapb.(*ExprHelper).Eval (expr.go:248) github.com/cockroachdb/cockroach/pkg/sql/execinfrapb execinfra.(*ProcOutputHelper).ProcessRow (processorsbase.go:275) github.com/cockroachdb/cockroach/pkg/sql/execinfra ... ``` The left of the Plus is referring to the `IndexedVar` column at ordinal 0: ``` result = {*tree.BinaryExpr | 0xc0022e27c0} Operator = {treebin.BinaryOperator} Symbol = {treebin.BinaryOperatorSymbol} Plus (3) IsExplicitOperator = {bool} false Left = {tree.Expr | *tree.IndexedVar} Idx = {int} 0 Used = {bool} true col = {tree.NodeFormatter | *colinfo.varFormatter} typeAnnotation = {tree.typeAnnotation} Right = {tree.Expr | *tree.DInt} 1 typeAnnotation = {tree.typeAnnotation} Op = {*tree.BinOp | 0xc0001817a0} ``` which has been CAST as a decimal, as indicated above. To fully support casting of seed expressions, that may require replacing all occurences of those expressions in `recursiveScope` with the new CASTed expressions, and possibly re-doing type checking to catch any illegal casts, etc. This issue is solved by reverting #62808 in order to maintain compatibility with Postgres, which errors out this query. Relaxing the requirement for explicit CASTs in recursive CTE seed expressions could possibly be supported in the future, but more work is required to substitute seed expressions present in the recursive branch with the implicitly CASTed expressions and make sure type checking is complete. Release note(bug fix): This patch fixes recursive CTE expressions which cause internal errors when explicit CASTs of initial expressions to output types are missing. 94674: colfetcher: don't use the cFetcher's allocator for other stuff r=yuzefovich a=yuzefovich This commit fixes a minor bug in how we were accounting for the copy of the spans in the ColBatchScan. In particular, the `cFetcher` (or, more precisely, the `colmem.SetAccountingHelper`) requires that the allocator is not shared with other components (so that we could quickly get the footprint of the batch). This was violated in a single spot and is now fixed. The impact of this oversight is very minor - the set accounting helper would overestimate the footprint of the batch, so it could be returning batches smaller than the limit. Thus, the fix doesn't need to be backported. I also audited other users of the set accounting helper and didn't find any other violations. Epic: None Release note: None Co-authored-by: Mark Sirek <sirek@cockroachlabs.com> Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In #60560, we made the matching of types in UNIONs more strict. In the
recursive CTE code, we don't allow adding casts to the initial
expression, so the change caused us to regress in terms of supported
queries.
This change fixes this by allowing casts to the initial expression.
Not sure why I didn't allow this from the get-go.
Release note (bug fix): fixed "types cannot be matched for WITH
RECURSIVE" error in cases where we can cast the type in the initial
expression.