Skip to content

opt: allow casts in initial CTE expression#62808

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
RaduBerinde:cte-cast
Mar 30, 2021
Merged

opt: allow casts in initial CTE expression#62808
craig[bot] merged 1 commit intocockroachdb:masterfrom
RaduBerinde:cte-cast

Conversation

@RaduBerinde
Copy link
Copy Markdown
Member

In #60560, we made the matching of types in UNIONs more strict. In the
recursive CTE code, we don't allow adding casts to the initial
expression, so the change caused us to regress in terms of supported
queries.

This change fixes this by allowing casts to the initial expression.
Not sure why I didn't allow this from the get-go.

Release note (bug fix): fixed "types cannot be matched for WITH
RECURSIVE" error in cases where we can cast the type in the initial
expression.

In cockroachdb#60560, we made the matching of types in UNIONs more strict. In the
recursive CTE code, we don't allow adding casts to the initial
expression, so the change caused us to regress in terms of supported
queries.

This change fixes this by allowing casts to the initial expression.
Not sure why I didn't allow this from the get-go.

Release note (bug fix): fixed "types cannot be matched for WITH
RECURSIVE" error in cases where we can cast the type in the initial
expression.
@RaduBerinde RaduBerinde requested review from mgartner and rytaft March 30, 2021 16:50
@RaduBerinde RaduBerinde requested a review from a team as a code owner March 30, 2021 16:50
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

Copy link
Copy Markdown
Contributor

@mgartner mgartner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm: But a little nervous that February Radu might have had a good reason to disallow this. I can't think of any reason though.

Reviewed 3 of 3 files at r1.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @rytaft)

Copy link
Copy Markdown
Member Author

@RaduBerinde RaduBerinde left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TFTR! In that PR I followed the old code that was there which didn't allow casts on the initial expression (and it was that way just for simplicity's sake).

Red CI is a flake in version-upgrade.

bors r+

Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @rytaft)

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Mar 30, 2021

Build succeeded:

@craig craig bot merged commit a21edd5 into cockroachdb:master Mar 30, 2021
@RaduBerinde RaduBerinde deleted the cte-cast branch April 2, 2021 02:04
@rafiss rafiss added this to the 21.1 milestone Apr 22, 2021
msirek pushed a commit to msirek/cockroach that referenced this pull request Jan 2, 2023
Fixes cockroachdb#93371

Given the query:
```
SET VECTORIZE=off;
WITH RECURSIVE foo(i) AS
    (SELECT i FROM (VALUES(1),(2)) t(i)
    UNION ALL
    SELECT (i+1)::numeric(10,0) FROM foo WHERE i < 10)
 SELECT * FROM foo;
```
column `i` is CAST to the output type of the UNION ALL due to cockroachdb#62808
in order to support more recursive CTEs without explicit CASTS in
seed (initial) expressions:
https://github.com/cockroachdb/cockroach/blob/0eac48e70abbdd163d4617e50659321b99d678d4/pkg/sql/opt/optbuilder/with.go#L335-L337

Column `i` is assigned column id 3 in `WithScanPrivate.InCols` and
the expression in `recursiveScope.cols[0].scalar` references it as
follows:
```
recursiveScope.cols[0].scalar:
result = {opt.ScalarExpr | *memo.CastExpr}
 Input = {opt.ScalarExpr | *memo.PlusExpr}
  Left = {opt.ScalarExpr | *memo.VariableExpr}
   Col = {opt.ColumnID} 3
   Typ = {*types.T | 0x976b060}
    InternalType = {types.InternalType}
     Family = {types.Family} IntFamily (1)
...
    TypeMeta = {types.UserDefinedTypeMetadata}
   rank = {opt.ScalarRank} 5
  Right = {opt.ScalarExpr | *memo.ConstExpr}
   Value = {tree.Datum | *tree.DInt} 1
```
After the CAST is applied, column id 3 is initialized as:
```
initialScope.cols[0].scalar:
 result = {opt.ScalarExpr | *memo.CastExpr}
 Input = {opt.ScalarExpr | *memo.VariableExpr}
  Col = {opt.ColumnID} 1
  Typ = {*types.T | 0x976b060}
  rank = {opt.ScalarRank} 12
 Typ = {*types.T | 0xc0037f2780}
  InternalType = {types.InternalType}
   Family = {types.Family} DecimalFamily (3)
   Width = {int32} 0
   Precision = {int32} 10
```
During execution, an assertion fails where the input to the `PlusExpr` is
expected to be an integer, but is a decimal:
```
tree.MustBeDInt (datum.go:722) github.com/cockroachdb/cockroach/pkg/sql/sem/tree
eval.(*evaluator).EvalPlusIntOp (binary_op.go:1325) github.com/cockroachdb/cockroach/pkg/sql/sem/eval
tree.(*PlusIntOp).Eval (eval_op_generated.go:761) github.com/cockroachdb/cockroach/pkg/sql/sem/tree
eval.(*evaluator).EvalBinaryExpr (expr.go:120) github.com/cockroachdb/cockroach/pkg/sql/sem/eval
tree.(*BinaryExpr).Eval (eval_expr_generated.go:86) github.com/cockroachdb/cockroach/pkg/sql/sem/tree
eval.(*evaluator).EvalCastExpr (expr.go:180) github.com/cockroachdb/cockroach/pkg/sql/sem/eval
tree.(*CastExpr).Eval (eval_expr_generated.go:96) github.com/cockroachdb/cockroach/pkg/sql/sem/tree
eval.Expr (expr.go:26) github.com/cockroachdb/cockroach/pkg/sql/sem/eval
execinfrapb.(*ExprHelper).Eval (expr.go:248) github.com/cockroachdb/cockroach/pkg/sql/execinfrapb
execinfra.(*ProcOutputHelper).ProcessRow (processorsbase.go:275) github.com/cockroachdb/cockroach/pkg/sql/execinfra
...
```
The left of the Plus is referring to the `IndexedVar` column at ordinal 0:
```
result = {*tree.BinaryExpr | 0xc0022e27c0}
 Operator = {treebin.BinaryOperator}
  Symbol = {treebin.BinaryOperatorSymbol} Plus (3)
  IsExplicitOperator = {bool} false
 Left = {tree.Expr | *tree.IndexedVar}
  Idx = {int} 0
  Used = {bool} true
  col = {tree.NodeFormatter | *colinfo.varFormatter}
  typeAnnotation = {tree.typeAnnotation}
 Right = {tree.Expr | *tree.DInt} 1
 typeAnnotation = {tree.typeAnnotation}
 Op = {*tree.BinOp | 0xc0001817a0}
 ```
which has been CAST as a decimal, as indicated above.

To fully support casting of seed expressions, that may require replacing
all occurences of those expressions in `recursiveScope` with the new
CASTed expressions, and possibly re-doing type checking to catch any
illegal casts, etc.

This issue is solved by reverting cockroachdb#62808 in order to maintain
compatibility with Postgres, which errors out this query.
Relaxing the requirement for explicit CASTs in recursive CTE seed
expressions could possibly be supported in the future, but more work is
required to substitute seed expressions present in the recursive branch
with the implicitly CASTed expressions and make sure type checking is
complete.

Release note(bug fix): This patch fixes recursive CTE expressions which
cause internal errors when explicit CASTs of initial expressions to
output types are missing.
msirek pushed a commit to msirek/cockroach that referenced this pull request Jan 4, 2023
Fixes cockroachdb#93371

Given the query:
```
SET VECTORIZE=off;
WITH RECURSIVE foo(i) AS
    (SELECT i FROM (VALUES(1),(2)) t(i)
    UNION ALL
    SELECT (i+1)::numeric(10,0) FROM foo WHERE i < 10)
 SELECT * FROM foo;
```
column `i` is CAST to the output type of the UNION ALL due to cockroachdb#62808
in order to support more recursive CTEs without explicit CASTS in
seed (initial) expressions:
https://github.com/cockroachdb/cockroach/blob/0eac48e70abbdd163d4617e50659321b99d678d4/pkg/sql/opt/optbuilder/with.go#L335-L337

Column `i` is assigned column id 3 in `WithScanPrivate.InCols` and
the expression in `recursiveScope.cols[0].scalar` references it as
follows:
```
recursiveScope.cols[0].scalar:
result = {opt.ScalarExpr | *memo.CastExpr}
 Input = {opt.ScalarExpr | *memo.PlusExpr}
  Left = {opt.ScalarExpr | *memo.VariableExpr}
   Col = {opt.ColumnID} 3
   Typ = {*types.T | 0x976b060}
    InternalType = {types.InternalType}
     Family = {types.Family} IntFamily (1)
...
    TypeMeta = {types.UserDefinedTypeMetadata}
   rank = {opt.ScalarRank} 5
  Right = {opt.ScalarExpr | *memo.ConstExpr}
   Value = {tree.Datum | *tree.DInt} 1
```
After the CAST is applied, column id 3 is initialized as:
```
initialScope.cols[0].scalar:
 result = {opt.ScalarExpr | *memo.CastExpr}
 Input = {opt.ScalarExpr | *memo.VariableExpr}
  Col = {opt.ColumnID} 1
  Typ = {*types.T | 0x976b060}
  rank = {opt.ScalarRank} 12
 Typ = {*types.T | 0xc0037f2780}
  InternalType = {types.InternalType}
   Family = {types.Family} DecimalFamily (3)
   Width = {int32} 0
   Precision = {int32} 10
```
During execution, an assertion fails where the input to the `PlusExpr` is
expected to be an integer, but is a decimal:
```
tree.MustBeDInt (datum.go:722) github.com/cockroachdb/cockroach/pkg/sql/sem/tree
eval.(*evaluator).EvalPlusIntOp (binary_op.go:1325) github.com/cockroachdb/cockroach/pkg/sql/sem/eval
tree.(*PlusIntOp).Eval (eval_op_generated.go:761) github.com/cockroachdb/cockroach/pkg/sql/sem/tree
eval.(*evaluator).EvalBinaryExpr (expr.go:120) github.com/cockroachdb/cockroach/pkg/sql/sem/eval
tree.(*BinaryExpr).Eval (eval_expr_generated.go:86) github.com/cockroachdb/cockroach/pkg/sql/sem/tree
eval.(*evaluator).EvalCastExpr (expr.go:180) github.com/cockroachdb/cockroach/pkg/sql/sem/eval
tree.(*CastExpr).Eval (eval_expr_generated.go:96) github.com/cockroachdb/cockroach/pkg/sql/sem/tree
eval.Expr (expr.go:26) github.com/cockroachdb/cockroach/pkg/sql/sem/eval
execinfrapb.(*ExprHelper).Eval (expr.go:248) github.com/cockroachdb/cockroach/pkg/sql/execinfrapb
execinfra.(*ProcOutputHelper).ProcessRow (processorsbase.go:275) github.com/cockroachdb/cockroach/pkg/sql/execinfra
...
```
The left of the Plus is referring to the `IndexedVar` column at ordinal 0:
```
result = {*tree.BinaryExpr | 0xc0022e27c0}
 Operator = {treebin.BinaryOperator}
  Symbol = {treebin.BinaryOperatorSymbol} Plus (3)
  IsExplicitOperator = {bool} false
 Left = {tree.Expr | *tree.IndexedVar}
  Idx = {int} 0
  Used = {bool} true
  col = {tree.NodeFormatter | *colinfo.varFormatter}
  typeAnnotation = {tree.typeAnnotation}
 Right = {tree.Expr | *tree.DInt} 1
 typeAnnotation = {tree.typeAnnotation}
 Op = {*tree.BinOp | 0xc0001817a0}
 ```
which has been CAST as a decimal, as indicated above.

To fully support casting of seed expressions, that may require replacing
all occurences of those expressions in `recursiveScope` with the new
CASTed expressions, and possibly re-doing type checking to catch any
illegal casts, etc.

This issue is solved by reverting cockroachdb#62808 in order to maintain
compatibility with Postgres, which errors out this query.
Relaxing the requirement for explicit CASTs in recursive CTE seed
expressions could possibly be supported in the future, but more work is
required to substitute seed expressions present in the recursive branch
with the implicitly CASTed expressions and make sure type checking is
complete.

Release note (bug fix): This patch fixes recursive CTE expressions which
cause internal errors when explicit CASTs of initial expressions to
output types are missing.
craig bot pushed a commit that referenced this pull request Jan 4, 2023
94581: optbuilder: do not CAST recursive CTE seed columns to output types r=msirek a=msirek

Fixes #93371

Given the query:
```
SET VECTORIZE=off;
WITH RECURSIVE foo(i) AS
    (SELECT i FROM (VALUES(1),(2)) t(i)
    UNION ALL
    SELECT (i+1)::numeric(10,0) FROM foo WHERE i < 10)
 SELECT * FROM foo;
```
column `i` is CAST to the output type of the UNION ALL due to #62808 in order to support more recursive CTEs without explicit CASTS in seed (initial) expressions:
https://github.com/cockroachdb/cockroach/blob/0eac48e70abbdd163d4617e50659321b99d678d4/pkg/sql/opt/optbuilder/with.go#L335-L337

Column `i` is assigned column id 3 in `WithScanPrivate.InCols` and the expression in `recursiveScope.cols[0].scalar` references it as follows:
```
recursiveScope.cols[0].scalar:
result = {opt.ScalarExpr | *memo.CastExpr}
 Input = {opt.ScalarExpr | *memo.PlusExpr}
  Left = {opt.ScalarExpr | *memo.VariableExpr}
   Col = {opt.ColumnID} 3
   Typ = {*types.T | 0x976b060}
    InternalType = {types.InternalType}
     Family = {types.Family} IntFamily (1)
...
    TypeMeta = {types.UserDefinedTypeMetadata}
   rank = {opt.ScalarRank} 5
  Right = {opt.ScalarExpr | *memo.ConstExpr}
   Value = {tree.Datum | *tree.DInt} 1
```
After the CAST is applied, column id 3 is initialized as:
```
initialScope.cols[0].scalar:
 result = {opt.ScalarExpr | *memo.CastExpr}
 Input = {opt.ScalarExpr | *memo.VariableExpr}
  Col = {opt.ColumnID} 1
  Typ = {*types.T | 0x976b060}
  rank = {opt.ScalarRank} 12
 Typ = {*types.T | 0xc0037f2780}
  InternalType = {types.InternalType}
   Family = {types.Family} DecimalFamily (3)
   Width = {int32} 0
   Precision = {int32} 10
```
During execution, an assertion fails where the input to the `PlusExpr` is expected to be an integer, but is a decimal:
```
tree.MustBeDInt (datum.go:722) github.com/cockroachdb/cockroach/pkg/sql/sem/tree
eval.(*evaluator).EvalPlusIntOp (binary_op.go:1325) github.com/cockroachdb/cockroach/pkg/sql/sem/eval
tree.(*PlusIntOp).Eval (eval_op_generated.go:761) github.com/cockroachdb/cockroach/pkg/sql/sem/tree
eval.(*evaluator).EvalBinaryExpr (expr.go:120) github.com/cockroachdb/cockroach/pkg/sql/sem/eval
tree.(*BinaryExpr).Eval (eval_expr_generated.go:86) github.com/cockroachdb/cockroach/pkg/sql/sem/tree
eval.(*evaluator).EvalCastExpr (expr.go:180) github.com/cockroachdb/cockroach/pkg/sql/sem/eval
tree.(*CastExpr).Eval (eval_expr_generated.go:96) github.com/cockroachdb/cockroach/pkg/sql/sem/tree
eval.Expr (expr.go:26) github.com/cockroachdb/cockroach/pkg/sql/sem/eval
execinfrapb.(*ExprHelper).Eval (expr.go:248) github.com/cockroachdb/cockroach/pkg/sql/execinfrapb
execinfra.(*ProcOutputHelper).ProcessRow (processorsbase.go:275) github.com/cockroachdb/cockroach/pkg/sql/execinfra
...
```
The left of the Plus is referring to the `IndexedVar` column at ordinal 0:
```
result = {*tree.BinaryExpr | 0xc0022e27c0}
 Operator = {treebin.BinaryOperator}
  Symbol = {treebin.BinaryOperatorSymbol} Plus (3)
  IsExplicitOperator = {bool} false
 Left = {tree.Expr | *tree.IndexedVar}
  Idx = {int} 0
  Used = {bool} true
  col = {tree.NodeFormatter | *colinfo.varFormatter}
  typeAnnotation = {tree.typeAnnotation}
 Right = {tree.Expr | *tree.DInt} 1
 typeAnnotation = {tree.typeAnnotation}
 Op = {*tree.BinOp | 0xc0001817a0}
 ```
which has been CAST as a decimal, as indicated above.

To fully support casting of seed expressions, that may require replacing
all occurences of those expressions in `recursiveScope` with the new
CASTed expressions, and possibly re-doing type checking to catch any
illegal casts, etc.

This issue is solved by reverting #62808 in order to maintain
compatibility with Postgres, which errors out this query.
Relaxing the requirement for explicit CASTs in recursive CTE seed
expressions could possibly be supported in the future, but more work is
required to substitute seed expressions present in the recursive branch
with the implicitly CASTed expressions and make sure type checking is
complete.

Release note(bug fix): This patch fixes recursive CTE expressions which
cause internal errors when explicit CASTs of initial expressions to
output types are missing.

94674: colfetcher: don't use the cFetcher's allocator for other stuff r=yuzefovich a=yuzefovich

This commit fixes a minor bug in how we were accounting for the copy of the spans in the ColBatchScan. In particular, the `cFetcher` (or, more precisely, the `colmem.SetAccountingHelper`) requires that the allocator is not shared with other components (so that we could quickly get the footprint of the batch). This was violated in a single spot and is now fixed. The impact of this oversight is very minor - the set accounting helper would overestimate the footprint of the batch, so it could be returning batches smaller than the limit. Thus, the fix doesn't need to be backported.

I also audited other users of the set accounting helper and didn't find any other violations.

Epic: None

Release note: None

Co-authored-by: Mark Sirek <sirek@cockroachlabs.com>
Co-authored-by: Yahor Yuzefovich <yahor@cockroachlabs.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants