Skip to content

opt: add indexed var for inverted column when execbuilding inverted joins#50754

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
rytaft:fix-inverted-join
Jun 30, 2020
Merged

opt: add indexed var for inverted column when execbuilding inverted joins#50754
craig[bot] merged 1 commit intocockroachdb:masterfrom
rytaft:fix-inverted-join

Conversation

@rytaft
Copy link
Copy Markdown
Collaborator

@rytaft rytaft commented Jun 29, 2020

This commit fixes an issue identified in #50709 in which the execbuilder
code for building inverted joins was failing due to the lack of an indexed
var for the column indexed by the inverted index. The indexed var is
necessary since the inverted expression contains a reference to the inverted
column. This commit fixes the issue by including the column in the indexed var
helper.

This commit does not include a release note since this issue was introduced
a few days ago and has not been released.

Release note: None

@rytaft rytaft requested a review from sumeerbhola June 29, 2020 12:52
@rytaft rytaft requested a review from a team as a code owner June 29, 2020 12:52
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@rytaft
Copy link
Copy Markdown
Collaborator Author

rytaft commented Jun 29, 2020

Hold off on reviewing this for now -- I'm testing it with #50709 to see if I can identify any other issues.

Copy link
Copy Markdown
Collaborator

@sumeerbhola sumeerbhola left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @rytaft and @sumeerbhola)


pkg/sql/opt/exec/execbuilder/relational.go, line 1403 at r1 (raw file):

	lookupOrdinals, lookupColMap := b.getColumns(lookupCols, join.Table)
	allCols := joinOutputMap(input.outputCols, lookupColMap)

I think this will include this inverted column in allCols which is the output of this join, which isn't quite correct.

Are we going down the path of the invertedFilterer integration where we couldn't figure out how to do the col mappings in a way that the inverted column was provided in the input but not the output and needed the projection?

@rytaft rytaft force-pushed the fix-inverted-join branch from b0a4fe7 to dd7e9da Compare June 29, 2020 14:06
@rytaft
Copy link
Copy Markdown
Collaborator Author

rytaft commented Jun 29, 2020


pkg/sql/opt/exec/execbuilder/relational.go, line 1403 at r1 (raw file):

Previously, sumeerbhola wrote…

I think this will include this inverted column in allCols which is the output of this join, which isn't quite correct.

Are we going down the path of the invertedFilterer integration where we couldn't figure out how to do the col mappings in a way that the inverted column was provided in the input but not the output and needed the projection?

Yea, just added the post projection here. PTAL.

I tried to remove it from the outputCols directly, but it still showed up in the EXPLAIN, so I don't think it was working.

@rytaft rytaft requested a review from RaduBerinde June 29, 2020 14:50
@rytaft
Copy link
Copy Markdown
Collaborator Author

rytaft commented Jun 29, 2020


pkg/sql/opt/exec/execbuilder/relational.go, line 1403 at r1 (raw file):

Previously, rytaft (Rebecca Taft) wrote…

Yea, just added the post projection here. PTAL.

I tried to remove it from the outputCols directly, but it still showed up in the EXPLAIN, so I don't think it was working.

cc @RaduBerinde do you know if there is a better way to do this?

Copy link
Copy Markdown
Member

@RaduBerinde RaduBerinde left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @RaduBerinde and @sumeerbhola)


pkg/sql/opt/exec/execbuilder/relational.go, line 1403 at r1 (raw file):

Previously, rytaft (Rebecca Taft) wrote…

cc @RaduBerinde do you know if there is a better way to do this?

Not sure I understand the issue.. ConstructInvertedJoin is documented as producing all the lookup columns, if that's the case, yes a projection is needed.

…oins

This commit fixes an issue identified in cockroachdb#50709 in which the execbuilder
code for building inverted joins was failing due to the lack of an indexed
var for the column indexed by the inverted index. The indexed var is
necessary since the inverted expression contains a reference to the inverted
column. This commit fixes the issue by including the column in the indexed var
helper.

This commit does not include a release note since this issue was introduced
a few days ago and has not been released.

Release note: None
@rytaft rytaft force-pushed the fix-inverted-join branch from dd7e9da to 59286e4 Compare June 29, 2020 19:39
@rytaft
Copy link
Copy Markdown
Collaborator Author

rytaft commented Jun 29, 2020


pkg/sql/opt/exec/execbuilder/relational.go, line 1403 at r1 (raw file):

Previously, RaduBerinde wrote…

Not sure I understand the issue.. ConstructInvertedJoin is documented as producing all the lookup columns, if that's the case, yes a projection is needed.

The issue is that it should be possible for the invertedJoiner DistSQL processor to avoid returning the inverted column, which would make the simple project that I added at the end of this function unnecessary. But I couldn't figure out how to pass the correct columns (i.e., get the expected EXPLAIN output and avoid errors and panics) without the project. I'm not sure how much overhead the project actually adds, so this may not be a real issue, but just wanted to get your opinion.

The documentation for ConstructInvertedJoin was stale, so I just updated it to change the sentence "Note that lookupCols does not include the inverted column" to "Note that lookupCols includes the inverted column."

Copy link
Copy Markdown
Collaborator

@sumeerbhola sumeerbhola left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 3 of 4 files at r1, 2 of 2 files at r2.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @sumeerbhola)

@rytaft
Copy link
Copy Markdown
Collaborator Author

rytaft commented Jun 30, 2020

bors r+

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Jun 30, 2020

Build failed

@rytaft
Copy link
Copy Markdown
Collaborator Author

rytaft commented Jun 30, 2020

Opened #50795

bors r+

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Jun 30, 2020

Build succeeded

@craig craig bot merged commit 9cecf3b into cockroachdb:master Jun 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants