Skip to content

optimize joins, redirect dml for reference tables#11875

Merged
systay merged 24 commits intomainfrom
maxeng-route-unqualified-ref
Dec 8, 2022
Merged

optimize joins, redirect dml for reference tables#11875
systay merged 24 commits intomainfrom
maxeng-route-unqualified-ref

Conversation

@maxenglander
Copy link
Copy Markdown
Collaborator

@maxenglander maxenglander commented Dec 4, 2022

Addresses #11864.

Description

Make some improvements to routing for reference tables:

  • Support for globally routing reference tables that are identically named with their source.
  • Optimize joining so that reference tables are joined with the optimal keyspace.
  • Redirect DML for reference tables back to their source.

Given:

  • An unsharded keyspace k1 and an sharded keyspace k2.
  • Source table k1.r and reference table k2.r.
  • Unsharded table k1.a
  • Sharded table k2.b.

And these queries:

  1. SELECT k2.b JOIN k1.r
  2. SELECT a JOIN r
  3. SELECT b JOIN r
  4. INSERT INTO r

The first query will succeed but will issue queries to both k1 and k2, even though the query could be satisifed just by routing to k2.

The next three queries will fail with

return nil, fmt.Errorf("ambiguous table reference: %s", tablename)

This PR adds limited support for optimally joining and globally routing these three queries:

  1. The first query is routed exclusively to k2.
    2 The second query will join k1.a to k1.r.
  • The third query will join k2.b to k2.r.
  • The fourth query will be routed to the unsharded source table r.

Details

Some notes about the implementation:

  • Have intentionally put a lot of constraints (more on this in docs) on this feature. For example requiring that a source is unique in a given keyspace. Future PRs could relax these constraints, but figured it's better to start with a smaller initially.
  • This feature will override qualified queries. E.g. SELECT k2.a JOIN k1.r will route r to k2, even though the user has explicitly requested k1.r.
  • Routing rules take precedence over this feature.

Checklist

  • Add support for source to table VSchema definition.
  • Treat reference tables and their source as globally unambiguous.
  • Route unqualified INSERT to reference tables back to the source.
  • Join unqualified table to the keyspace-local reference.
  • Write tests
  • Add docs

@systay

This comment was marked as resolved.

Copy link
Copy Markdown
Collaborator

@systay systay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really nice

@harshit-gangal
Copy link
Copy Markdown
Member

As discussed offline. Few more things to be addressed

  • Always tell VTGate to select optimal route, unless routing rules dictate otherwise. Later on can opt out with a query hint. Probably will do this with a second FindRoutedTable call in route planning code.
  • Add txt-file based route planning tests, in a separate file from other tests.
  • The reference routing does not have to rely on global routing.

Signed-off-by: Max Englander <max@planetscale.com>
Signed-off-by: Max Englander <max@planetscale.com>
…te loop)

Signed-off-by: Max Englander <max@planetscale.com>
Signed-off-by: Max Englander <max@planetscale.com>
Signed-off-by: Max Englander <max@planetscale.com>
Signed-off-by: Max Englander <max@planetscale.com>
Signed-off-by: Max Englander <max@planetscale.com>
Signed-off-by: Max Englander <max@planetscale.com>
Signed-off-by: Max Englander <max@planetscale.com>
Signed-off-by: Max Englander <max@planetscale.com>
Signed-off-by: Max Englander <max@planetscale.com>
…sUsed)

Signed-off-by: Max Englander <max@planetscale.com>
Signed-off-by: Max Englander <max@planetscale.com>
Signed-off-by: Max Englander <max@planetscale.com>
Signed-off-by: Max Englander <max@planetscale.com>
…ne.(*Route).TableName

Signed-off-by: Max Englander <max@planetscale.com>
Signed-off-by: Max Englander <max@planetscale.com>
Signed-off-by: Max Englander <max@planetscale.com>
Signed-off-by: Max Englander <max@planetscale.com>
@maxenglander maxenglander force-pushed the maxeng-route-unqualified-ref branch from d9f5b0f to 73d8cd6 Compare December 8, 2022 18:20
@maxenglander
Copy link
Copy Markdown
Collaborator Author

@harshit-gangal @systay thanks so much for the great reviews and for the approvals. Please hold off on merging until I do a last round of validation, thanks!

@maxenglander
Copy link
Copy Markdown
Collaborator Author

OK tested ready to merge from my POV :)

@systay systay merged commit e64b03e into main Dec 8, 2022
@systay systay deleted the maxeng-route-unqualified-ref branch December 8, 2022 20:40
@systay
Copy link
Copy Markdown
Collaborator

systay commented Dec 8, 2022

🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Component: Query Serving Type: Enhancement Logical improvement (somewhere between a bug and feature)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants