-
Notifications
You must be signed in to change notification settings - Fork 4.1k
colexec: add a way to propagate metadata in a streaming fashion #55758
Copy link
Copy link
Closed
Labels
A-sql-executionRelating to SQL execution.Relating to SQL execution.A-sql-vecSQL vectorized engineSQL vectorized engineC-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)O-supportWould prevent or help troubleshoot a customer escalation - bugs, missing observability/tooling, docsWould prevent or help troubleshoot a customer escalation - bugs, missing observability/tooling, docsP-3Issues/test failures with no fix SLAIssues/test failures with no fix SLAT-sql-queriesSQL Queries TeamSQL Queries Teamtarget-release-26.2.0v26.2.0-prerelease
Metadata
Metadata
Assignees
Labels
A-sql-executionRelating to SQL execution.Relating to SQL execution.A-sql-vecSQL vectorized engineSQL vectorized engineC-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)O-supportWould prevent or help troubleshoot a customer escalation - bugs, missing observability/tooling, docsWould prevent or help troubleshoot a customer escalation - bugs, missing observability/tooling, docsP-3Issues/test failures with no fix SLAIssues/test failures with no fix SLAT-sql-queriesSQL Queries TeamSQL Queries Teamtarget-release-26.2.0v26.2.0-prerelease
Type
Projects
Status
Done
While working on #55713, I realized that the current way of propagating the metadata in the vectorized engine is incompatible with some things we would like to do. As a reminder, we currently buffer up all metadata in
MetadataSources and return it when draining the whole flow, but there are cases when we would like to propagate the metadata once it is created:rowexec.TableReaders which is then used in combination with the estimated number of rows to be scanned (which is obtained during planning) in order provide the progress estimate for a query (shown onSHOW QUERIESinphasecolumn)The current thinking is that we probably want to implement a pull-based model that is able to return the buffered meta since the previous call (something like
DrainBufferedMetawhich - unlikeDrainMeta- doesn't have an assumption of being called when the flow is shutdown) and then have the root materializer periodically poll its metadata sources.Jira issue: CRDB-3634