-
Notifications
You must be signed in to change notification settings - Fork 4.1k
colexec: add a way to propagate metadata in a streaming fashion #55758
Copy link
Copy link
Closed
Labels
A-sql-executionRelating to SQL execution.Relating to SQL execution.A-sql-vecSQL vectorized engineSQL vectorized engineC-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)O-supportWould prevent or help troubleshoot a customer escalation - bugs, missing observability/tooling, docsWould prevent or help troubleshoot a customer escalation - bugs, missing observability/tooling, docsP-3Issues/test failures with no fix SLAIssues/test failures with no fix SLAT-sql-queriesSQL Queries TeamSQL Queries Teamtarget-release-26.2.0v26.2.0-prerelease
Description
While working on #55713, I realized that the current way of propagating the metadata in the vectorized engine is incompatible with some things we would like to do. As a reminder, we currently buffer up all metadata in MetadataSources and return it when draining the whole flow, but there are cases when we would like to propagate the metadata once it is created:
- we emit the metadata about the number of rows read from
rowexec.TableReaders which is then used in combination with the estimated number of rows to be scanned (which is obtained during planning) in order provide the progress estimate for a query (shown onSHOW QUERIESinphasecolumn) - the table statistics collection emit progress metadata as well, and although we don't have the vectorized table statistics collection yet, it'd be nice to support the progress update when we do.
The current thinking is that we probably want to implement a pull-based model that is able to return the buffered meta since the previous call (something like DrainBufferedMeta which - unlike DrainMeta - doesn't have an assumption of being called when the flow is shutdown) and then have the root materializer periodically poll its metadata sources.
Jira issue: CRDB-3634
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
A-sql-executionRelating to SQL execution.Relating to SQL execution.A-sql-vecSQL vectorized engineSQL vectorized engineC-enhancementSolution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)O-supportWould prevent or help troubleshoot a customer escalation - bugs, missing observability/tooling, docsWould prevent or help troubleshoot a customer escalation - bugs, missing observability/tooling, docsP-3Issues/test failures with no fix SLAIssues/test failures with no fix SLAT-sql-queriesSQL Queries TeamSQL Queries Teamtarget-release-26.2.0v26.2.0-prerelease
Type
Projects
Status
Done