Vtgate metrics: Added plan and query type metrics by harshit-gangal · Pull Request #17727 · vitessio/vitess

harshit-gangal · 2025-02-10T04:22:15Z

Description

This PR adds classification of query plans based on their execution strategy.
Additionally, it deprecates four metrics and introduces three replacements.

The Queries metric counter, which tracks queries processed at VTGate, will be published with Query, Plan, and Tablet type labels. This improves observability into query distribution and execution patterns.

Plan Classification

Query plans are categorized into different types based on their execution behavior:

Local – Queries executed locally on VTGate without involving any shard.
Passthrough – Queries forwarded to single shard without having any additional processing at VTGate.
MultiShard – Queries executed across multiple shards with controlled routing.
Lookup – Queries using lookup vindexes to resolve keyspace IDs efficiently and route to specific shards.
Scatter – Queries broadcast to all shards.
JoinOp – Queries involving join operations across multiple shards, with Join on VTGate.
ForeignKey – Queries handling foreign key constraints, such as cascades and validations.
Complex – Queries with intricate execution logic requiring VTGate to process results.
OnlineDDL – DDLs executed though Online Schema change workflow.
DirectDDL – DDLs directly executed on the shards.
Transaction – Queries managing transactions, including BEGIN, COMMIT, ROLLBACK, and SAVEPOINT.
Topology - Queries that involves accessing Topology Server for Get/Put.

Published Metrics

Sample metrics from VTGate's /metrics path:

# HELP vtgate_query_executions Counts queries executed at VTGate by query type, plan type, and tablet type.
# TYPE vtgate_query_executions counter
vtgate_query_executions{plan="Complex",query="SELECT",tablet="PRIMARY"} 1
vtgate_query_executions{plan="Join",query="SELECT",tablet="PRIMARY"} 1
vtgate_query_executions{plan="Local",query="SET",tablet="PRIMARY"} 2
vtgate_query_executions{plan="Local",query="USE",tablet="PRIMARY"} 1
vtgate_query_executions{plan="Lookup",query="SELECT",tablet="PRIMARY"} 1
vtgate_query_executions{plan="MultiShard",query="DELETE",tablet="PRIMARY"} 1
vtgate_query_executions{plan="MultiShard",query="SET",tablet="PRIMARY"} 1
vtgate_query_executions{plan="MultiShard",query="UPDATE",tablet="PRIMARY"} 1
vtgate_query_executions{plan="Passthrough",query="SHOW",tablet="PRIMARY"} 1
vtgate_query_executions{plan="Scatter",query="DELETE",tablet="PRIMARY"} 20
vtgate_query_executions{plan="Scatter",query="SELECT",tablet="PRIMARY"} 1
vtgate_query_executions{plan="Topology",query="SET",tablet="PRIMARY"} 1
vtgate_query_executions{plan="Transaction",query="ROLLBACK",tablet="PRIMARY"} 1
vtgate_query_executions{plan="Transaction",query="SAVEPOINT",tablet="PRIMARY"} 1


# HELP vtgate_query_routes Counts queries routed from VTGate to VTTablet by query type, plan type, and tablet type.
# TYPE vtgate_query_routes counter
vtgate_query_routes{plan="Complex",query="SELECT",tablet="PRIMARY"} 6
vtgate_query_routes{plan="Join",query="SELECT",tablet="PRIMARY"} 3
vtgate_query_routes{plan="Local",query="SET",tablet="PRIMARY"} 0
vtgate_query_routes{plan="Local",query="USE",tablet="PRIMARY"} 0
vtgate_query_routes{plan="Lookup",query="SELECT",tablet="PRIMARY"} 2
vtgate_query_routes{plan="MultiShard",query="DELETE",tablet="PRIMARY"} 2
vtgate_query_routes{plan="MultiShard",query="SET",tablet="PRIMARY"} 1
vtgate_query_routes{plan="MultiShard",query="UPDATE",tablet="PRIMARY"} 2
vtgate_query_routes{plan="Passthrough",query="SHOW",tablet="PRIMARY"} 1
vtgate_query_routes{plan="Scatter",query="DELETE",tablet="PRIMARY"} 56
vtgate_query_routes{plan="Scatter",query="SELECT",tablet="PRIMARY"} 2
vtgate_query_routes{plan="Topology",query="SET",tablet="PRIMARY"} 0
vtgate_query_routes{plan="Transaction",query="ROLLBACK",tablet="PRIMARY"} 0
vtgate_query_routes{plan="Transaction",query="SAVEPOINT",tablet="PRIMARY"} 0


# HELP vtgate_query_executions_by_table Counts queries executed at VTGate per table by query type and table.
# TYPE vtgate_query_executions_by_table counter
vtgate_query_executions_by_table{query="DELETE",table="ks_t1"} 2
vtgate_query_executions_by_table{query="DELETE",table="ks_t2"} 1
vtgate_query_executions_by_table{query="DELETE",table="ks_t3"} 1
vtgate_query_executions_by_table{query="DELETE",table="ks_t4"} 1
vtgate_query_executions_by_table{query="SELECT",table="ks_t1"} 4
vtgate_query_executions_by_table{query="SELECT",table="ks_t2"} 1
vtgate_query_executions_by_table{query="SELECT",table="ks_t3"} 1
vtgate_query_executions_by_table{query="SELECT",table="ks_t4"} 1
vtgate_query_executions_by_table{query="UPDATE",table="ks_t1"} 1

Related Issue(s)

RFC: New vtgate Metrics #17585

Checklist

Tests were added or are not required
Did the new or modified tests pass consistently locally and on CI?
Documentation was added or is not required Doc: Query Plans Classification and Monitoring website#1935 update metric name "Queries" to "QueryExecutions" website#1949

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

vitess-bot · 2025-02-10T04:22:17Z

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

codecov · 2025-02-10T06:20:23Z

Codecov Report

Attention: Patch coverage is 94.82759% with 12 lines in your changes missing coverage. Please review.

Project coverage is 67.52%. Comparing base (b05df12) to head (9f80097).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
go/vt/vtgate/engine/plan.go	93.02%	12 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #17727      +/-   ##
==========================================
+ Coverage   67.45%   67.52%   +0.06%     
==========================================
  Files        1594     1594              
  Lines      259064   259472     +408     
==========================================
+ Hits       174760   175211     +451     
+ Misses      84304    84261      -43

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

…st for metrics Signed-off-by: Harshit Gangal <harshit@planetscale.com>

…nd using lookup vindex as lookup plan type Signed-off-by: Harshit Gangal <harshit@planetscale.com>

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

harshit-gangal · 2025-02-11T15:53:58Z

Website Doc PR: vitessio/website#1935

deepthi · 2025-02-12T23:19:12Z

I understand that we can't re-use QueriesProcessed and QueriesRouted, because that might break people's dashboards, but we need to come up with some other names for QueryProcessed and QueryRouted which are grammatical. Leaving this comment as a place-holder while I try to think of alternatives.

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

Signed-off-by: Andres Taylor <andres@planetscale.com>

deepthi · 2025-02-18T16:53:54Z

I understand that we can't re-use QueriesProcessed and QueriesRouted, because that might break people's dashboards, but we need to come up with some other names for QueryProcessed and QueryRouted which are grammatical. Leaving this comment as a place-holder while I try to think of alternatives.

Here are my suggestions

QueryProcessed -> QueryCounts
QueryRouted -> QueryRouteCounts

harshit-gangal · 2025-02-18T17:17:45Z

QueryProcessed -> QueryCounts

QueryRouted -> QueryRouteCounts

Count is self-explanatory, as it already implies a counter. Using Counts looks redundant in this context.

frouioui · 2025-02-18T22:07:31Z

go/vt/vtgate/engine/plan.go

+const (
+	PlanUnknown PlanType = iota
+	PlanLocal
+	PlanPassthrough
+	PlanMultiShard
+	PlanLookup
+	PlanScatter
+	PlanJoinOp
+	PlanForeignKey
+	PlanComplex
+	PlanOnlineDDL
+	PlanDirectDDL
+	PlanTransaction
+	PlanTopoOp
+)


It could be good to have a link to the docs here as a comment (https://github.com/vitessio/website/pull/1935/files#diff-4badcfb66b7411dea206a299af9b9f8e3d349d817ad332798f747b0b9a40ad9f), both to help us know what each value mean and also to remind us that modifying this list implies that we need to change the docs.

frouioui

Looks good to me, just QueryRouted -> "query" in singular feels a bit odd. But I don't have better suggestion for the name. Not approving as deepthi has an ongoing conversation.

GuptaManan100

LGTM to me too

deepthi · 2025-02-21T18:10:08Z

QueryProcessed -> QueryCounts

QueryRouted -> QueryRouteCounts

Count is self-explanatory, as it already implies a counter. Using Counts looks redundant in this context.

The problem is that we don't have any good options because QueriesProcessed is taken. I proposed Counts because we already have that on vttablet, so there is precedence for it.
If you don't like that, then the next proposal is simply "Queries" and "QueryRoutes" :)

deepthi · 2025-02-21T18:11:44Z

go/vt/vtgate/executor.go

+	// TODO: @harshit/@systay - Remove these deprecated stats once we have a replacement in a released version.
+	queriesProcessed        = stats.NewCountersWithSingleLabel("QueriesProcessed", "Deprecated: Queries processed at vtgate by plan type", "Plan")
+	queriesRouted           = stats.NewCountersWithSingleLabel("QueriesRouted", "Deprecated: Queries routed from vtgate to vttablet by plan type", "Plan")
+	queriesProcessedByTable = stats.NewCountersWithMultiLabels("QueriesProcessedByTable", "Deprecated: Queries processed at vtgate by plan type, keyspace and table", []string{"Plan", "Keyspace", "Table"})


We seem to be dropping the table metrics. Can we add that as a dimension to the new metrics? They have been useful in the past in debugging production issues.

I have added QueryTables as replacement metric for it.

deepthi · 2025-02-21T18:12:44Z

changelog/22.0/22.0.0/summary.md

+2. QueryRouted - This counts the number of shards the query was executed on.
+
+Deprecated:
+1. QueriesProcessed


In code you are deprecating the table metrics, but that is missing from here.

I have added QueryTables as replacement metric and updated the summary notes.

deepthi · 2025-02-21T18:13:39Z

changelog/22.0/22.0.0/summary.md

+### <a id="vtgate-metrics"/>VTGate Metrics
+
+Added two new metrics for queries with query type, plan type and tablet type as dimension.
+1. QueryProcessed - This counts the number of query executed.


Suggested change

1. QueryProcessed - This counts the number of query executed.

1. QueryProcessed - This counts the number of queries processed by this vtgate.

I am not sure if adding this VTGate helps.
Any time-series database will accumulate data from all VTGates and show a single view.
However, users can add instance as a dimension to view it on a per-VTGate basis.
The metrics themselves are collected at a single-process level.

deepthi · 2025-02-21T18:15:31Z

changelog/22.0/22.0.0/summary.md

+
+Added two new metrics for queries with query type, plan type and tablet type as dimension.
+1. QueryProcessed - This counts the number of query executed.
+2. QueryRouted - This counts the number of shards the query was executed on.


This says shards but the actual description of the metric says "routed to vttablets". Which one is it?

For this metric, ‘shards’ and ‘vttablets’ are effectively similar in meaning. A query is routed to either the primary or a replica of a shard, but not both.

I have updated it.

go/vt/vtgate/executor.go

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

harshit-gangal added 2 commits February 6, 2025 16:13

feat: added plan type

db6115c

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

added more to plan type classification

c3427b6

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

github-actions bot added this to the v22.0.0 milestone Feb 10, 2025

harshit-gangal added 2 commits February 10, 2025 11:17

grouped migration statements

d2062ea

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

fix and update ddl test with plan type

cfc2806

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

feat: added query processed metric by query, plan and tablet type

128ab90

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

harshit-gangal added 2 commits February 10, 2025 16:43

sizegen update

fc8d595

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

updated show plan type and transaction query plan types, added e2e te…

c7ec2af

…st for metrics Signed-off-by: Harshit Gangal <harshit@planetscale.com>

harshit-gangal removed the NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work label Feb 11, 2025

harshit-gangal marked this pull request as ready for review February 11, 2025 08:20

harshit-gangal requested review from GuptaManan100, arthurschreiber, frouioui and systay as code owners February 11, 2025 08:20

harshit-gangal added 2 commits February 11, 2025 14:55

update and delete having vindex changes be classified as multishard a…

329f5be

…nd using lookup vindex as lookup plan type Signed-off-by: Harshit Gangal <harshit@planetscale.com>

added queryrouted metrics and deprecated similar old metrics

edc12c3

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

harshit-gangal requested a review from deepthi as a code owner February 11, 2025 09:35

harshit-gangal removed the NeedsWebsiteDocsUpdate What it says label Feb 11, 2025

harshit-gangal and others added 5 commits February 14, 2025 11:50

remove onecase

de31a7e

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

Merge remote-tracking branch 'upstream/main' into vtgate-metrics

686106c

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

fix test expectation

478c59d

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

feat: limit op doesn't count - use the input for category

aa4c4c7

Signed-off-by: Andres Taylor <andres@planetscale.com>

add complex example

6c55d04

Signed-off-by: Andres Taylor <andres@planetscale.com>

systay approved these changes Feb 17, 2025

View reviewed changes

frouioui reviewed Feb 18, 2025

View reviewed changes

GuptaManan100 reviewed Feb 19, 2025

View reviewed changes

deepthi reviewed Feb 21, 2025

View reviewed changes

harshit-gangal added 3 commits February 28, 2025 13:47

added by table metrics and addressed review comments

e9acec1

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

Merge remote-tracking branch 'upstream/main' into vtgate-metrics

a2b532a

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

test: update test

72c561a

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

harshit-gangal force-pushed the vtgate-metrics branch from 60bebfc to 72c561a Compare February 28, 2025 11:42

update metric name 'Queries' -> 'QueryExecutions'

30a5c69

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

harshit-gangal force-pushed the vtgate-metrics branch from dca525e to 30a5c69 Compare February 28, 2025 12:23

harshit-gangal added 2 commits February 28, 2025 17:55

rollback test change

ca5565c

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

queryTables -> queryExecutionsByTable metric name change

9f80097

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

deepthi approved these changes Feb 28, 2025

View reviewed changes

harshit-gangal merged commit f8bc9fa into vitessio:main Feb 28, 2025
103 checks passed

harshit-gangal deleted the vtgate-metrics branch February 28, 2025 17:30

harshit-gangal mentioned this pull request Mar 4, 2025

RFC: New vtgate Metrics #17585

Closed

BrewTestBot mentioned this pull request Apr 29, 2025

vitess 22.0.0 Homebrew/homebrew-core#221874

Merged

BrewTestBot mentioned this pull request Nov 4, 2025

vitess 23.0.0 Homebrew/homebrew-core#252716

Merged

clubanderson mentioned this pull request Mar 13, 2026

🌱 vitess: RFC: New vtgate Metrics kubestellar/console-kb#898

Closed

	1. QueryProcessed - This counts the number of query executed.
	1. QueryProcessed - This counts the number of queries processed by this vtgate.

Conversation

harshit-gangal commented Feb 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Plan Classification

Published Metrics

Related Issue(s)

Checklist

Uh oh!

vitess-bot bot commented Feb 10, 2025

Review Checklist

General

Tests

Documentation

New flags

If a workflow is added or modified:

Backward compatibility

Uh oh!

codecov bot commented Feb 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

harshit-gangal commented Feb 11, 2025

Uh oh!

deepthi commented Feb 12, 2025

Uh oh!

deepthi commented Feb 18, 2025

Uh oh!

harshit-gangal commented Feb 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

frouioui left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GuptaManan100 left a comment

Choose a reason for hiding this comment

Uh oh!

deepthi commented Feb 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harshit-gangal Feb 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harshit-gangal Feb 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

harshit-gangal commented Feb 10, 2025 •

edited

Loading

codecov bot commented Feb 10, 2025 •

edited

Loading

frouioui left a comment •

edited

Loading

harshit-gangal Feb 28, 2025 •

edited

Loading

harshit-gangal Feb 28, 2025 •

edited

Loading