This is the main tracking issue for all persisted SQL Stats related work:

Infrastructure:

• system table schema:
  • sql: create system tables for SQL stats #65374
  • sql: store plan as jsonb in system.statement_statistics #67331
  • sql: add transaction_fingerprint_id to system.statement_statistics pk #69320
  • sql: change plan_hash in system.statement_statistics from Int to Bytes #69502
• JSON encoding for stats objects:
  • sql: add json encoding/decoding support for stmt/txn stats. #67805
• flush mechanism:
  • sql: periodically flush sqlstats #67090
• a feature flag + version gate :
  • sql: periodically flush sqlstats #67090
  • sql: proper version gate sql stats #69592
• Clean up old stats using job
  • sql,jobs: create SQL Stats Compaction Job and resumer #68434
• Runs the clean up job on a schedule
  • sql: create sql stats compaction scheduled job #68401
  • sql: fix incorrect follower read and testing knob for sql stats compaction job #69346
• Running benchmark with new persisted SQL stats turned on to measure the performance impact.
  • Initial benchmark shows ~2ms increase in average latency
  • More benchmark on SQL Stats flush with different configurations.
  • testing flushing behaviour with injected errors in an overloaded cluster. (chaos testing)

Minimum Requirement to enable it by default by 21.2

• Serve both cluster-wide in-memory stats with persisted SQL stats through a virtual table
  • sql: support decode ExplainTreePlanNode from JSON format #68555
  • sql: reimplement sqlstats API using iterator #68620
  • sql: implement IterateStatementStats() for PersistedSQLStats #68675
  • sql: introduce crdb_internal.statement_statistics virtual table #68715
  • sql: implement combined iterator for transaction statistics #69022
  • sql: implement crdb_internal.transaction_statistics #69049
  • sql: fix in-memory stmt stats not properly updating #69441
• Update status RPC/HTTP endpoint to be able to be queried by aggregation interval.
  • server: create api to query persisted stats by date range #69238
  • sql: deflake TestStatusAPIStatements and TestStatusAPICombinedStatements #69591
• Update the DB Console Statement/Transaction Page to include a time picker to select aggregation interval.
  • ui: add date range selector in stmts and txns pages #68831
• Extend Reset SQL Stats to support resetting on-disk stats.
  • sql: crdb_internal.reset_sql_stats() now resets persisted SQL Stats  #69273

Further Enhancement

• sql: implement sql stats compaction for clean up job #69302
• sql: better eviction policy for sql stats in-memory store #69303
• sql: batch SQL stats flush #67867
• server: add memory accouting for Statements gRPC endpoint #69032
• sql: implement virtual index on crdb_internal.{statement, transaction}_statistics #69321
• sql: surface sql stats compaction job details in SQL CLI #69429
• Query plan changes can be easily detected through our internal tables and the SQL CLI #67500
  • Depends on sql: add ability to get a hash value from a logical plan #63885

Clean Up

• server: Consolidate serverpb.CollectedStatementStatistics #69326
• ui: Transaction page inconsistent grouping of statement stats #59205
• sql: use persisted sql stats for telemetry reporting #66047

RFC PR: #63752

Epic: CRDB-9866

Jira issue: CRDB-7226