-
Notifications
You must be signed in to change notification settings - Fork 26
Description
Problem
ArchiveService exports rows older than 7 days to parquet files and deletes them from DuckDB hot tables. RetentionService keeps parquet files for 90 days. But none of the 17 LocalDataService display queries read from parquet — they only query DuckDB tables directly.
This means archived data is invisible to the UI. Users can only see the last 7 days of data in any chart or grid, despite having up to 90 days of archived data on disk.
Expected behavior
Display queries should UNION hot DuckDB tables with read_parquet() glob scans of the archive directory, so users can query the full retention window (up to 90 days).
Suggested approach
Create DuckDB views per table that UNION hot data with archived parquet:
CREATE OR REPLACE VIEW v_wait_stats AS
SELECT * FROM wait_stats
UNION ALL
SELECT * FROM read_parquet('archive/*_wait_stats.parquet')Update all LocalDataService.*.cs files to query views instead of tables. Handle the case where no parquet files exist yet for a given table.
Files involved
Lite/Database/DuckDbInitializer.cs— create/refresh viewsLite/Services/LocalDataService.*.cs(17 files) — change table references to viewsLite/Services/ArchiveService.cs— trigger view refresh after archival