Skip to content

Lite: Display queries should include archived parquet data #160

@erikdarlingdata

Description

@erikdarlingdata

Problem

ArchiveService exports rows older than 7 days to parquet files and deletes them from DuckDB hot tables. RetentionService keeps parquet files for 90 days. But none of the 17 LocalDataService display queries read from parquet — they only query DuckDB tables directly.

This means archived data is invisible to the UI. Users can only see the last 7 days of data in any chart or grid, despite having up to 90 days of archived data on disk.

Expected behavior

Display queries should UNION hot DuckDB tables with read_parquet() glob scans of the archive directory, so users can query the full retention window (up to 90 days).

Suggested approach

Create DuckDB views per table that UNION hot data with archived parquet:

CREATE OR REPLACE VIEW v_wait_stats AS
SELECT * FROM wait_stats
UNION ALL
SELECT * FROM read_parquet('archive/*_wait_stats.parquet')

Update all LocalDataService.*.cs files to query views instead of tables. Handle the case where no parquet files exist yet for a given table.

Files involved

  • Lite/Database/DuckDbInitializer.cs — create/refresh views
  • Lite/Services/LocalDataService.*.cs (17 files) — change table references to views
  • Lite/Services/ArchiveService.cs — trigger view refresh after archival

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions