Skip to content

[Task] docs: Document stream processing and aggregation framework #459

Description

@kcenon

Summary

Document the stream processing framework including stream aggregator, aggregation processor, time series management, and ring buffer for real-time metric aggregation.

Parent Issue

Part of: [EPIC] docs: Address documentation gaps across all ecosystem systems (kcenon/common_system#325)

Background (Why)

monitoring_system includes a stream processing subsystem at include/kcenon/monitoring/utils/ that enables real-time metric aggregation, but it has no dedicated documentation. Users cannot discover or configure windowed aggregation, time series buffering, or statistical processing.

Source files:

  • include/kcenon/monitoring/utils/stream_aggregator.h — Real-time stream aggregation
  • include/kcenon/monitoring/utils/aggregation_processor.h — Aggregation pipeline
  • include/kcenon/monitoring/utils/time_series.h — Time series data management
  • include/kcenon/monitoring/utils/time_series_buffer.h — Buffered time series
  • include/kcenon/monitoring/utils/statistics.h — Statistical computation
  • include/kcenon/monitoring/utils/ring_buffer.h — Fixed-size circular buffer
  • include/kcenon/monitoring/utils/metric_storage.h — Metric persistence

Scope (What)

Create docs/guides/STREAM_PROCESSING.md covering:

1. Stream Processing Architecture

  • Data pipeline: raw metrics → aggregation → storage/export
  • Windowing strategies (tumbling, sliding, session)
  • Buffer management and memory control

2. Stream Aggregator

  • Real-time aggregation operations (sum, avg, min, max, percentiles)
  • Window configuration
  • Flush triggers and intervals

3. Time Series Management

  • Time series data model
  • Retention and downsampling
  • Query patterns

4. Ring Buffer

  • Fixed-size circular buffer for bounded memory
  • Thread safety guarantees
  • Configuration and sizing

5. Statistics Engine

  • Available statistical functions
  • Online vs batch computation
  • Accuracy guarantees

6. Usage Examples

  • Real-time request latency aggregation
  • System resource trend analysis
  • Custom aggregation pipeline

Acceptance Criteria

  • All 7 stream processing components documented
  • Windowing strategies explained
  • Memory management patterns documented
  • At least 3 usage examples

Metadata

Metadata

Assignees

Labels

area/metricsMetric collection and processingdocumentationImprovements or additions to documentationpriority/mediumMedium priority - Important but not urgent

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions