-
-
Notifications
You must be signed in to change notification settings - Fork 202
Feat: Batch Streaming & Progressive Rendering for Proxy Loader #9094
Copy link
Copy link
Closed
Labels
aicoreCore framework functionalityCore framework functionalityperformancePerformance improvements and optimizationsPerformance improvements and optimizations
Description
Feat: Batch Streaming & Progressive Rendering for Proxy Loader
Objective
Optimize the Neo.data.proxy.Stream and Neo.data.Store integration to solve the performance regression caused by single-record event firing. Implement chunked streaming and progressive UI updates.
Problem
The initial implementation fired a data event for every single record in the NDJSON stream. For 11k records, this caused massive overhead (11k events, 11k store updates, 11k microtasks), blocking the App Worker for ~10s.
Solution
- Batching (
chunkSize): UpdateStream.mjsto buffer parsed records and fire thedataevent only whenchunkSize(e.g., 500) is reached. - Progressive Rendering: Update
Store.mjsto removesuspendEventsduring the stream. Instead, listen for the chunkeddataevents, add the chunk to the store, and fire aloadevent immediately. This allows the Grid to render the first chunk (~500 rows) almost instantly while the rest of the stream continues in the background.
Tasks
- Add
chunkSizeconfig toNeo.data.proxy.Stream. - Implement buffering logic in
Stream.read()to yield arrays of records. - Refactor
Neo.data.Store.load()to support progressive loading (removesuspendEvents, fire intermediateloadevents). - Update Unit Tests (
Stream.spec.mjs,StoreProxy.spec.mjs).
Expected Outcome
- Time to First Byte (TTFB): Unchanged.
- Time to First Render (TTFR): Drastically reduced (O(chunkSize) instead of O(Total)).
- UX: User sees the grid populate immediately. Scrollbar grows as data streams in.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
aicoreCore framework functionalityCore framework functionalityperformancePerformance improvements and optimizationsPerformance improvements and optimizations