Add flush task #534
Add flush task #534
Conversation
| Duration::from_secs(context.config.misc.sync_conf.write_lock_timeout_s); | ||
| let sync_writer = Arc::new(RwLock::new(SeafowlDataSyncWriter::new(context.clone()))); | ||
| let handler = SeafowlFlightHandler::new(context, sync_writer.clone()); | ||
| tokio::spawn(flush_task(flush_interval, lock_timeout, sync_writer)); |
There was a problem hiding this comment.
Do we need to worry about properly terminating this when Seafowl ends / restarting if it crashes / should this use the subsystem setup from the top level that Seafowl uses to run the GC task and other frontends?
There was a problem hiding this comment.
Perhaps? I'd keep it as is for now and potentially add it as a subsystem later on (needs additional work for integration tests setup).
| batches, | ||
| )?; | ||
|
|
||
| sync_writer.flush().await?; |
There was a problem hiding this comment.
It's still the case that this flush() op won't return until we've had a chance to see if we need to flush and wait for the flush to finish, right? (i.e. Seafowl will never end up using more memory than the limit)
There was a problem hiding this comment.
The flush call there checks the criteria and while it is met it flushes table after table. The memory usage can only go down inside/after the call.
Try to flush on a regular interval, so as not to rely on sync calls to do the pushing.
Also:
enqueue_syncnon-async by decoupling table creation and flushing from itflushmethod is to be used by the handler