Streaming queries model with cursors#63312
Streaming queries model with cursors#63312Michicosun wants to merge 236 commits intoClickHouse:masterfrom
Conversation
…/42990/simple_streaming
…/42990/simple_streaming
…/42990/simple_streaming
…/42990/simple_streaming
…/42990/block_row_column
|
Test |
|
I'm quite interested in this implementation, is this still being driven in any way? |
|
@Michicosun Really great work! Will you implement dataflow model https://www.oreilly.com/radar/the-world-beyond-batch-streaming-101, like time window, watermark, triggers, etc. ? |
|
Hi @zhanglistar , if you are interested in watermark, out of order processing, etc, you may check our stream processing engine built with ClickHouse codebase https://github.com/timeplus-io/proton We are covered in another O'Reilly book https://www.oreilly.com/library/view/streaming-databases/9781098154820/ It's not exactly dataflow but can do many things similar to Apache Flink, but in C++ speed and low footprint. Happy to work with ClickHouse community to work on stream processing together. |
@jovezhong Thanks for your reply. I have seen proton, great job! And there is another streamming processing framework like arroyo, risingwave writen in Rust. I am studying these things, and also, we may impletement native flink task like https://developer.aliyun.com/article/1634363 in gluten project https://github.com/apache/incubator-gluten, and we already have a POC, but there still lots of work to do, like stateful functions. |
|
Could this hypothetically interact with or build on top of #79417 and #79471? Those PRs combined seem to support a stable low-level notion of a cursor ( Just wondering, because I have no real knowledge here but the above PRs seem to be useful for this feature. |
|
Is this still planned for 2025? |
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
This pull request is a combination of
with implementation of cursors on top of that.
Implemented:
queue_mode=1)queue_mode=1) same as for regular MergeTree butSELECT * FROM dist STREAM CURSOR {'0.{dst_cursor}'}will be rewritten to query:SELECT * FROM local STREAM CURSOR dst_cursorextension for sql:
Documentation entry for user-facing changes
Modify your CI run
NOTE: If your merge the PR with modified CI you MUST KNOW what you are doing
NOTE: Checked options will be applied if set before CI RunConfig/PrepareRunConfig step
Include tests (required builds will be added automatically):
Exclude tests:
Extra options:
Only specified batches in multi-batch jobs:
Details