docs: Fluss example#1688
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds an end-to-end Fluss + Lakekeeper example: README, Docker Compose stack (Lakekeeper, migrations/init, RustFS, Fluss, Flink), a default warehouse JSON, and a Bash demo script that deploys, configures catalogs/tables, runs a tiering job, ingests sample data, and verifies via DuckDB. Changes
Sequence Diagram(s)sequenceDiagram
participant Dev as Developer
participant Compose as Docker Compose
participant Init as Init/Migrations
participant RustFS as RustFS/MinIO
participant Lakekeeper as Lakekeeper (REST)
participant Flink as Flink (JobManager)
participant Fluss as Fluss (Tiering Job)
participant DuckDB as DuckDB (verification)
Dev->>Compose: docker compose up -d
Compose->>Init: run migrations & bootstrap jobs
Init->>Lakekeeper: POST /management/v1/bootstrap, /warehouse
Init->>RustFS: create bucket `storage/fluss`
Compose->>Lakekeeper: start and report /health
Compose->>Flink: start JobManager & TaskManager (mount jars)
Compose->>Fluss: start coordinator & tablet (mount plugins)
Dev->>Flink: sql-client -> create catalog, db, table
Dev->>Fluss: submit Flink tiering job (detached)
Dev->>Flink: insert generated rows into `orders`
Fluss->>Lakekeeper: tiered writes -> Iceberg via REST catalog
Dev->>DuckDB: query Lakekeeper REST catalog
DuckDB->>Dev: return tiered data
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
examples/fluss/run-demo.sh (1)
30-38: Consider adding a brief wait or health check before starting the tiering job.The
-dflag correctly runs the Flink job in the background, but the subsequentsleep 5is a fixed delay. If the tiering job takes longer to initialize, the data insertion might happen before the job is ready to process it.This is likely fine for the demo since Fluss buffers data, but worth noting.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@examples/fluss/run-demo.sh` around lines 30 - 38, The script uses a fixed sleep after launching the tiering job (docker compose exec -d jobmanager ./bin/flink run ...) which can let data be inserted before the job is ready; replace the static sleep 5 with a readiness loop that polls the Flink job status or jobmanager logs until the tiering job is running (for example by repeatedly invoking a Flink CLI status command or checking jobmanager logs for the job name) and only proceed when the job reports RUNNING or a ready-ready log line; update run-demo.sh to perform this health check before continuing with data insertion so the tiering job is guaranteed initialized.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@examples/fluss/run-demo.sh`:
- Around line 66-76: The hard-coded network name "fluss_iceberg_net" in the run
loop can break when Docker Compose uses a different project name; update the
script to avoid a fixed network string by either exporting a project name or
building the network name dynamically: set COMPOSE_PROJECT_NAME=fluss near the
top before bringing up compose, or replace occurrences of "fluss_iceberg_net"
with "${COMPOSE_PROJECT_NAME:-fluss}_iceberg_net" so the docker run line uses
the computed network name when running the DuckDB command (the DUCKDB_QUERY
usage in the docker run remains unchanged).
---
Nitpick comments:
In `@examples/fluss/run-demo.sh`:
- Around line 30-38: The script uses a fixed sleep after launching the tiering
job (docker compose exec -d jobmanager ./bin/flink run ...) which can let data
be inserted before the job is ready; replace the static sleep 5 with a readiness
loop that polls the Flink job status or jobmanager logs until the tiering job is
running (for example by repeatedly invoking a Flink CLI status command or
checking jobmanager logs for the job name) and only proceed when the job reports
RUNNING or a ready-ready log line; update run-demo.sh to perform this health
check before continuing with data insertion so the tiering job is guaranteed
initialized.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: c9954593-e937-4628-8b20-a40a7e8a94f4
📒 Files selected for processing (4)
examples/fluss/README.mdexamples/fluss/create-default-warehouse.jsonexamples/fluss/docker-compose.yamlexamples/fluss/run-demo.sh
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@examples/fluss/run-demo.sh`:
- Around line 5-7: The docker compose exec call can fail if the jobmanager
container hasn't started; replace the single-line `docker compose exec -T
jobmanager bash -c 'until curl -sf http://lakekeeper:8181/health > /dev/null
2>&1; do sleep 1; done'` with a guarded retry: first wait for the jobmanager
container to be reported as running (e.g., loop on `docker compose ps`/`docker
compose ps --filter status=running` and grep for jobmanager) and only then run
the curl loop with `docker compose exec`; implement this by adding an outer
`until` loop that checks container running status (or `docker compose ps -q
jobmanager` + inspect) and sleeps/retries, then run the existing inner
curl-until check once the container is up.
- Around line 83-85: The current parsing grabs the last numeric token from
docker/duckdb output (variables output/count), which can pull numbers from logs
or errors; change the pipeline to capture only the query result line and
validate it's purely numeric. Specifically, run the docker compose command the
same way (preserve 2>&1) into output, then replace the count extraction with
something like: extract the last non-empty line (e.g., using tail -n1) and
require it matches ^[0-9]+$ (e.g., via grep -E '^[0-9]+$') before assigning to
count so only a deterministic numeric query result from DUCKDB_QUERY is used in
the if check. Ensure you still redirect errors (2>&1) and keep the if condition
checking count as a numeric value.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 8b7a8409-9578-49bf-84dc-c43e144ae046
📒 Files selected for processing (3)
examples/fluss/create-default-warehouse.jsonexamples/fluss/docker-compose.yamlexamples/fluss/run-demo.sh
✅ Files skipped from review due to trivial changes (2)
- examples/fluss/create-default-warehouse.json
- examples/fluss/docker-compose.yaml
c-thiel
left a comment
There was a problem hiding this comment.
LGTM in general, just one minor remark. I will give it a more thorough look the coming days.
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@examples/fluss/run-demo.sh`:
- Around line 32-39: The detached Flink tiering start (docker compose exec ...
./bin/flink run ...) and the fixed sleeps (sleep 5 and the later sleep ~30) are
fragile; replace them with readiness checks that poll the Iceberg table until
data appears or a timeout elapses. After launching the job (the command starting
with "docker compose exec -d jobmanager ./bin/flink run" in run-demo.sh)
implement a loop that queries the Iceberg table (the same check used later
before DuckDB) at intervals, with a configurable timeout, and exit non-zero if
no rows are returned within the timeout; remove the fixed sleep lines and the
later blind 30s pause and use the same polling routine before running the DuckDB
query so the script deterministically fails on tiering timeout.
- Around line 6-7: Replace the unbounded readiness loops for "until docker
compose exec -T jobmanager true" and the inner "docker compose exec -T
jobmanager bash -c 'until curl -sf http://lakekeeper:8181/health ...'" with
bounded loops that fail after 120 seconds; implement by recording a start time,
polling until success or until elapsed > 120, and on timeout printing a clear
error message (e.g., "jobmanager did not become ready within 120s" / "lakekeeper
healthcheck failed within 120s") and exiting with non‑zero status; ensure both
readiness checks follow the same pattern so CI/demos stop deterministically on
failure.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 4c39f40d-ed12-47cf-9d8c-8382d347a29f
📒 Files selected for processing (2)
examples/fluss/docker-compose.yamlexamples/fluss/run-demo.sh
🚧 Files skipped from review as they are similar to previous changes (1)
- examples/fluss/docker-compose.yaml
There was a problem hiding this comment.
🧹 Nitpick comments (2)
examples/fluss/run-demo.sh (2)
89-105: Data availability check could be more deterministic.The
grep -q "0 rows"check relies on DuckDB's default output format. If the output format changes or an error message doesn't include this pattern, the script may behave unexpectedly.A more robust approach would use structured output:
♻️ Optional: Use COUNT(*) with structured output
+COUNT_QUERY=" +INSTALL iceberg; LOAD iceberg; INSTALL httpfs; LOAD httpfs; +CREATE SECRET (TYPE s3, KEY_ID 'rustfs-root-user', SECRET 'rustfs-root-password', + ENDPOINT 'localtest.me:9000', USE_SSL false, URL_STYLE 'path'); +CREATE SECRET (TYPE ICEBERG, ENDPOINT 'http://lakekeeper:8181/catalog', TOKEN 'dummy'); +ATTACH 'fluss-warehouse' AS lk (TYPE ICEBERG); +SELECT COUNT(*) FROM lk.demo.orders; +" + echo "--- Waiting for data to be tiered to Iceberg ---" SECONDS=0 while [ $SECONDS -lt 120 ]; do - output=$(docker compose run --rm -T duckdb duckdb -c "$DUCKDB_QUERY" 2>&1) - if echo "$output" | grep -q "0 rows"; then + count=$(docker compose run --rm -T duckdb duckdb -csv -noheader -c "$COUNT_QUERY" 2>&1 | tail -n1 | tr -d '\r') + if [[ "$count" =~ ^[0-9]+$ ]] && (( count > 0 )); then + echo "--- DuckDB query result ---" + docker compose run --rm -T duckdb duckdb -c "$DUCKDB_QUERY" + # ... rest of success output + exit 0 + fi - sleep 5 - continue - fi + sleep 5However, for demo purposes, the current approach is functional and the bounded timeout ensures deterministic failure.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@examples/fluss/run-demo.sh` around lines 89 - 105, The current loop checks DuckDB output text for the string "0 rows" (see the docker compose run --rm -T duckdb duckdb -c "$DUCKDB_QUERY" invocation and the grep -q "0 rows" test), which is brittle; instead run a deterministic COUNT(*) query and parse the numeric count to decide readiness. Update the command that sets output to execute a SELECT COUNT(*) AS cnt FROM (...) (or wrap $DUCKDB_QUERY) and then parse the returned number (e.g., with awk/sed) rather than grepping for "0 rows", and use that numeric value in the while loop to break when count > 0; keep the SECONDS timeout logic unchanged.
5-12: Consider avoidingevalfor slightly safer command execution.The static analysis flags
eval "$@"(SC2294). While this is acceptable in a controlled demo script where all commands are hardcoded, passing commands as function arguments and using"$@"directly (withouteval) would be safer if the commands didn't require shell parsing (pipes, redirections).Since the commands here include shell constructs like
grep -qandcurl ... > /dev/null,evalis necessary. This is fine for demo purposes.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@examples/fluss/run-demo.sh` around lines 5 - 12, The wait_for function currently uses eval "$@" which static analysis flags; change wait_for to avoid eval by either (a) accepting the command as separate args and executing them directly with "$@" (update callers to pass commands/components without shell metacharacters), or (b) if callers need shell parsing (pipes/redirections), accept a single command string and run it explicitly with bash -c "$cmd" to make the intent explicit; update the wait_for function signature and its callers accordingly (refer to the wait_for function and all places invoking it).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@examples/fluss/run-demo.sh`:
- Around line 89-105: The current loop checks DuckDB output text for the string
"0 rows" (see the docker compose run --rm -T duckdb duckdb -c "$DUCKDB_QUERY"
invocation and the grep -q "0 rows" test), which is brittle; instead run a
deterministic COUNT(*) query and parse the numeric count to decide readiness.
Update the command that sets output to execute a SELECT COUNT(*) AS cnt FROM
(...) (or wrap $DUCKDB_QUERY) and then parse the returned number (e.g., with
awk/sed) rather than grepping for "0 rows", and use that numeric value in the
while loop to break when count > 0; keep the SECONDS timeout logic unchanged.
- Around line 5-12: The wait_for function currently uses eval "$@" which static
analysis flags; change wait_for to avoid eval by either (a) accepting the
command as separate args and executing them directly with "$@" (update callers
to pass commands/components without shell metacharacters), or (b) if callers
need shell parsing (pipes/redirections), accept a single command string and run
it explicitly with bash -c "$cmd" to make the intent explicit; update the
wait_for function signature and its callers accordingly (refer to the wait_for
function and all places invoking it).
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 1680c0d7-428c-4977-b82a-65b31b0e20a0
📒 Files selected for processing (1)
examples/fluss/run-demo.sh
96418ce to
a3f7367
Compare
|
@fresh-borzoni Thanks again for your PR and sorry for taking a bit of time for the review. LGTM :) I just pushed some small additions to the example doc and your script. I also added a small section to the lakekeeper docs describing your example. Should be ready to be merged when checks are done. |
✅ PR Title Formatted CorrectlyThe title of this PR match the correct format. Thank you! |
Fluss streaming data tiered to Iceberg through Lakekeeper.
Uses RustFS instead of MinIO
run-demo.shdoes the full flow: create table, insert, tier, query with DuckDB via REST catalog.Summary by CodeRabbit