Skip to content

docs: Fluss example#1688

Merged
brgr-s merged 8 commits into
lakekeeper:mainfrom
fresh-borzoni:fluss-example
Apr 14, 2026
Merged

docs: Fluss example#1688
brgr-s merged 8 commits into
lakekeeper:mainfrom
fresh-borzoni:fluss-example

Conversation

@fresh-borzoni

@fresh-borzoni fresh-borzoni commented Apr 1, 2026

Copy link
Copy Markdown
Contributor

Fluss streaming data tiered to Iceberg through Lakekeeper.
Uses RustFS instead of MinIO

run-demo.sh does the full flow: create table, insert, tier, query with DuckDB via REST catalog.

Summary by CodeRabbit

  • Documentation
    • Added a comprehensive Fluss + Lakekeeper integration guide with quick-start and manual steps for setting up catalogs, databases, datalake-enabled tables, and running tiering jobs.
  • Examples
    • Added a runnable local demo stack, a sample S3/MinIO warehouse configuration, and an automated demo script to run ingestion, start tiering, and verify Iceberg queryability.

@cla-assistant

cla-assistant Bot commented Apr 1, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@coderabbitai

coderabbitai Bot commented Apr 1, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Adds an end-to-end Fluss + Lakekeeper example: README, Docker Compose stack (Lakekeeper, migrations/init, RustFS, Fluss, Flink), a default warehouse JSON, and a Bash demo script that deploys, configures catalogs/tables, runs a tiering job, ingests sample data, and verifies via DuckDB.

Changes

Cohort / File(s) Summary
Example docs & demo
examples/fluss/README.md
New README describing the Fluss + Lakekeeper example, service endpoints, RustFS credentials, quick start, and manual usage including Flink SQL and tiering job steps.
Docker Compose stack
examples/fluss/docker-compose.yaml
New compose file provisioning Lakekeeper, migrations/bootstrap/init jobs, PostgreSQL, RustFS (S3/MinIO), bucket creation, download-jars one-shot, Fluss (ZK, coordinator, tablet), and Flink (jobmanager, taskmanager) with shared jars volume and coordinated startup/health checks.
Warehouse config
examples/fluss/create-default-warehouse.json
New default warehouse JSON (fluss-warehouse) with S3/MinIO storage-profile (bucket/prefix, endpoint, region, path-style-access, STS enabled) and access-key credentials.
Demo runner script
examples/fluss/run-demo.sh
New Bash demo that waits for services, creates Fluss catalog/database/table via Flink SQL client, starts the Fluss tiering job, ingests generated rows, and verifies tiered data via a DuckDB-based query against Lakekeeper’s REST catalog.

Sequence Diagram(s)

sequenceDiagram
  participant Dev as Developer
  participant Compose as Docker Compose
  participant Init as Init/Migrations
  participant RustFS as RustFS/MinIO
  participant Lakekeeper as Lakekeeper (REST)
  participant Flink as Flink (JobManager)
  participant Fluss as Fluss (Tiering Job)
  participant DuckDB as DuckDB (verification)

  Dev->>Compose: docker compose up -d
  Compose->>Init: run migrations & bootstrap jobs
  Init->>Lakekeeper: POST /management/v1/bootstrap, /warehouse
  Init->>RustFS: create bucket `storage/fluss`
  Compose->>Lakekeeper: start and report /health
  Compose->>Flink: start JobManager & TaskManager (mount jars)
  Compose->>Fluss: start coordinator & tablet (mount plugins)
  Dev->>Flink: sql-client -> create catalog, db, table
  Dev->>Fluss: submit Flink tiering job (detached)
  Dev->>Flink: insert generated rows into `orders`
  Fluss->>Lakekeeper: tiered writes -> Iceberg via REST catalog
  Dev->>DuckDB: query Lakekeeper REST catalog
  DuckDB->>Dev: return tiered data
Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

🐰 I hopped through YAML, JSON, and a script so spry,

I started containers, made buckets, and watched records fly,
Fluss stacked tiers while Lakekeeper kept the key,
DuckDB peeked in and said “rows I see!”
🥕 A little rabbit dance — demo done with glee.

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'docs: Fluss example' directly describes the main change: adding documentation and example files for a Fluss streaming integration with Lakekeeper.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@fresh-borzoni fresh-borzoni changed the title Fluss example docs: Fluss example Apr 1, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
examples/fluss/run-demo.sh (1)

30-38: Consider adding a brief wait or health check before starting the tiering job.

The -d flag correctly runs the Flink job in the background, but the subsequent sleep 5 is a fixed delay. If the tiering job takes longer to initialize, the data insertion might happen before the job is ready to process it.

This is likely fine for the demo since Fluss buffers data, but worth noting.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@examples/fluss/run-demo.sh` around lines 30 - 38, The script uses a fixed
sleep after launching the tiering job (docker compose exec -d jobmanager
./bin/flink run ...) which can let data be inserted before the job is ready;
replace the static sleep 5 with a readiness loop that polls the Flink job status
or jobmanager logs until the tiering job is running (for example by repeatedly
invoking a Flink CLI status command or checking jobmanager logs for the job
name) and only proceed when the job reports RUNNING or a ready-ready log line;
update run-demo.sh to perform this health check before continuing with data
insertion so the tiering job is guaranteed initialized.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@examples/fluss/run-demo.sh`:
- Around line 66-76: The hard-coded network name "fluss_iceberg_net" in the run
loop can break when Docker Compose uses a different project name; update the
script to avoid a fixed network string by either exporting a project name or
building the network name dynamically: set COMPOSE_PROJECT_NAME=fluss near the
top before bringing up compose, or replace occurrences of "fluss_iceberg_net"
with "${COMPOSE_PROJECT_NAME:-fluss}_iceberg_net" so the docker run line uses
the computed network name when running the DuckDB command (the DUCKDB_QUERY
usage in the docker run remains unchanged).

---

Nitpick comments:
In `@examples/fluss/run-demo.sh`:
- Around line 30-38: The script uses a fixed sleep after launching the tiering
job (docker compose exec -d jobmanager ./bin/flink run ...) which can let data
be inserted before the job is ready; replace the static sleep 5 with a readiness
loop that polls the Flink job status or jobmanager logs until the tiering job is
running (for example by repeatedly invoking a Flink CLI status command or
checking jobmanager logs for the job name) and only proceed when the job reports
RUNNING or a ready-ready log line; update run-demo.sh to perform this health
check before continuing with data insertion so the tiering job is guaranteed
initialized.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: c9954593-e937-4628-8b20-a40a7e8a94f4

📥 Commits

Reviewing files that changed from the base of the PR and between 294dbbf and f51d869.

📒 Files selected for processing (4)
  • examples/fluss/README.md
  • examples/fluss/create-default-warehouse.json
  • examples/fluss/docker-compose.yaml
  • examples/fluss/run-demo.sh

Comment thread examples/fluss/run-demo.sh Outdated
coderabbitai[bot]
coderabbitai Bot previously approved these changes Apr 1, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@examples/fluss/run-demo.sh`:
- Around line 5-7: The docker compose exec call can fail if the jobmanager
container hasn't started; replace the single-line `docker compose exec -T
jobmanager bash -c 'until curl -sf http://lakekeeper:8181/health > /dev/null
2>&1; do sleep 1; done'` with a guarded retry: first wait for the jobmanager
container to be reported as running (e.g., loop on `docker compose ps`/`docker
compose ps --filter status=running` and grep for jobmanager) and only then run
the curl loop with `docker compose exec`; implement this by adding an outer
`until` loop that checks container running status (or `docker compose ps -q
jobmanager` + inspect) and sleeps/retries, then run the existing inner
curl-until check once the container is up.
- Around line 83-85: The current parsing grabs the last numeric token from
docker/duckdb output (variables output/count), which can pull numbers from logs
or errors; change the pipeline to capture only the query result line and
validate it's purely numeric. Specifically, run the docker compose command the
same way (preserve 2>&1) into output, then replace the count extraction with
something like: extract the last non-empty line (e.g., using tail -n1) and
require it matches ^[0-9]+$ (e.g., via grep -E '^[0-9]+$') before assigning to
count so only a deterministic numeric query result from DUCKDB_QUERY is used in
the if check. Ensure you still redirect errors (2>&1) and keep the if condition
checking count as a numeric value.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 8b7a8409-9578-49bf-84dc-c43e144ae046

📥 Commits

Reviewing files that changed from the base of the PR and between 6768ff3 and 2c302ba.

📒 Files selected for processing (3)
  • examples/fluss/create-default-warehouse.json
  • examples/fluss/docker-compose.yaml
  • examples/fluss/run-demo.sh
✅ Files skipped from review due to trivial changes (2)
  • examples/fluss/create-default-warehouse.json
  • examples/fluss/docker-compose.yaml

Comment thread examples/fluss/run-demo.sh
Comment thread examples/fluss/run-demo.sh Outdated
Comment thread examples/fluss/docker-compose.yaml

@c-thiel c-thiel left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM in general, just one minor remark. I will give it a more thorough look the coming days.

Comment thread examples/fluss/docker-compose.yaml Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@examples/fluss/run-demo.sh`:
- Around line 32-39: The detached Flink tiering start (docker compose exec ...
./bin/flink run ...) and the fixed sleeps (sleep 5 and the later sleep ~30) are
fragile; replace them with readiness checks that poll the Iceberg table until
data appears or a timeout elapses. After launching the job (the command starting
with "docker compose exec -d jobmanager ./bin/flink run" in run-demo.sh)
implement a loop that queries the Iceberg table (the same check used later
before DuckDB) at intervals, with a configurable timeout, and exit non-zero if
no rows are returned within the timeout; remove the fixed sleep lines and the
later blind 30s pause and use the same polling routine before running the DuckDB
query so the script deterministically fails on tiering timeout.
- Around line 6-7: Replace the unbounded readiness loops for "until docker
compose exec -T jobmanager true" and the inner "docker compose exec -T
jobmanager bash -c 'until curl -sf http://lakekeeper:8181/health ...'" with
bounded loops that fail after 120 seconds; implement by recording a start time,
polling until success or until elapsed > 120, and on timeout printing a clear
error message (e.g., "jobmanager did not become ready within 120s" / "lakekeeper
healthcheck failed within 120s") and exiting with non‑zero status; ensure both
readiness checks follow the same pattern so CI/demos stop deterministically on
failure.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 4c39f40d-ed12-47cf-9d8c-8382d347a29f

📥 Commits

Reviewing files that changed from the base of the PR and between 2c302ba and 7742175.

📒 Files selected for processing (2)
  • examples/fluss/docker-compose.yaml
  • examples/fluss/run-demo.sh
🚧 Files skipped from review as they are similar to previous changes (1)
  • examples/fluss/docker-compose.yaml

Comment thread examples/fluss/run-demo.sh Outdated
Comment thread examples/fluss/run-demo.sh Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
examples/fluss/run-demo.sh (2)

89-105: Data availability check could be more deterministic.

The grep -q "0 rows" check relies on DuckDB's default output format. If the output format changes or an error message doesn't include this pattern, the script may behave unexpectedly.

A more robust approach would use structured output:

♻️ Optional: Use COUNT(*) with structured output
+COUNT_QUERY="
+INSTALL iceberg; LOAD iceberg; INSTALL httpfs; LOAD httpfs;
+CREATE SECRET (TYPE s3, KEY_ID 'rustfs-root-user', SECRET 'rustfs-root-password',
+    ENDPOINT 'localtest.me:9000', USE_SSL false, URL_STYLE 'path');
+CREATE SECRET (TYPE ICEBERG, ENDPOINT 'http://lakekeeper:8181/catalog', TOKEN 'dummy');
+ATTACH 'fluss-warehouse' AS lk (TYPE ICEBERG);
+SELECT COUNT(*) FROM lk.demo.orders;
+"
+
 echo "--- Waiting for data to be tiered to Iceberg ---"
 SECONDS=0
 while [ $SECONDS -lt 120 ]; do
-    output=$(docker compose run --rm -T duckdb duckdb -c "$DUCKDB_QUERY" 2>&1)
-    if echo "$output" | grep -q "0 rows"; then
+    count=$(docker compose run --rm -T duckdb duckdb -csv -noheader -c "$COUNT_QUERY" 2>&1 | tail -n1 | tr -d '\r')
+    if [[ "$count" =~ ^[0-9]+$ ]] && (( count > 0 )); then
+        echo "--- DuckDB query result ---"
+        docker compose run --rm -T duckdb duckdb -c "$DUCKDB_QUERY"
+        # ... rest of success output
+        exit 0
+    fi
-        sleep 5
-        continue
-    fi
+    sleep 5

However, for demo purposes, the current approach is functional and the bounded timeout ensures deterministic failure.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@examples/fluss/run-demo.sh` around lines 89 - 105, The current loop checks
DuckDB output text for the string "0 rows" (see the docker compose run --rm -T
duckdb duckdb -c "$DUCKDB_QUERY" invocation and the grep -q "0 rows" test),
which is brittle; instead run a deterministic COUNT(*) query and parse the
numeric count to decide readiness. Update the command that sets output to
execute a SELECT COUNT(*) AS cnt FROM (...) (or wrap $DUCKDB_QUERY) and then
parse the returned number (e.g., with awk/sed) rather than grepping for "0
rows", and use that numeric value in the while loop to break when count > 0;
keep the SECONDS timeout logic unchanged.

5-12: Consider avoiding eval for slightly safer command execution.

The static analysis flags eval "$@" (SC2294). While this is acceptable in a controlled demo script where all commands are hardcoded, passing commands as function arguments and using "$@" directly (without eval) would be safer if the commands didn't require shell parsing (pipes, redirections).

Since the commands here include shell constructs like grep -q and curl ... > /dev/null, eval is necessary. This is fine for demo purposes.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@examples/fluss/run-demo.sh` around lines 5 - 12, The wait_for function
currently uses eval "$@" which static analysis flags; change wait_for to avoid
eval by either (a) accepting the command as separate args and executing them
directly with "$@" (update callers to pass commands/components without shell
metacharacters), or (b) if callers need shell parsing (pipes/redirections),
accept a single command string and run it explicitly with bash -c "$cmd" to make
the intent explicit; update the wait_for function signature and its callers
accordingly (refer to the wait_for function and all places invoking it).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@examples/fluss/run-demo.sh`:
- Around line 89-105: The current loop checks DuckDB output text for the string
"0 rows" (see the docker compose run --rm -T duckdb duckdb -c "$DUCKDB_QUERY"
invocation and the grep -q "0 rows" test), which is brittle; instead run a
deterministic COUNT(*) query and parse the numeric count to decide readiness.
Update the command that sets output to execute a SELECT COUNT(*) AS cnt FROM
(...) (or wrap $DUCKDB_QUERY) and then parse the returned number (e.g., with
awk/sed) rather than grepping for "0 rows", and use that numeric value in the
while loop to break when count > 0; keep the SECONDS timeout logic unchanged.
- Around line 5-12: The wait_for function currently uses eval "$@" which static
analysis flags; change wait_for to avoid eval by either (a) accepting the
command as separate args and executing them directly with "$@" (update callers
to pass commands/components without shell metacharacters), or (b) if callers
need shell parsing (pipes/redirections), accept a single command string and run
it explicitly with bash -c "$cmd" to make the intent explicit; update the
wait_for function signature and its callers accordingly (refer to the wait_for
function and all places invoking it).

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 1680c0d7-428c-4977-b82a-65b31b0e20a0

📥 Commits

Reviewing files that changed from the base of the PR and between 7742175 and 96418ce.

📒 Files selected for processing (1)
  • examples/fluss/run-demo.sh

coderabbitai[bot]
coderabbitai Bot previously approved these changes Apr 1, 2026
@brgr-s

brgr-s commented Apr 13, 2026

Copy link
Copy Markdown
Contributor

@fresh-borzoni Thanks again for your PR and sorry for taking a bit of time for the review.

LGTM :) I just pushed some small additions to the example doc and your script. I also added a small section to the lakekeeper docs describing your example. Should be ready to be merged when checks are done.

@github-actions

Copy link
Copy Markdown
Contributor

✅ PR Title Formatted Correctly

The title of this PR match the correct format. Thank you!

@brgr-s brgr-s merged commit 6f5a52a into lakekeeper:main Apr 14, 2026
63 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants