[Managed Iceberg] unbounded source by ahmedabu98 · Pull Request #33504 · apache/beam

ahmedabu98 · 2025-01-06T18:16:10Z

Unbounded (streaming) source for Managed Iceberg.

See design doc for high level overview: https://s.apache.org/beam-iceberg-incremental-source

…erg_streaming_source

github-actions · 2025-01-30T22:06:51Z

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

ahmedabu98 · 2025-02-03T15:51:02Z

R: @kennknowles
R: @regadas

Can y'all take a look? I still have to write some tests, but it's at a good spot for a first round of reviews. I ran a bunch of pipelines (w/Legacy DataflowRunner) at different scales and the throughput/scalability looks good.

github-actions · 2025-02-03T15:52:21Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

…erg_streaming_source

kennknowles

Overall, I think all the pieces are in the right place. Just a question about why an SDF is the way it is and a couple code-level comments.

This seems like something you want to test a lot of different ways before it gets into a release. Maybe get another set of eyes like @chamikaramj or @Abacn too. But I'm approving and leaving to your judgment.

sdks/java/io/iceberg/bqms/build.gradle

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/SnapshotRange.java

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/ReadTask.java

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/ReadTaskDescriptor.java

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergIO.java

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/ReadFromGroupedTasks.java

kennknowles

Wait actually I forgot I want to have the discussion about the high level toggle between incremental scan source and bounded source.

…erg_streaming_source

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergIO.java

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/ReadFromGroupedTasks.java

…erg_streaming_source

ahmedabu98 · 2025-03-05T18:37:01Z

@chamikaramj this is ready for another review

…erg_streaming_source

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IncrementalScanSource.java

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/ReadFromTasks.java

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/WatchForSnapshots.java

chamikaramj

Thanks. LGTM.

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergIO.java

sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/Managed.java

…remove window step; add --strea ming=true validation; add IO links to Managed java doc

codecov · 2025-03-11T17:21:09Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 56.32%. Comparing base (c1d0fa4) to head (fce87dc).
Report is 17 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #33504      +/-   ##
============================================
+ Coverage     56.28%   56.32%   +0.03%     
  Complexity     3286     3286              
============================================
  Files          1166     1172       +6     
  Lines        178704   178936     +232     
  Branches       3398     3398              
============================================
+ Hits         100591   100786     +195     
- Misses        74860    74897      +37     
  Partials       3253     3253

Flag	Coverage Δ
python	`81.29% <ø> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…erg_streaming_source

…Getsize

…Getsize; add resilience to watch snapshots transform

chamikaramj · 2025-03-19T05:50:22Z

LGTM. Thanks!

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/CreateReadTasksDoFn.java

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/ReadFromTasks.java

scwhittle · 2025-03-19T12:11:45Z

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/ReadUtils.java

+/** Helper class for source operations. */
+public class ReadUtils {
+  // default is 8MB. keep this low to avoid overwhelming memory
+  static final int MAX_FILE_BUFFER_SIZE = 1 << 18; // 256KB


might want this to be configurble or pipeline option.

I'd rather we wait until there's a need to expose it

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableCache.java

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/WatchForSnapshots.java

scwhittle

Thanks!

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableCache.java

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/ReadUtils.java

ahmedabu98 · 2025-03-20T15:08:45Z

Thanks for the review @scwhittle! If all else is good, I'll merge it when tests pass

…u98/beam into iceberg_streaming_source

initial

bb87511

github-actions bot added java io labels Jan 6, 2025

ahmedabu98 marked this pull request as draft January 6, 2025 18:16

ahmedabu98 added 7 commits January 7, 2025 15:50

let CombinedScanTask do splitting (based on Parquet row groups)

853de4d

perf improv

69fd988

create one read task descriptor per snapshot range

da2f33f

Merge branch 'master' of https://github.com/ahmedabu98/beam into iceb…

73c8992

…erg_streaming_source

some improvements

81ca709

use GiB for streaming, Redistribute for batch; update docs

e319d76

Merge branch 'master' of https://github.com/ahmedabu98/beam into iceb…

c25cd75

…erg_streaming_source

ahmedabu98 marked this pull request as ready for review January 30, 2025 21:09

use static value

af1ec85

ahmedabu98 added 3 commits February 3, 2025 16:27

add some test

f5d3268

Merge branch 'master' of https://github.com/ahmedabu98/beam into iceb…

df40239

…erg_streaming_source

Merge branch 'master' of https://github.com/ahmedabu98/beam into iceb…

43ab88f

…erg_streaming_source

kennknowles approved these changes Feb 5, 2025

View reviewed changes

kennknowles requested changes Feb 5, 2025

View reviewed changes

ahmedabu98 added 3 commits February 7, 2025 08:32

Merge branch 'master' of https://github.com/ahmedabu98/beam into iceb…

20db0ee

…erg_streaming_source

add a java doc; don't use static block to create coder

622625f

spotless

4c25d3f

kennknowles approved these changes Feb 11, 2025

View reviewed changes

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergIO.java Outdated Show resolved Hide resolved

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/ReadFromGroupedTasks.java Outdated Show resolved Hide resolved

ahmedabu98 added 2 commits February 13, 2025 12:35

add options: from/to timestamp, starting strategy, and streaming toggle

8666166

trigger integration tests

297c309

github-actions bot added the build label Feb 13, 2025

ahmedabu98 added 2 commits February 13, 2025 13:47

small test fix

5e3a2cc

Merge branch 'master' of https://github.com/ahmedabu98/beam into iceb…

8b131fd

…erg_streaming_source

Merge branch 'master' of https://github.com/ahmedabu98/beam into iceb…

5263a13

…erg_streaming_source

scwhittle requested changes Mar 6, 2025

View reviewed changes

sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/WatchForSnapshots.java Outdated Show resolved Hide resolved

chamikaramj reviewed Mar 6, 2025

View reviewed changes

ahmedabu98 mentioned this pull request Mar 7, 2025

[Task]: Add Python AfterSynchronizedProcessingTime trigger and add an Iceberg CDC streaming read test #34212

Closed

17 tasks

ahmedabu98 added 4 commits March 7, 2025 10:56

use cached schemas

40fe4ab

watermark based on snapshot timestamp; remove infinite allowed skew; …

db3c570

…remove window step; add --strea ming=true validation; add IO links to Managed java doc

remove check

7078f20

remove watermark support; add a counter and some helpful logging

fce87dc

ahmedabu98 added 4 commits March 11, 2025 17:56

Merge branch 'master' of https://github.com/ahmedabu98/beam into iceb…

adbbfaf

…erg_streaming_source

change metric name

30f3ced

use Redistribute for unbounded; make ReadFromTasks an SDF to leverage @…

078b63a

…Getsize; add resilience to watch snapshots transform

remove CDC schema and output the data record normally

941e8aa

scwhittle requested changes Mar 19, 2025

View reviewed changes

ahmedabu98 added 2 commits March 19, 2025 18:36

use LoadingCache; leverage iceberg bin-packing; address other comments

44923f3

spotless

1c0f7d7

scwhittle reviewed Mar 20, 2025

View reviewed changes

ahmedabu98 force-pushed the iceberg_streaming_source branch from 1f47712 to 1c0f7d7 Compare March 20, 2025 12:18

validate catalog cache consistency; added a TODO

7f12282

Merge branch 'master' into iceberg_streaming_source

17816bd

github-actions bot removed the build label Mar 20, 2025

ahmedabu98 added 2 commits March 20, 2025 14:23

trigger integration tests

24d1782

Merge branch 'iceberg_streaming_source' of https://github.com/ahmedab…

3b8bac1

…u98/beam into iceberg_streaming_source

github-actions bot added the build label Mar 20, 2025

ahmedabu98 merged commit 332fe03 into apache:master Mar 20, 2025
110 checks passed

prodriguezdefino pushed a commit to prodriguezdefino/beam-pabs that referenced this pull request Mar 26, 2025

[Managed Iceberg] unbounded source (apache#33504)

c81b86e

Conversation

ahmedabu98 commented Jan 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 30, 2025

Uh oh!

ahmedabu98 commented Feb 3, 2025

Uh oh!

github-actions bot commented Feb 3, 2025

Uh oh!

kennknowles left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kennknowles left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ahmedabu98 commented Mar 5, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chamikaramj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Mar 11, 2025

Codecov Report

Uh oh!

chamikaramj commented Mar 19, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

scwhittle Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

ahmedabu98 Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

scwhittle left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ahmedabu98 commented Mar 20, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ahmedabu98 commented Jan 6, 2025 •

edited

Loading