[BEAM-13015] Add state caching benchmark and move benchmarks to their own module. #16542

lukecwik · 2022-01-18T04:24:01Z

This simplifies adding unit tests and decreases the clutter in the sdks/java/harness module.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

`ValidatesRunner` compliance status (on master branch)

Lang	ULR	Twister2
Go	---	---
Java
Python	---	---
XLang		---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

lukecwik · 2022-01-18T04:25:04Z

R: @ibzib
CC: @scwhittle

codecov · 2022-01-18T04:46:23Z

Codecov Report

Merging #16542 (f64f505) into master (6814a06) will increase coverage by 37.26%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master   #16542       +/-   ##
===========================================
+ Coverage   46.38%   83.64%   +37.26%     
===========================================
  Files         202      452      +250     
  Lines       19897    62168    +42271     
===========================================
+ Hits         9230    52003    +42773     
- Misses       9682    10165      +483     
+ Partials      985        0      -985

Impacted Files	Coverage Δ
sdks/go/pkg/beam/io/synthetic/source.go
sdks/go/pkg/beam/io/filesystem/filesystem.go
sdks/go/pkg/beam/internal/errors/errors.go
sdks/go/pkg/beam/transforms/stats/mean.go
sdks/go/pkg/beam/io/filesystem/gcs/gcs.go
sdks/go/pkg/beam/core/runtime/xlangx/expand.go
sdks/go/pkg/beam/core/runtime/exec/reshuffle.go
sdks/go/pkg/beam/core/graph/coder/int.go
sdks/go/pkg/beam/util.go
sdks/go/pkg/beam/core/util/reflectx/util.go
... and 644 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6814a06...f64f505. Read the comment docs.

ibzib · 2022-01-18T22:40:29Z

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

Why is this commented out?

hold over from temp change.

ibzib · 2022-01-18T23:43:21Z

sdks/java/harness/jmh/src/main/java/org/apache/beam/fn/harness/jmh/ProcessBundleBenchmark.java

Can we use a library to do this? e.g. https://guava.dev/releases/19.0/api/docs/com/google/common/base/Strings.html#padStart(java.lang.String,%20int,%20char)

ibzib · 2022-01-18T23:49:52Z

sdks/java/harness/jmh/src/main/java/org/apache/beam/fn/harness/jmh/ProcessBundleBenchmark.java

What are the two expected stages? 1. impulse+toKeyAndValue 2. stateful+gbk?

My understanding is that toKeyAndValue is just a dummy transform that is added only for pipeline construction, and then its stage is not actually executed. Is that correct? And can we add a comment to clarify it?

Your understanding is correct since the stateful dofn causes a fusion break in the GreedyPipelineFuser.

Added comment.

aaltay · 2022-02-03T00:59:36Z

What is the next step on this PR?

… own module. This simplifies adding unit tests and decreases the clutter in the sdks/java/harness module.

Also clean-up comments around args and fix classpath to only be runtime classpath.

lukecwik · 2022-02-04T18:48:56Z

@ibzib this is ready for another review.

lukecwik · 2022-02-04T21:24:09Z

Run Java PreCommit

ibzib

LGTM

lukecwik · 2022-02-05T00:39:43Z

Run Java PreCommit

lukecwik · 2022-02-05T02:26:39Z

Run Java PreCommit

lukecwik · 2022-02-05T02:29:01Z

Run Website_Stage_GCS

lukecwik · 2022-02-05T02:29:18Z

Run Java PreCommit

lukecwik · 2022-02-05T03:22:54Z

Run Java PreCommit

lukecwik · 2022-02-05T03:24:32Z

Run Website_Stage_GCS PreCommit

lukecwik · 2022-02-05T05:45:05Z

Run Java PreCommit

lukecwik · 2022-02-05T07:00:08Z

Run Java PreCommit

lukecwik · 2022-02-05T16:51:25Z

Run Java PreCommit

lukecwik · 2022-02-05T18:44:13Z

Run Java PreCommit

lukecwik · 2022-02-05T19:07:55Z

Run Java PreCommit

lukecwik · 2022-02-06T04:20:55Z

Run Java PreCommit

aaltay · 2022-02-07T16:44:28Z

Run Java PreCommit

aaltay · 2022-02-07T16:44:55Z

For reference, link to the failed test: https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/4593/

lukecwik · 2022-02-07T17:24:24Z

It looks like Java PreCommit is perma-red. Now that #16748 is in. Rerunning Java PreCommit.

lukecwik · 2022-02-07T17:24:42Z

Run Java PreCommit

lukecwik · 2022-02-07T18:32:53Z

Run Java PreCommit

lukecwik mentioned this pull request Jan 18, 2022

[BEAM-13015] Provide caching statistics in the status client. #16495

Merged

4 tasks

ibzib reviewed Jan 18, 2022

View reviewed changes

[BEAM-13015] Add state caching benchmark and move benchmarks to their…

b79a01e

… own module. This simplifies adding unit tests and decreases the clutter in the sdks/java/harness module.

lukecwik force-pushed the beam13015.3 branch from 6797460 to b79a01e Compare February 4, 2022 18:20

github-actions bot added build java labels Feb 4, 2022

Make jmhTest run only one iteration without forking to speed it up.

c9e43c7

Also clean-up comments around args and fix classpath to only be runtime classpath.

increase timeout to 60 seconds

f64f505

ibzib approved these changes Feb 5, 2022

View reviewed changes

lukecwik merged commit eef048b into apache:master Feb 7, 2022

[BEAM-13015] Add state caching benchmark and move benchmarks to their own module. #16542

[BEAM-13015] Add state caching benchmark and move benchmarks to their own module. #16542

Uh oh!

Conversation

lukecwik commented Jan 18, 2022

ValidatesRunner compliance status (on master branch)

Examples testing status on various runners

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

Uh oh!

lukecwik commented Jan 18, 2022

Uh oh!

codecov bot commented Jan 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ibzib Jan 18, 2022

Choose a reason for hiding this comment

Uh oh!

lukecwik Feb 4, 2022

Choose a reason for hiding this comment

Uh oh!

ibzib Jan 18, 2022

Choose a reason for hiding this comment

Uh oh!

lukecwik Feb 4, 2022

Choose a reason for hiding this comment

Uh oh!

ibzib Jan 18, 2022

Choose a reason for hiding this comment

Uh oh!

lukecwik Feb 4, 2022

Choose a reason for hiding this comment

Uh oh!

aaltay commented Feb 3, 2022

Uh oh!

lukecwik commented Feb 4, 2022

Uh oh!

lukecwik commented Feb 4, 2022

Uh oh!

ibzib left a comment

Choose a reason for hiding this comment

Uh oh!

lukecwik commented Feb 5, 2022

Uh oh!

lukecwik commented Feb 5, 2022

Uh oh!

lukecwik commented Feb 5, 2022

Uh oh!

lukecwik commented Feb 5, 2022

Uh oh!

lukecwik commented Feb 5, 2022

Uh oh!

lukecwik commented Feb 5, 2022

Uh oh!

lukecwik commented Feb 5, 2022

Uh oh!

lukecwik commented Feb 5, 2022

Uh oh!

lukecwik commented Feb 5, 2022

Uh oh!

lukecwik commented Feb 5, 2022

Uh oh!

lukecwik commented Feb 5, 2022

Uh oh!

lukecwik commented Feb 6, 2022

Uh oh!

aaltay commented Feb 7, 2022

Uh oh!

aaltay commented Feb 7, 2022

Uh oh!

lukecwik commented Feb 7, 2022

Uh oh!

lukecwik commented Feb 7, 2022

Uh oh!

lukecwik commented Feb 7, 2022

Uh oh!

Reviewers

Assignees

`ValidatesRunner` compliance status (on master branch)

codecov bot commented Jan 18, 2022 •

edited

Loading