Skip to content

Automatically switch to upload_graph when the graph is large#28621

Merged
Abacn merged 13 commits intoapache:masterfrom
liferoad:upload-graph1
Oct 27, 2023
Merged

Automatically switch to upload_graph when the graph is large#28621
Abacn merged 13 commits intoapache:masterfrom
liferoad:upload-graph1

Conversation

@liferoad
Copy link
Copy Markdown
Contributor

@liferoad liferoad commented Sep 22, 2023 •
edited
Loading

Now the job graph is larger than 10 MB, it will be automatically uploaded by enabling upload_graph for Java SDK.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@liferoad
Copy link
Copy Markdown
Contributor Author

liferoad commented Sep 25, 2023

I plan to manually test this later (might add one integration test). So no need to merge this now. Thanks.

@liferoad liferoad marked this pull request as ready for review September 30, 2023 16:07
@liferoad
Copy link
Copy Markdown
Contributor Author

liferoad commented Sep 30, 2023

Tested these changes internally. The test passed.

@liferoad
Copy link
Copy Markdown
Contributor Author

liferoad commented Sep 30, 2023

R: @Abacn

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 30, 2023

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

@kennknowles
Copy link
Copy Markdown
Member

kennknowles commented Oct 2, 2023

Let us get a test externally. It should be very easy - just a single small test method with a big graph.

Abacn
Abacn reviewed Oct 3, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@Abacn Abacn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the integration test, /runners/google-cloud-dataflow-java/src/test could be a choice to settle these tests. Although currently there is only unit tests, this path is added to the integration test class path:

beam/runners/google-cloud-dataflow-java/build.gradle

Line 138 in f3df03d

googleCloudPlatformIntegrationTest project(project.path)

@liferoad
Copy link
Copy Markdown
Contributor Author

liferoad commented Oct 20, 2023

Let us get a test externally. It should be very easy - just a single small test method with a big graph.

Added one unit test.

@liferoad liferoad requested review from Abacn and kennknowles October 20, 2023 20:39
kennknowles
kennknowles approved these changes Oct 23, 2023
View reviewed changes
Copy link
Copy Markdown
Member

@kennknowles kennknowles left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM except it isn't really a breaking change.

Abacn
Abacn reviewed Oct 23, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@Abacn Abacn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! the code changes LGTM. Had a comment about CHANGES.md item

@liferoad liferoad changed the title Automatically Switch to upload_graph when the graph is large Automatically switch to upload_graph when the graph is large Oct 26, 2023
@liferoad liferoad requested a review from Abacn October 26, 2023 18:35
@liferoad liferoad requested a review from kennknowles October 26, 2023 18:49
@kennknowles
Copy link
Copy Markdown
Member

kennknowles commented Oct 26, 2023

Once it is green again (for the record) will merge. Or any committer can merge.

@liferoad
Copy link
Copy Markdown
Contributor Author

liferoad commented Oct 26, 2023

Run Java PreCommit

1 similar comment
@liferoad
Copy link
Copy Markdown
Contributor Author

liferoad commented Oct 27, 2023

Run Java PreCommit

@liferoad
Copy link
Copy Markdown
Contributor Author

liferoad commented Oct 27, 2023

Note "org.apache.beam.runners.dataflow.worker.StreamingDataflowWorkerTest.testActiveThreadMetric[*]" seems flaky. I tried several times to get all greens.

@Abacn
Copy link
Copy Markdown
Contributor

Abacn commented Oct 27, 2023

Note "org.apache.beam.runners.dataflow.worker.StreamingDataflowWorkerTest.testActiveThreadMetric[*]" seems flaky. I tried several times to get all greens.

Yes this is unrelated. Tracking bug: #28957 caused by #28513 there is a pending PR include a fix: #29041 (comment)

@Abacn Abacn merged commit 3b89039 into apache:master Oct 27, 2023
@kennknowles
Copy link
Copy Markdown
Member

kennknowles commented Oct 27, 2023

I think it is OK to disable the test or even better to roll back the change so we do not have untested stuff in the codebase.

@Abacn Abacn mentioned this pull request Nov 30, 2023
Merged
@liferoad liferoad mentioned this pull request Jan 19, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kennknowles kennknowles kennknowles approved these changes

@Abacn Abacn Awaiting requested review from Abacn

Assignees

No one assigned

Labels

dataflow runners

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@liferoad @kennknowles @Abacn