[Python] Hugging Face pipeline support#27399
Conversation
Codecov Report
@@ Coverage Diff @@
## master #27399 +/- ##
==========================================
- Coverage 71.09% 70.84% -0.26%
==========================================
Files 859 862 +3
Lines 104555 104939 +384
==========================================
+ Hits 74338 74347 +9
- Misses 28660 29035 +375
Partials 1557 1557
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 16 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
Run Python 3.8 PostCommit |
|
hugging face integration tests passed on 3.8 PostCommit |
|
Run Python 3.8 PostCommit |
|
PTAL, ready for review. Hugging Face integration tests passed. Some other tests unrelated to this PR are failing. Opened #27734 |
|
Run Python 3.8 PostCommit |
|
assign to next reviewer |
|
R: @damccorm |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control |
damccorm
left a comment
There was a problem hiding this comment.
Left some comments, but the core pieces look good to me
sdks/python/apache_beam/examples/inference/huggingface_question_answering.py
Show resolved
Hide resolved
|
Run Python 3.8 PostCommit |
damccorm
left a comment
There was a problem hiding this comment.
LGTM once all suites pass
|
Tests passed but not reflected here. Opened #27808 for the failing PreCommits. It is not related to this change. Merging this PR! |
|
Can you resolve my comments? |
|
I've added the enum you suggested. Are there any comments you didn't publish yet accidentally? |
|
Interesting. I probably forgot to submit them. Sorry about that. |
|
Thought so! I wanted to get this in for 2.50 so Danny reviewed it as you were OOO. Some of your comments are already addressed in later commits. I'll send a short PR for rest of them to you. |
|
Thanks, feel free to resolve my comments. |
This PR adds Hugging Face pipeline support to RunInference by adding a new model handler for it. Build on top of #26632
Example Job with Question Answering Pipeline on Dataflow
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.