[BEAM-4181] Add ReadFiles transform to TfRecordIO#9088
Conversation
|
R: @chamikaramj Can you take a look? |
|
Run JavaPortabilityApi PreCommit |
|
Heyo -- just a quick question after thinking about this. Should I refactor the existing If yes, how do you feel about the It's weird to me to refactor the original |
|
R: @pabloem Sorry about the delay here. Pablo agreed to take over this review so passing to him. |
|
Usually, the difference between Read and ReadFiles transforms is that, ReadFiles currently does not support dynamic work rebalancing while Read does (this will change with SDF). I see that, for TFRecordIO, we do not support dynamic work rebalancing anyways, so refactoring Read transform to use proposed ReadFiles should be fine. I don't think we need the hint if we use ReadFiles for both implementations. |
|
Thanks Cham. It's useful to get your advice here : ) I've looked at the implementation, and it LGTM. @RyanSkraba let me know if you'd like to do any refactoring, or if you'd just like me to merge this. |
|
Thanks for the review! Please go ahead and merge it -- before doing any additional internal refactoring (which wouldn't make much difference here), I'd like to take a tour around the existing file-based IOs to identify where they're consistent or different. |
Minor improvement to add a FileIO-based
readFiles()to the TensorFlow IO.https://issues.apache.org/jira/browse/BEAM-4181
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.