Skip to content

Worker and external worker service clean up#21

Closed
sjvanrossum wants to merge 5 commits intonivaldoh:rust_sdkfrom
sjvanrossum:rust_sdk
Closed

Worker and external worker service clean up#21
sjvanrossum wants to merge 5 commits intonivaldoh:rust_sdkfrom
sjvanrossum:rust_sdk

Conversation

@sjvanrossum
Copy link
Copy Markdown
Collaborator

@sjvanrossum sjvanrossum commented Jan 9, 2023

  • Clean up:
    • Moved main into examples, removed a dependency on http::Uri, instead using tonic::transport::Uri (which is effectively just a http::Uri re-export, but could obviously change).
    • Removed refs in external worker service, replaced std::sync::Mutex with tokio::sync::Mutex, made sure to avoid creating, starting and re-inserting a worker if their worker id already exists and made sure to start the worker. I'm not sure if external Rust workers will be necessary ever, but we can look into that somewhere in the future.
  • Worker::stop is async now and closes the control channel
  • Worker::new now just returns Self, I checked that all synchronization is handled in impl Worker
  • An implementation for Register instructions has been added (1 down and 6 to go, although register is deprecated)

@sjvanrossum sjvanrossum changed the title Rust sdk Worker and external worker service clean up Jan 9, 2023
@sjvanrossum
Copy link
Copy Markdown
Collaborator Author

sjvanrossum commented Jan 10, 2023

@nivaldoh I've also cleaned up the proto module and usage thereof.
The include_file bundle had the somewhat ugly side effect of producing modules like apache_beam::proto::org::apache::beam::model..., so I've added a proto.rs file which imports the handful of separate proto bundles, which produces apache_beam::proto::..., which makes more sense hierarchically. Re-exports would still highlight the original package name, so this keeps those module names brief and informative (e.g. apache_beam::proto::org::apache::beam::model::pipeline::v1 vs. apache_beam::proto::pipeline::v1).
I tried to name the module apache_beam::model (like Java and Go use, Python uses apache_beam.portability.api of which I'm not a big fan) as well, but found that having the apache_beam::proto::... prefix was more informative of the context.
Since the proto structs are so similarly named to concepts used in the SDK I've refactored usage of protos to always use a module prefix. This makes it easier to distinguish where we're dealing with protobuf messages and gRPC services.
The vN suffix makes maintenance for transitions to vN+1 a little easier and keeps the context around usage clear, so for every imported submodule from a proto package (e.g. instruction_request) I've also suffixed them with the version number (e.g. instruction_request_v1).
I've also added the corresponding version number to the external worker pool service struct name for the same reasons.

@github-actions
Copy link
Copy Markdown

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.

@github-actions github-actions bot added the stale label Mar 11, 2023
@github-actions
Copy link
Copy Markdown

This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

@github-actions github-actions bot closed this Mar 18, 2023
@sjvanrossum sjvanrossum reopened this Apr 21, 2023
@github-actions github-actions bot removed the stale label Apr 21, 2023
@laysakura
Copy link
Copy Markdown

@sjvanrossum Could you kindly make a PR to https://github.com/laysakura/beam/ repository for these changes?

@laysakura
Copy link
Copy Markdown

@sjvanrossum I did in laysakura#31 🙂

@sjvanrossum
Copy link
Copy Markdown
Collaborator Author

@laysakura That's great, thanks! I'll close this one.
I've had a busy few weeks, but will get that data channel implementation with tests over to you before EOW.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants