Skip to content

proto_format: Use aspect to protoprint formatted api#21750

Merged
htuch merged 3 commits intoenvoyproxy:mainfrom
phlax:bazel-proto_sync
Jun 30, 2022
Merged

proto_format: Use aspect to protoprint formatted api#21750
htuch merged 3 commits intoenvoyproxy:mainfrom
phlax:bazel-proto_sync

Conversation

@phlax
Copy link
Copy Markdown
Member

@phlax phlax commented Jun 16, 2022

Commit Message:

This switches from calling protoprint in scripts with python multiproc to using a bazel aspect

Both the proto_format.sh job and the protoprint_test are switched to using a prebuilt protoprinted version of the api/test protos

Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

@phlax phlax force-pushed the bazel-proto_sync branch from aab244c to 0800693 Compare June 16, 2022 19:27
@phlax phlax changed the title bazel: Shift to bazel and refactor to class bazel: Shift proto_format to bazel and refactor to class Jun 16, 2022
@phlax phlax changed the title bazel: Shift proto_format to bazel and refactor to class [WIP] bazel: Shift proto_format to bazel and refactor to class Jun 16, 2022
@phlax phlax marked this pull request as draft June 16, 2022 19:29
@phlax phlax force-pushed the bazel-proto_sync branch 2 times, most recently from daed48d to ae69158 Compare June 16, 2022 19:50
@phlax phlax changed the title [WIP] bazel: Shift proto_format to bazel and refactor to class [WIP] bazel: Shift proto_sync to bazel and refactor to class Jun 16, 2022
@phlax phlax force-pushed the bazel-proto_sync branch 2 times, most recently from d4e30de to 435b2d5 Compare June 17, 2022 12:21
@phlax phlax changed the title [WIP] bazel: Shift proto_sync to bazel and refactor to class [WIP] proto_format: Refactor to class Jun 17, 2022
@phlax phlax changed the title [WIP] proto_format: Refactor to class [WIP] proto_format: Refactor proto_sync to class Jun 17, 2022
@phlax phlax force-pushed the bazel-proto_sync branch 2 times, most recently from cfc302d to ca34ec4 Compare June 18, 2022 08:16
@repokitteh-read-only repokitteh-read-only bot added the deps Approval required for changes to Envoy's external dependencies label Jun 18, 2022
@repokitteh-read-only
Copy link
Copy Markdown

CC @envoyproxy/dependency-shepherds: Your approval is needed for changes made to (bazel/.*repos.*\.bzl)|(bazel/dependency_imports\.bzl)|(api/bazel/.*\.bzl)|(.*/requirements\.txt)|(.*\.patch).
envoyproxy/dependency-shepherds assignee is @wrowe

🐱

Caused by: #21750 was synchronize by phlax.

see: more, trace.

@repokitteh-read-only
Copy link
Copy Markdown

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy[\w/]*/(v1alpha\d?|v1|v2alpha\d?|v2))|(api/envoy/type/(matcher/)?\w+.proto).
CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy/|docs/root/api-docs/).
envoyproxy/api-shepherds assignee is @htuch
CC @envoyproxy/api-watchers: FYI only for changes made to (api/envoy/|docs/root/api-docs/).

🐱

Caused by: #21750 was synchronize by phlax.

see: more, trace.

@phlax phlax changed the title [WIP] proto_format: Refactor proto_sync to class [WIP] proto_format: Use aspect to protoprint formatted api Jun 20, 2022
@phlax phlax force-pushed the bazel-proto_sync branch 5 times, most recently from 9146144 to 3470614 Compare June 21, 2022 15:20
@phlax phlax force-pushed the bazel-proto_sync branch 7 times, most recently from b6fd374 to 93affd7 Compare June 27, 2022 05:28
@phlax
Copy link
Copy Markdown
Member Author

phlax commented Jun 27, 2022

Signed-off-by: Ryan Northey <ryan@synca.io>
@phlax phlax force-pushed the bazel-proto_sync branch from 93affd7 to b37e935 Compare June 27, 2022 05:36
@phlax phlax changed the title [WIP] proto_format: Use aspect to protoprint formatted api proto_format: Use aspect to protoprint formatted api Jun 27, 2022
@phlax phlax marked this pull request as ready for review June 27, 2022 05:37
@phlax
Copy link
Copy Markdown
Member Author

phlax commented Jun 27, 2022

this one can be slightly slower in the uncached situation (but only sometimes) - it should be faster in every other situation

class ProtoprintTraverser:

def traverse_file(self, file_proto, visitor):
# This breaks the Visitor contract, ie class props should not be set on individual visitation.
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this reflects what was being done already in the main function and is i think the reason why this was not implemented as a protoc plugin originally - either way this is necessary without making other changes to traverser/visitor

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this is kind of evil. Can you add some explanation of "why" for future readers?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo added

@phlax phlax removed the v2-freeze label Jun 27, 2022
from bazel_tools.tools.python.runfiles import runfiles

from tools.api_proto_plugin import annotations, traverse, visitor
sys.path = [p for p in sys.path if not p.endswith('bazel_tools')]
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems a bit evil..

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the usual hack, as added elsewhere - if you use runfiles you cant import tools without mangling sys.path

Copy link
Copy Markdown
Member Author

@phlax phlax Jun 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added here because i needed runfiles to get the clang-tidy config - otherwise i generally avoid runfiles if i can

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you factor this pattern out into some library so it's clear what is going on?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not really - its one line and needs to be explicit

this is really nothing new - you and i have added a few of these - and i have removed them whenever i can

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ill add a comment to make it clearer

Signed-off-by: Ryan Northey <ryan@synca.io>
@wrowe
Copy link
Copy Markdown
Contributor

wrowe commented Jun 28, 2022

/lgtm deps
This looks more promising, still needs an api stamp if concerns are asked and answered.

@repokitteh-read-only
Copy link
Copy Markdown

no relevant owners for "deps"

🐱

Caused by: a #21750 (comment) was created by @wrowe.

see: more, trace.

Copy link
Copy Markdown
Member

@htuch htuch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM modulo this sys.path hack nit.

Signed-off-by: Ryan Northey <ryan@synca.io>
Copy link
Copy Markdown
Member

@htuch htuch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks - as per discussion on Slack let's land it with comment for later followup.

@htuch htuch merged commit f0711a6 into envoyproxy:main Jun 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api deps Approval required for changes to Envoy's external dependencies

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants