Skip to content
This repository was archived by the owner on Nov 18, 2025. It is now read-only.

feat: compileProtos bin script#547

Merged
alexander-fenster merged 9 commits intomasterfrom
compile-protos
Jul 29, 2019
Merged

feat: compileProtos bin script#547
alexander-fenster merged 9 commits intomasterfrom
compile-protos

Conversation

@alexander-fenster
Copy link
Contributor

This is a step towards using static protos in client libraries (both browser and Node.js).

Background: we cannot switch to using protoc to compile JavaScript stubs in client libraries right now since it will be a big refactor, a major change, and we don't really know how deeply we depend on protobufjs (e.g. some end users' code might actually use it).

But what we can and will do is to stop loading .proto files in the runtime (which is (a) slow, (b) sometimes unpredictable, and (c) not webpack-able), and use the protobufjs-supported JSON format which is:

  • simple
  • just a JSON, so can be loaded in browser
  • does not need any changes in the code (regular protobuf.loadSync accepts it, so does the @grpc/proto-loader, so no changes are needed)
  • can be produced offline.

This script will be placed into a bin folder and will be used by client libraries to compile protos to JSON.

  • Tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Jul 25, 2019
@codecov
Copy link

codecov bot commented Jul 25, 2019

Codecov Report

Merging #547 into master will decrease coverage by 7.29%.
The diff coverage is 79.74%.

Impacted file tree graph

@@            Coverage Diff            @@
##           master     #547     +/-   ##
=========================================
- Coverage   89.88%   82.59%   -7.3%     
=========================================
  Files          52       53      +1     
  Lines        3519     3544     +25     
  Branches      268      269      +1     
=========================================
- Hits         3163     2927    -236     
- Misses        290      550    +260     
- Partials       66       67      +1
Impacted Files Coverage Δ
tools/compileProtos.ts 70.83% <70.83%> (ø)
test/compileProtos.ts 93.54% <93.54%> (ø)
...google-gax-packaging-test-app/src/v1beta1/index.js 0% <0%> (-100%) ⬇️
...oogle-gax-packaging-test-app/test/gapic-v1beta1.js 0% <0%> (-95.52%) ⬇️
...-gax-packaging-test-app/src/v1beta1/echo_client.js 0% <0%> (-87.7%) ⬇️
...ixtures/google-gax-packaging-test-app/src/index.js 0% <0%> (-70.12%) ⬇️
tools/prepublish.ts
src/index.ts 91.17% <0%> (+0.26%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b86a2c6...4c41b54. Read the comment docs.

@codecov
Copy link

codecov bot commented Jul 25, 2019

Codecov Report

❗ No coverage uploaded for pull request base (master@a431c63). Click here to learn what that means.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff            @@
##             master     #547   +/-   ##
=========================================
  Coverage          ?   44.44%           
=========================================
  Files             ?       10           
  Lines             ?      693           
  Branches          ?        4           
=========================================
  Hits              ?      308           
  Misses            ?      382           
  Partials          ?        3

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a431c63...dafecb6. Read the comment docs.

Copy link

@bcoe bcoe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mostly nits 👍

excited to see this in action.

if (fileStat.isFile() && file.match(PROTO_LIST_REGEX)) {
result.push(fullPath);
} else if (fileStat.isDirectory()) {
const nested = await findProtoJsonFiles(fullPath);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clever 😄 seems like a nice simple implementation that should work.

'--target',
'json',
'-p',
path.join('node_modules', 'google-gax', 'protos'),
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does google-gax export this as a bin? it's better to execute ./node_modules/.bin/protos if possible; because this will work regardless of hoisting.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a value for -p option of pbjs, it basically points to an output directory, it's not executing anything.

@noahdietz
Copy link

lgtm! Cool stuff Alex

@alexander-fenster
Copy link
Contributor Author

As discussed in the chat: we'll run this script during the generation step and will commit the resulting JSON file so that no new npm script would be needed for the client libraries, and no proto compilation would happen when a client library is installed.

@JustinBeckwith - do you have any concerns, or can you put your formal ✅ on this?

@alexander-fenster alexander-fenster merged commit 1334c6d into master Jul 29, 2019
@alexander-fenster alexander-fenster deleted the compile-protos branch July 29, 2019 16:56
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

cla: yes This human has signed the Contributor License Agreement.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants