Skip to content

🏗✨ Dynamically generate custom closure compiler runner.jar#23434

Merged
rsimha merged 6 commits intoampproject:masterfrom
rsimha:2019-07-19-GenerateRunner
Jul 24, 2019
Merged

🏗✨ Dynamically generate custom closure compiler runner.jar#23434
rsimha merged 6 commits intoampproject:masterfrom
rsimha:2019-07-19-GenerateRunner

Conversation

@rsimha
Copy link
Copy Markdown
Contributor

@rsimha rsimha commented Jul 19, 2019

In #23417, we switched from the custom-built third_party/closure-compiler/compiler.jar to the natively installed node_modules/google-closure-compiler-java/compiler.jar. With this, the coast is clear to dynamically generate runner.jar instead of checking in a pre-built copy.

PR highlights:

  • Adds a copy of Apache Ant v1.10.6 to third_party/ant/ Edit: Moved to 🏗 Add Apache Ant v1.10.6 to third_party/ant #23490
  • Removes all unnecessary files from build-system/runner including the pre-built runner.jar
  • Adds build-system/tasks/generate-runner.js, which dynamically generates runner.jar whenever a gulp task needs to invoke closure compiler (dist, check-types, and nailgun-start)

Notes:

  • The dynamically generated runner.jar is written to a subdirectory of build-system/runner/dist/ in order to allow for the concurrent running of gulp dist and gulp check-types
  • Information about the commit at which runner.jar was last generated is written to the directory, so that it is only regenerated when required
  • It is no longer necessary to manually update runner.jar while upgrading closure compiler
  • Changes to build-system/runner/src/org/ampproject/*.java are automatically applied when you run gulp dist or gulp check-types
  • New releases of closure compiler will automatically create a Renovate PR whose Travis build will generate and test a new runner.jar

Fixes #22452

@rsimha
Copy link
Copy Markdown
Contributor Author

rsimha commented Jul 22, 2019

@jridgewell and @estherkim have helped verify that this works on older machines, and machines that have never had to run ant before. (Thanks!)

This is now ready for review.

@jridgewell
Copy link
Copy Markdown
Contributor

A few questions:

  • Does this have to be generated every time?
  • Why is a checked-in ant better than a checked in runner?

@rsimha
Copy link
Copy Markdown
Contributor Author

rsimha commented Jul 22, 2019

@jridgewell answers below

  • Does this have to be generated every time?
  • There's no good way to tell if there are new changes (local edits? synced changes?) in build-system/runner/ that require runner.jar to be regenerated
  • With this PR, we generate / use a separate instance of runner.jar for each instance of nailgun-server.jar which allows you to concurrently run gulp dist and gulp check-types
  • Generating runner.jar takes ~1s to run, and does not materially increase the running time of gulp tasks that use it
  • Why is a checked-in ant better than a checked in runner?
  • With a checked-in runner.jar, updating custom closure code requires these steps:
    1. Edit .java file(s) in build-system/runner
    2. Install the latest version of ant
    3. Set the default JAVA version on your machine (see Fix hardcoded CSS filename in closure compiler code. #22485 (comment))
    4. Run ant jar to generate a new runner.jar
    5. Commit, test, merge
  • With a checked-in ant, updating custom closure code becomes simpler:
    1. Edit .java file(s) in build-system/runner
    2. Commit, test, merge

@jridgewell
Copy link
Copy Markdown
Contributor

I don't think this is a good change to make.

  • There's no good way to tell if there are new changes (local edits? synced changes?) in build-system/runner/ that require runner.jar to be regenerated
  • Generating runner.jar takes ~1s to run, and does not materially increase the running time of gulp tasks that use it

That's just unnecessary extra work, though. (And it took 4s on my machine). I understand that it's small compared to overall closure run time, so this isn't a blocker.

  • With this PR, we generate / use a separate instance of runner.jar for each instance of nailgun-server.jar which allows you to concurrently run gulp dist and gulp check-types

Running closure makes my machine usable for ~5min. Running both dist and check-types at the same time would make my machine even more unusable.

  • With a checked-in runner.jar, updating custom closure code requires these steps:
  • With a checked-in ant, updating custom closure code becomes simpler:

This is really only run buy You, Erwin, and myself. I think a checked in bash script to automate this would be better.

@rsimha
Copy link
Copy Markdown
Contributor Author

rsimha commented Jul 22, 2019

This is really only run buy You, Erwin, and myself. I think a checked in bash script to automate this would be better.

Requiring a bash script to be run manually will prevent the real benefit of this PR from being realized: The ability to automatically test new closure compiler versions on Travis as they are released (via Renovate PRs). Today, upgrading closure compiler is a painstaking manual process that happens only when someone sets out to do so. Dynamically generating runner.jar will significantly reduce the friction involved.

That's just unnecessary extra work, though.

It's a necessary part of enabling the automatic testing of new releases of closure compiler.

Running closure makes my machine usable for ~5min. Running both dist and check-types at the same time would make my machine even more unusable.

This is only a problem on older machines. Running concurrent instances of closure is entirely possible on current macbooks and standard linux workstations, and is a fairly useful workflow if you choose to do so. Dynamically generating runner.jar happens prior to launching the compiler, and shouldn't affect this aspect.

@jridgewell
Copy link
Copy Markdown
Contributor

Requiring a bash script to be run manually will prevent the real benefit of this PR from being realized: The ability to automatically test new closure compiler versions on Travis as they are released (via Renovate PRs)

That's also easily done via a gulp task before running?

This is only a problem on old machines like yours. Running concurrent instances of closure is entirely possible on current macbooks and standard linux workstations, and is a fairly useful workflow.

Man, I wish Apple would make a decent laptop again.

Current gen: https://browser.geekbench.com/v4/cpu/13995682
Mine: https://browser.geekbench.com/v4/cpu/13995433

@rsimha
Copy link
Copy Markdown
Contributor Author

rsimha commented Jul 22, 2019

That's also easily done via a gulp task before running?

As discussed in person, this isn't possible, since it will result in the need to add a new commit to a Github PR from a Travis build.

@rsimha
Copy link
Copy Markdown
Contributor Author

rsimha commented Jul 23, 2019

@jridgewell I've added commit dd9431d, which generates runner.jar only in the following cases:

  1. runner.jar hasn't been generated before
  2. The first time a new commit is pulled from master
  3. There are local changes to build-system/runner/

Let me know what you think.

@rsimha
Copy link
Copy Markdown
Contributor Author

rsimha commented Jul 23, 2019

@jridgewell @erwinmombay All comments addressed. Third party changes separated out into #23490. PTAL.

@erwinmombay
Copy link
Copy Markdown
Member

just 1 last nit from me. I'll leave it to @jridgewell for final approval since i know he has concerns above

Copy link
Copy Markdown
Contributor

@jridgewell jridgewell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still don't like this.

@rsimha
Copy link
Copy Markdown
Contributor Author

rsimha commented Jul 24, 2019

I still don't like this.

What specifically don't you like? Do you have a better idea that will still allow renovate to upgrade closure compiler?

I'm merging this for now. Happy to discuss improvements in a follow up.

Edit: Seems like you didn't see #23434 (comment) which prevents a regeneration every time closure is invoked.

@rsimha rsimha merged commit d2909bb into ampproject:master Jul 24, 2019
@rsimha rsimha deleted the 2019-07-19-GenerateRunner branch July 24, 2019 05:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dynamically generate custom closure compiler for AMP

4 participants