Update Node.js from 14.17.15 to 16.13.2 LTS #5095

jwhitlock · 2022-01-28T01:08:32Z

Update Node.js from 14.17.15 to 16.13.2, the LTS version until 2022-10-18.

This was originally part of PR #5084, where it failed on the docker-worker-chunk-X steps. These (mostly) pass after updating the worker-ci image from Ubuntu 14.04 to 20.04, which also includes:

Updating from Docker 18.06.3 to 20.10.12
Implementing a new method for registering the Docker package repo
Switch from GZip to the XZ-compressed Node.js package

jwhitlock · 2022-01-28T01:34:40Z

This is a new one - I haven't seen the decision task fail before. We did recently update it to get the Python 3 version. I'll try again in the morning.

jwhitlock · 2022-01-28T01:43:12Z

Ah, found it - a poorly encoded special quote character in the commit message:

> change in behavior, and it’s still possible to switch modes using the

This showed up as the byte sequence \xe2\x80\x99. I changed it to:

> change in behavior, and it's still possible to switch modes using the

jwhitlock · 2022-01-28T04:13:33Z

I added a commit that updates the worker-ci image to Ubuntu 20.04, and I've pushed the new image to the node16.13.2 tag (which continues to be used by just these node update PRs). 4 of the 5 docker-worker-chunk-X tests passed, and docker-worker-chunk-2 had one failing test, the cat stress test:

taskcluster/workers/docker-worker/test/integration/interactive_test.js

Line 226 in af2ac88

test('cat stress test', async () => {

I'm trying again (making a small change to the commit message and force pushing), since it looks like it is sensitive to timing.

lotas

👍

jwhitlock · 2022-01-28T21:16:44Z

The same test failed on a re-run, so I think this is a real regression. I find it very challenging to determine what is happening in the build step. I'm inclined to disable the test and work on it in a new PR, since everything else is working. However, this could be demonstrating something in Node 16.12.2, or Ubuntu 20.04, that will impact other CI jobs.

I think my next attempt will be to add a new CI step that just runs this test, to help with debugging and remove other distractions. I'll work on that Monday.

jwhitlock · 2022-01-31T23:35:19Z

Made some progress.

The test is waiting forever on this line:

taskcluster/workers/docker-worker/test/integration/interactive_test.js

Line 259 in af2ac88

await new Promise(accept => client.stdout.on('end', accept));

I noticed that this test was similar but different from one labelled "cat". Most importantly, this is the only test that uses DockerExecClient from docker-exec-websocket-client. Which is a SysEng project - https://github.com/taskcluster/docker-exec-websocket-client. All the tests in that project pass on Node.JS v14.17.5, but one fails on v16.13.2. AND, I can run those tests locally rather than push to community-ci.

I filed taskcluster/docker-exec-websocket-client#1, and will try to get that working.

jwhitlock · 2022-02-04T23:41:14Z

After a lot of work, I found and fixed an issue in taskcluster/docker-exec-websocket-server#35. A few steps are needed:

Review and merge docker-exec-websocket-server PR 35
Release a new version of docker-exec-websocket-server
Update Refresh codebase, use Taskcluster CI docker-exec-websocket-client#2 to use that version, get passing tests
Review and merge docker-exec-websocket-client PR 2
Release a new version of docker-exec-websocket-client
Rebase and update this PR to use both, get tests to pass

jwhitlock · 2022-02-08T01:28:01Z

With the 3.0.0 versions of docker-exec-websocket-client and docker-exec-websocket-server, the cat stress test is still failing, but no longer stuck waiting for the Promise. Now it is failing because the output does not match the input. I'll debug in the morning.

From the release notes: > As of Node.js 15, the default mode for `unhandledRejection` is changed > to `throw` (from `warn`). In throw mode, if an `unhandledRejection` > hook is not set, the `unhandledRejection` is raised as an uncaught > exception. Users that have an `unhandledRejection` hook should see no > change in behavior, and it's still possible to switch modes using the > `--unhandled-rejections=mode` process flag. The Node.js 14 behaviour is available with `--unhandled-rejections=warn`

Updating from 14.04 to 20.04, the current LTS version. This includes: * Updating from Docker 18.06.3 to 20.10.12 * Implementing a new method for registering the Docker package repo * Switch from GZip to the XZ-compressed Node.js package

jwhitlock · 2022-02-08T22:30:32Z

As discussed at #5154 (comment), the problem was that the output does match the input, but it didn't before... The test now passes under Node v14, and this is rebased to include the docker-exec-websocket-* libraries and the corrected test.

jwhitlock · 2022-02-08T22:43:40Z

So far one error that looks like an intermittent. In service-secrets, the tests passed, but failed to upload the coverage report:

[taskcluster:error] Error uploading "public/coverage-final.json" artifact. getaddrinfo EAI_AGAIN community-tc.services.mozilla.com

I could re-run, but I think its ready to merge.

matt-boris

LG(reat)TM! Thanks John for this huge effort right as you're transitioning off the team. This is awesome. 👏🏻 🎉

matt-boris · 2022-02-08T22:48:07Z

Lol v16.14.0 was just released 😂

jwhitlock · 2022-02-08T22:55:42Z

The diff doesn't show that there was a huge effort, which is how it should be for most Node updates. I hope the 16.14.0 is just a matter of following the docs!

jwhitlock requested a review from a team as a code owner January 28, 2022 01:08

jwhitlock requested review from lotas and petemoore January 28, 2022 01:08

jwhitlock marked this pull request as draft January 28, 2022 01:08

jwhitlock force-pushed the node-16.13.2 branch 2 times, most recently from ccf1b3d to dc9bc02 Compare January 28, 2022 01:17

jwhitlock force-pushed the node-16.13.2 branch from dc9bc02 to eeb5da5 Compare January 28, 2022 01:38

jwhitlock force-pushed the node-16.13.2 branch from 74b0dff to ddc84e7 Compare January 28, 2022 04:13

lotas approved these changes Jan 28, 2022

View reviewed changes

jwhitlock force-pushed the node-16.13.2 branch 3 times, most recently from 0586bee to bd69216 Compare January 31, 2022 22:52

jwhitlock mentioned this pull request Feb 1, 2022

Update to Go 1.17.6, Node.js 16.13.2 #5084

Closed

jwhitlock force-pushed the node-16.13.2 branch from bd69216 to 8d1acbc Compare February 7, 2022 23:12

jwhitlock mentioned this pull request Feb 8, 2022

Update to docker-exec-websocket-* 3.0.0 #5154

Merged

jwhitlock added 5 commits February 8, 2022 16:18

Update Node.js from 14.17.15 to 16.13.2 LTS

9fc6a4e

Remove temporary fix for node 14.17.5 issue

6aec0a4

Update the worker-ci image to Ubuntu 20.04

1aad5ad

Updating from 14.04 to 20.04, the current LTS version. This includes: * Updating from Docker 18.06.3 to 20.10.12 * Implementing a new method for registering the Docker package repo * Switch from GZip to the XZ-compressed Node.js package

Add yarn install to Node.js upgrade

7ca751d

jwhitlock force-pushed the node-16.13.2 branch from c0a4532 to 7ca751d Compare February 8, 2022 22:18

jwhitlock marked this pull request as ready for review February 8, 2022 22:40

jwhitlock requested a review from lotas February 8, 2022 22:43

matt-boris approved these changes Feb 8, 2022

View reviewed changes

jwhitlock merged commit 4f8e697 into main Feb 8, 2022

jwhitlock deleted the node-16.13.2 branch February 8, 2022 22:55

Update Node.js from 14.17.15 to 16.13.2 LTS #5095

Update Node.js from 14.17.15 to 16.13.2 LTS #5095

Uh oh!

Conversation

jwhitlock commented Jan 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jwhitlock commented Jan 28, 2022

Uh oh!

jwhitlock commented Jan 28, 2022

Uh oh!

jwhitlock commented Jan 28, 2022

Uh oh!

lotas left a comment

Choose a reason for hiding this comment

Uh oh!

jwhitlock commented Jan 28, 2022

Uh oh!

jwhitlock commented Jan 31, 2022

Uh oh!

jwhitlock commented Feb 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jwhitlock commented Feb 8, 2022

Uh oh!

jwhitlock commented Feb 8, 2022

Uh oh!

jwhitlock commented Feb 8, 2022

Uh oh!

matt-boris left a comment

Choose a reason for hiding this comment

Uh oh!

matt-boris commented Feb 8, 2022

Uh oh!

jwhitlock commented Feb 8, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jwhitlock commented Jan 28, 2022 •

edited

Loading

jwhitlock commented Feb 4, 2022 •

edited

Loading