Skip to content

Workaround selenium chromium binary detection failure resulting in SessionNotCreatedError when running Jasmine specs#724

Closed
floehopper wants to merge 1 commit intomainfrom
workaround-selenium-webdriver-failure-to-detect-chromium-browser-binary
Closed

Workaround selenium chromium binary detection failure resulting in SessionNotCreatedError when running Jasmine specs#724
floehopper wants to merge 1 commit intomainfrom
workaround-selenium-webdriver-failure-to-detect-chromium-browser-binary

Conversation

@floehopper
Copy link
Contributor

A number of people have been seeing the following error when running Jasmine specs (which use the selenium webdriver node package) in a govuk-docker container on Apple Silicon for a number of different apps:

SessionNotCreatedError: session not created:
  Chrome failed to start: exited normally.

See this gist for an example of the full output. Note that the problem has not been happening:

Since v4.6, selenium has shipped with a selenium-manager CLI app written in Rust which aims to act as a fallback provider of browser and driver binaries if they aren't found in the PATH. I assume this is why this has only become an issue relatively recently - I first saw it in Signon at least a couple of months ago and have been working around it since!

For some reason the selenium-manager CLI app is not finding the chromium binary installed from the chromium Debian package by Dockerfile.govuk-base even though it exists in /usr/bin which is in the PATH. However, it does find the chromedriver binary which I assume is also installed into /usr/bin from the chromium-driver Debian package. I can only assume this is due to the name including "chrome" rather than "chromium". You can see this in action by running the selenium-manager CLI app inside an app container as follows:

root@c4d5514eee6a:/govuk/signon# node_modules/selenium-webdriver/bin/linux/selenium-manager --browser chrome --debug --clear-cache
DEBUG	Clearing cache at: /root/.cache/selenium
DEBUG	Found chromedriver 120.0.6099.224 in PATH: /usr/bin/chromedriver
DEBUG	chrome not found in PATH
DEBUG	chrome not found in the system
DEBUG	Discovering versions from https://googlechromelabs.github.io/chrome-for-testing/last-known-good-versions-with-downloads.json
DEBUG	Required browser: chrome 121.0.6167.85
DEBUG	Downloading chrome 121.0.6167.85 from https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/121.0.6167.85/linux64/chrome-linux64.zip
DEBUG	chrome 121.0.6167.85 has been downloaded at /root/.cache/selenium/chrome/linux64/121.0.6167.85/chrome
DEBUG	Discovering versions from https://googlechromelabs.github.io/chrome-for-testing/known-good-versions-with-downloads.json
DEBUG	Required driver: chromedriver 121.0.6167.85
WARN	The chromedriver version (120.0.6099.224) detected in PATH at /usr/bin/chromedriver might not be compatible with the detected chrome version (121.0.6167.85); currently, chromedriver 121.0.6167.85 is recommended for chrome 121.*, so it is advised to delete the driver in PATH and retry
INFO	Driver path: /usr/bin/chromedriver
INFO	Browser path: /root/.cache/selenium/chrome/linux64/121.0.6167.85/chrome

You can see that while it does find /usr/bin/chromedriver, it does not find /usr/bin/chromium. This means that it downloads a chrome binary from the Chrome for Testing website and caches it in /root/.cache/selenium/chrome/linux64/121.0.6167.85/chrome. This binary does not work in a govuk-docker container running on Apple Silicon - I see the following error if I try to run it:

qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory

This is the underlying cause of the SessionNotCreatedError when running the Jasmine specs.

Although it might not be the best solution, I've worked around the problem by creating a symbolic link for /usr/bin chrome (which is what selenium-manager seems to expect to find) to /usr/bin/chromium which is what is installed by the chromium Debian package.

I did try changing the browser value in spec/support/jasmine-browser.json in the relevant app from "headlessChrome" to e.g. "headlessChromium" or "chromium", but nothing seemed to work. And looking at the node package source code, it's not obvious that there is any explicit support for "chromium" as opposed to "chrome". However, it might be worth pursuiing this with the maintainers of the jasmine-browser-runner and/or selenium webdriver node packages.

I also tried setting the SE_BROWSER_PATH environment variable mentioned in these Selenium docs, but that didn't seem to work either.

Anyway, my workaround is the best option I've come up with so far, but I'd welcome any other suggestions!

I suspect the reason why this problem is not affecting non-Apple Silicon devices is that the downloaded chrome binary does work on those devices. However, it's worth noting that the same problem is almost certainly why the following warning message is appearing on those devices:

The chromedriver version (120.0.6099.199) detected in PATH at
/usr/bin/chromedriver might not be compatible with the detected chrome
version (121.0.6167.85); currently, chromedriver 121.0.6167.85 is
recommended for chrome 121.*, so it is advised to delete the driver in
PATH and retry

My workaround should fix that warning too, because the Jasmine specs should use the chromium binary installed by the chromium Debian package as well as the chromedriver binary installed by the chromium-driver Debian package which are compatible versions.

I suspect the reason why this problem is not affecting Ruby tests which use the selenium-webdriver gem is that unlike the node pacakge it has explicit support for chromium.

I suspect the reason why this problem is not affecting CI builds which use the govuk-infrastructure jasmine workflow is that it runs in a GitHub Action runner which runs on ubuntu-latest and I suspect that provides a chrome binary.

A number of people have been seeing the following error when running
Jasmine specs (which use the selenium-webdriver node package) in a
govuk-docker container on Apple Silicon for a number of different apps:

    SessionNotCreatedError: session not created:
      Chrome failed to start: exited normally.

See this gist [1] for an example of the full output. Note that the
problem has *not* been happening:

* on non-Apple Silicon devices
* in Ruby tests which use the selenium-webdriver gem
* in CI builds which use the govuk-infrastructure jasmine workflow [2]

Since v4.6, selenium has shipped with a selenium-manager CLI app [3]
written in Rust which aims to act as a fallback provider of browser and
driver binaries if they aren't found in the `PATH`. I assume this is why
this has only become an issue _relatively_ recently - I first saw it in
Signon at least a couple of months ago and have been working around it
since!

For some reason the selenium-manager CLI app is not finding the
`chromium` binary installed from the `chromium` Debian package by
`Dockerfile.govuk-base` [4] even though it exists in `/usr/bin` which is
in the `PATH`. However, it *does* find the `chromedriver` binary which I
assume is also installed into `/usr/bin` from the `chromium-driver`
Debian package. I can only assume this is due to the name including
"chrome" rather than "chromium". You can see this in action by running
the selenium-manager CLI app inside an app container as follows:

    root@c4d5514eee6a:/govuk/signon# node_modules/selenium-webdriver/bin/linux/selenium-manager --browser chrome --debug --clear-cache
    DEBUG	Clearing cache at: /root/.cache/selenium
    DEBUG	Found chromedriver 120.0.6099.224 in PATH: /usr/bin/chromedriver
    DEBUG	chrome not found in PATH
    DEBUG	chrome not found in the system
    DEBUG	Discovering versions from https://googlechromelabs.github.io/chrome-for-testing/last-known-good-versions-with-downloads.json
    DEBUG	Required browser: chrome 121.0.6167.85
    DEBUG	Downloading chrome 121.0.6167.85 from https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/121.0.6167.85/linux64/chrome-linux64.zip
    DEBUG	chrome 121.0.6167.85 has been downloaded at /root/.cache/selenium/chrome/linux64/121.0.6167.85/chrome
    DEBUG	Discovering versions from https://googlechromelabs.github.io/chrome-for-testing/known-good-versions-with-downloads.json
    DEBUG	Required driver: chromedriver 121.0.6167.85
    WARN	The chromedriver version (120.0.6099.224) detected in PATH at /usr/bin/chromedriver might not be compatible with the detected chrome version (121.0.6167.85); currently, chromedriver 121.0.6167.85 is recommended for chrome 121.*, so it is advised to delete the driver in PATH and retry
    INFO	Driver path: /usr/bin/chromedriver
    INFO	Browser path: /root/.cache/selenium/chrome/linux64/121.0.6167.85/chrome

You can see that while it does find `/usr/bin/chromedriver`, it does
*not* find `/usr/bin/chromium`. This means that it downloads a chrome
binary from the Chrome for Testing website [5] and caches it in
`/root/.cache/selenium/chrome/linux64/121.0.6167.85/chrome`. This binary
does not work in a govuk-docker container running on Apple Silicon - I
see the following error if I try to run it:

    qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory

This is the underlying cause of the SessionNotCreatedError when running
the Jasmine specs.

Although it might not be the best solution, I've worked around the
problem by creating a symbolic link for `/usr/bin/chrome` (which is what
selenium-manager seems to expect to find) to `/usr/bin/chromium` which
is what is installed by the chromium Debian package.

I did try changing the `browser` value in
`spec/support/jasmine-browser.json` in the relevant app from
"headlessChrome" to e.g. "headlessChromium" or "chromium", but nothing
seemed to work. And looking at the node package source code, it's not
obvious that there is any _explicit_ support for "chromium" as opposed
to "chrome". However, it might be worth pursuiing this with the
maintainers of the jasmine-browser-runner and/or selenium-webdriver node
packages.

I also tried setting the `SE_BROWSER_PATH` environment variable
mentioned in the Selenium docs [6], but that didn't seem to work either.

Anyway, my workaround is the best option I've come up with so far, but
I'd welcome any other suggestions!

I suspect the reason why this problem is not affecting non-Apple Silicon
devices is that the downloaded chrome binary *does* work on those
devices. However, it's worth noting that the same problem is almost
certainly why the following warning message is appearing on those
devices:

    The chromedriver version (120.0.6099.199) detected in PATH at
    /usr/bin/chromedriver might not be compatible with the detected chrome
    version (121.0.6167.85); currently, chromedriver 121.0.6167.85 is
    recommended for chrome 121.*, so it is advised to delete the driver in
    PATH and retry

My workaround should fix that warning too, because the Jasmine specs
should use the chromium binary installed by the chromium Debian package
as well as the chromedriver binary installed by the chromium-driver
Debian package which are compatible versions.

I suspect the reason why this problem is not affecting Ruby tests which
use the selenium-webdriver gem is that unlike the node pacakge it has
explicit support for chromium [7].

I suspect the reason why this problem is not affecting CI builds which
use the govuk-infrastructure jasmine workflow is that it runs in a
GitHub Action runner which runs on `ubuntu-latest` and I suspect that
provides a `chrome` binary.

[1]: https://gist.github.com/floehopper/5531ac99ae615e9b5b03b97cb86dfdf0
[2]: https://github.com/alphagov/govuk-infrastructure/blob/af441f9eaab9afad7367f338b1d7e7b269f1642c/.github/workflows/jasmine.yml
[3]: https://www.selenium.dev/documentation/selenium_manager/
[4]: https://github.com/alphagov/govuk-docker/blob/a1bb1eb0f3e8fc415d26a1f4dc1d49a9bdf3e3ab/Dockerfile.govuk-base#L10-L11
[5]: https://googlechromelabs.github.io/chrome-for-testing
[6]: https://www.selenium.dev/documentation/selenium_manager/#configuration
[7]: https://github.com/SeleniumHQ/selenium/tree/trunk/rb/lib/selenium/webdriver/chromium
@richardTowers
Copy link

Looks like selenium-manager had an issue for this recently, merged Oct 7, 2023. Maybe a newer release of selenium would also resolve this? (If there is one that's new enough)

@richardTowers
Copy link

I guess selenium itself (and therefore selenium-manager) is brought in by the node package. At least for frontend's yarn.lock file it looks like there's an opportunity to try out a newer version of selenium webdriver (we're on 4.12 from ~5 months ago, latest is 4.17).

Copy link

@richardTowers richardTowers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💬 Some comments suggesting another idea, but if they don't work this workaround seems like a reasonable thing to do to me.

@floehopper
Copy link
Contributor Author

Looks like selenium-manager had an issue for this recently, merged Oct 7, 2023. Maybe a newer release of selenium would also resolve this? (If there is one that's new enough)

Ah, yes - I'd somehow missed that.

I guess selenium itself (and therefore selenium-manager) is brought in by the node package. At least for frontend's yarn.lock file it looks like there's an opportunity to try out a newer version of selenium webdriver (we're on 4.12 from ~5 months ago, latest is 4.17).

Yes, even though I hadn't seen the issue you mentioned above, I thought I had tried a newer version of the selenium-webdriver node package. However, I must've either not picked the latest version or done something wrong, because forcing the version to v4.17.0 does indeed fix the problem at least in Signon:

// package.json
  "resolutions": {
    "selenium-webdriver": "4.17.0",
    // ...
  }

With this in place, I can see that selenium-manager finds the chromium binary successfully without the symlink:

root@fd75306bbc07:/govuk/signon# node_modules/selenium-webdriver/bin/linux/selenium-manager --browser chrome --debug --clear-cache
DEBUG	Clearing cache at: /root/.cache/selenium
DEBUG	Found chromedriver 120.0.6099.224 in PATH: /usr/bin/chromedriver
DEBUG	Found chromium in PATH: /usr/bin/chromium
DEBUG	Running command: /usr/bin/chromium --version
DEBUG	Output: "Chromium 120.0.6099.224 built on Debian 11.8, running on Debian 11.9"
DEBUG	Detected browser: chrome 120.0.6099.224
DEBUG	Discovering versions from https://googlechromelabs.github.io/chrome-for-testing/known-good-versions-with-downloads.json
DEBUG	Required driver: chromedriver 120.0.6099.109
INFO	Driver path: /usr/bin/chromedriver
INFO	Browser path: /usr/bin/chromium

And runing the jasmine specs works fine:

root@fd75306bbc07:/govuk/signon# yarn run jasmine-browser-runner runSpecs
yarn run v1.22.19
$ /govuk/signon/node_modules/.bin/jasmine-browser-runner runSpecs
Jasmine server is running here: http://localhost:45119
Jasmine tests are here:         /govuk/signon/spec/javascripts
Source files are here:          /govuk/signon/public/assets/signon
Running tests in the browser...
Randomized with seed 52647
Started
..


2 specs, 0 failures
Finished in 0.016 seconds
Randomized with seed 52647 (jasmine-browser-runner runSpecs --seed=52647)
Done in 1.07s.

I haven't been able to work out the exact version of selenium-webdriver includes the fix - the selenium-manager Rust app seems to have its own versioning scheme - the fix appears in v0.4.14. However, using the latest version seems reasonable.

So I'll open a Signon PR for the above change, close this PR, and open an issue/PR on the jasmine-browser-runner node package which hasn't been updated in 5 months.

floehopper added a commit to alphagov/signon that referenced this pull request Feb 16, 2024
We've been seeing the following error running the Jasmine specs in a
govuk-docker container running on Apple silicon:

    SessionNotCreatedError: session not created:
      Chrome failed to start: exited normally.

For a full diagnosis of the problem see this govuk-docker PR [1] which
was a first attempt at fixing the problem.

Note that the problem was also triggering a warning like the following
even on non-Apple silicon:

    The chromedriver version (120.0.6099.199) detected in PATH at
    /usr/bin/chromedriver might not be compatible with the detected chrome
    version (121.0.6167.85); currently, chromedriver 121.0.6167.85 is
    recommended for chrome 121.*, so it is advised to delete the driver in
    PATH and retry

This PR forces yarn to install v4.17.0 of the `selenium-webdriver` node
package which includes a fix [2] to the `selenium-manager` CLI app
included in the `selenium-webdriver` node package so that it now
correctly finds the chromium binary. This fixes both the error on Apple
silicon and the warning on all devices.

[1]: alphagov/govuk-docker#724
[2]: SeleniumHQ/selenium#12890
@floehopper
Copy link
Contributor Author

Closing in favour of alphagov/signon#2723

@floehopper floehopper closed this Feb 16, 2024
@floehopper floehopper deleted the workaround-selenium-webdriver-failure-to-detect-chromium-browser-binary branch February 16, 2024 13:04
@floehopper
Copy link
Contributor Author

I've now opened a PR on jasmine-browser-runner to bump its version of selenium-webdriver. Once that PR is merged and released, we should be able to remove the resolution entry for selenium-webdriver in Signon and any other apps where it's been added.

floehopper added a commit to alphagov/signon that referenced this pull request Feb 16, 2024
We've been seeing the following error running the Jasmine specs in a
govuk-docker container running on Apple silicon:

    SessionNotCreatedError: session not created:
      Chrome failed to start: exited normally.

For a full diagnosis of the problem see this govuk-docker PR [1] which
was a first attempt at fixing the problem.

Note that the problem was also triggering a warning like the following
even on non-Apple silicon:

    The chromedriver version (120.0.6099.199) detected in PATH at
    /usr/bin/chromedriver might not be compatible with the detected chrome
    version (121.0.6167.85); currently, chromedriver 121.0.6167.85 is
    recommended for chrome 121.*, so it is advised to delete the driver in
    PATH and retry

This PR forces yarn to install v4.17.0 of the `selenium-webdriver` node
package which includes a fix [2] to the `selenium-manager` CLI app
included in the `selenium-webdriver` node package so that it now
correctly finds the chromium binary. This fixes both the error on Apple
silicon and the warning on all devices.

[1]: alphagov/govuk-docker#724
[2]: SeleniumHQ/selenium#12890
richardTowers pushed a commit to alphagov/whitehall that referenced this pull request Feb 16, 2024
We've been seeing the following error running the Jasmine specs in a
govuk-docker container running on Apple silicon:

    SessionNotCreatedError: session not created:
      Chrome failed to start: exited normally.

For a full diagnosis of the problem see this govuk-docker PR [1] which
was a first attempt at fixing the problem.

Note that the problem was also triggering a warning like the following
even on non-Apple silicon:

    The chromedriver version (120.0.6099.199) detected in PATH at
    /usr/bin/chromedriver might not be compatible with the detected chrome
    version (121.0.6167.85); currently, chromedriver 121.0.6167.85 is
    recommended for chrome 121.*, so it is advised to delete the driver in
    PATH and retry

This PR forces yarn to install v4.17.0 of the `selenium-webdriver` node
package which includes a fix [2] to the `selenium-manager` CLI app
included in the `selenium-webdriver` node package so that it now
correctly finds the chromium binary. This fixes both the error on Apple
silicon and the warning on all devices.

[1]: alphagov/govuk-docker#724
[2]: SeleniumHQ/selenium#12890

Co-Authored-By: <james.mead@digital.cabinet-office.gov.uk>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants