[build/docker] Add support for centos ARM builds by jbudz · Pull Request #84831 · elastic/kibana

jbudz · 2020-12-02T20:23:13Z

This adds support for passing an architecture flag to the docker
builder, enabling centos ARM package builds on native machines. The
changes here include:

Output files now include the architecture, e.g.
kibana-8.0.0-SNAPSHOT-docker-image.tar.gz -> kibana-8.0.0-SNAPSHOT-arm64-docker-image.tar.gz
nss is installed instead of libnss3.so. libnss3.so was not
available.

On an arm64 host, generating the image can be done with yarn build. On an x64 host, only the build context will be generated.

Closes #72884

Release note:
Adds docker images for ARM64 based operating systems

This adds support for passing an architecture flag to the docker builder, enabling centos ARM package builds on native machines. The changes here include: 1) Output files now include the architecture, e.g. kibana-8.0.0-SNAPSHOT-docker-image.tar.gz -> kibana-8.0.0-SNAPSHOT-arm64-docker-image.tar.gz 2) nss is installed instead of libnss3.so. libnss3.so was not available. Because this requires an ARM build machine, it's off by default. Building can be done with `yarn build --skip-os-packages --docker-arm`

jbudz · 2020-12-02T21:51:33Z

https://s3-us-west-1.amazonaws.com/kibana.budzenski.com/84831/f604e34/kibana-8.0.0-SNAPSHOT-arm64-docker-image.tar.gz

I tested a basic install with reporting - so far we're functioning.

elasticmachine · 2020-12-02T21:51:47Z

Pinging @elastic/kibana-operations (Team:Operations)

…ecture

tylersmalley · 2020-12-04T00:44:22Z

Chatted with @jbudz, he is going to change the docker builds to always build the context. This will allow us to use a downstream job to build the actual image artifact on an ARM machine.

jbudz · 2020-12-07T18:44:47Z

Chatted with @jbudz, he is going to change the docker builds to always build the context. This will allow us to use a downstream job to build the actual image artifact on an ARM machine.

Pushed with c10c26d. Removed the docker ARM specific flag, it'll generate all docker contexts and build the image corresponding to the host.

jbudz · 2020-12-08T18:44:50Z

@elasticmachine merge upstream

jbudz · 2020-12-08T20:31:09Z

I'm doing a secondary full integration sweep. ARM is fine so far. All bundle contexts were generated. amd64 image will be up shortly, but otherwise this LGTM for reviews.

ubuntu@ip-172-31-26-157:~/kibana$ docker load --input target/kibana-8.0.0-SNAPSHOT-arm64-docker-image.tar.gz
Loaded image: docker.elastic.co/kibana/kibana:8.0.0-SNAPSHOT-arm64
ubuntu@ip-172-31-26-157:~/kibana$ docker run -it docker.elastic.co/kibana/kibana:8.0.0-SNAPSHOT-arm64
  log   [20:25:29.436] [info][plugins-service] Plugin "visTypeXy" is disabled.

https://s3-us-west-1.amazonaws.com/kibana.budzenski.com/84831/353a305/kibana-8.0.0-SNAPSHOT-arm64-docker-image.tar.gz
https://s3-us-west-1.amazonaws.com/kibana.budzenski.com/84831/353a305/kibana-8.0.0-SNAPSHOT-arm64-docker-build-context.tar.gz
https://s3-us-west-1.amazonaws.com/kibana.budzenski.com/84831/353a305/kibana-8.0.0-SNAPSHOT-amd64-docker-build-context.tar.gz

jbudz · 2020-12-08T22:37:50Z

https://s3-us-west-1.amazonaws.com/kibana.budzenski.com/84831/353a305/kibana-8.0.0-SNAPSHOT-amd64-docker-image.tar.gz

tylersmalley · 2020-12-09T00:10:50Z

src/dev/build/tasks/os_packages/create_os_package_tasks.ts

  async run(config, log, build) {
    // Builds Docker targets for default and oss
-    await runDockerGenerator(config, log, build);
+    await runDockerGenerator(config, log, build, false, 'amd64');


Can we use x64 and aarch64 to maintain consistency with Node the the build?

https://github.com/elastic/kibana/blob/master/src/dev/build/lib/platform.ts#L59-L64

Added with 4cccd58 and the few followup commits for fixes.

jbudz · 2020-12-14T20:48:43Z

@elasticmachine merge upstream

jmlrt · 2021-01-21T08:46:11Z

docker-build-context-aarch64 should use ARM dependencies

@jbudz, sorry I misread this, thanks @tylersmalley for noticing it, the requirements are not to create a new docker-build-context-aarch64 but to update the existing context to support both architecture like in ES context.

jbudz · 2021-01-21T16:30:15Z

^ pushed with 034d478

jbudz · 2021-01-21T16:30:22Z

@elasticmachine merge upstream

jbudz · 2021-01-25T15:45:28Z

@elasticmachine merge upstream

tylersmalley · 2021-01-27T19:26:32Z

kibana-8.0.0-SNAPSHOT-docker-build-context.tar.gz

Testing:

mkdir test; cd test
wget https://tsmalley-elastic.s3-us-west-2.amazonaws.com/pr-84831/kibana-8.0.0-SNAPSHOT-docker-build-context.tar.gz
tar -xzf kibana-8.0.0-SNAPSHOT-docker-build-context.tar.gz
// update line 18 of Dockerfile to reference https://tsmalley-elastic.s3-us-west-2.amazonaws.com/pr-84831
docker build . -t docker.elastic.co/kibana/kibana:8.0.0-SNAPSHOT-aarch64
curl -L https://snapshots.elastic.co/8.0.0-e2ce6361/downloads/elasticsearch/elasticsearch-8.0.0-SNAPSHOT-docker-image-aarch64.tar.gz | docker load`
docker network create elastic
docker run -it --network="elastic" --name=elasticsearch -e discovery.type="single-node" docker.elastic.co/elasticsearch/elasticsearch:8.0.0-SNAPSHOT
docker run -it --network="elastic" -e "ELASTICSEARCH_HOSTS=http://elasticsearch:9200" -e "ELASTICSEARCH_USERNAME=kibana_system" -e "ELASTICSEARCH_PASSWORD=changeme" -p 5601:5601 docker.elastic.co/kibana/kibana:8.0.0-SNAPSHOT-aarch64

tylersmalley · 2021-01-27T20:10:24Z

So far this looks good, however I am blocked from fully testing this against a snapshot build of Elasticsearch. I have pinged Core. cc @rudolf

  log   [20:00:51.028] [error][savedobjects-service] [.kibana_task_manager] [invalid_index_name_exception]: Invalid index name [.kibana_task_manager_8.0.0-SNAPSHOT_001], must be lowercase
  log   [20:00:51.029] [error][savedobjects-service] [.kibana_task_manager] migration failed, dumping execution log:
  log   [20:00:51.030] [info][savedobjects-service] [.kibana_task_manager] INIT RESPONSE
  log   [20:00:51.031] [info][savedobjects-service] [.kibana_task_manager] INIT -> CREATE_NEW_TARGET
  log   [20:00:51.034] [fatal][root] Error: Unable to complete saved object migrations for the [.kibana_task_manager] index. Please check the health of your Elasticsearch cluster and try again. ResponseError: invalid_index_name_exception
    at migrationStateActionMachine (/usr/share/kibana/src/core/server/saved_objects/migrationsv2/migrations_state_action_machine.js:141:11)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async Promise.all (index 1)
    at SavedObjectsService.start (/usr/share/kibana/src/core/server/saved_objects/saved_objects_service.js:163:7)
    at Server.start (/usr/share/kibana/src/core/server/server.js:283:31)
    at Root.start (/usr/share/kibana/src/core/server/root/index.js:58:14)
    at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:100:5)
    at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:169:5)
  log   [20:00:51.040] [info][plugins-system] Stopping all plugins.

tylersmalley

LGTM - fully tested by disabling migrations v2.

kibanamachine · 2021-01-27T23:40:56Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 288710e

Metrics [docs]

✅ unchanged

History

💚 Build #101569 succeeded 6ab6178
💚 Build #100749 succeeded 1a6f2cd
💚 Build #100437 succeeded 33eab33
💔 Build #100378 failed 6326411
💔 Build #100187 failed 034d478

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Tyler Smalley <tyler.smalley@elastic.co>

jbudz · 2021-01-28T00:27:46Z

Spoke with tylersmalley about the most recent commit fix, had a slack approval.

7.x: 77a3d5c

…y-tests * 'master' of github.com:elastic/kibana: (31 commits) [Discover] Add grid flyout jest test (elastic#89088) [Search Sessions] Improve session restoration back button (elastic#87635) [TSVB] Remove vis_type_timeseries_enhanced plugin (elastic#89274) [Security Solution] Init Osquery plugin (elastic#87109) [Fleet] Do not defined aliases inside datastream template (elastic#89512) skip flaky suite (elastic#86950) chore(NA): bazel machinery installation on kbn bootstrap (elastic#89469) [build/docker] Add support for centos ARM builds (elastic#84831) Convert default_watch.json to a JS object in order to avoid TS complaints (elastic#89488) [CI] Decrease number of Jest workers (elastic#89504) [Maps] remove maps_oss TS project (elastic#89502) Adds migration settings to Docker (elastic#89501) [Lens] Fix crash in transition from unique count to last value (elastic#88916) [kbn-es] Always use bundled JDK when starting Elasticsearch (elastic#89437) unskip getting_started/shakespeare test elasticsearch 64016 (elastic#89346) [Maps] migrate maps, maps_file_upload, and maps_legacy_licensing to TS projects (elastic#89439) skip flaky suite (elastic#89478) skip flaky suite (elastic#89476) skip flaky suite (elastic#89477) skip flaky suite (elastic#89475) ... # Conflicts: # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/components/phases/cold_phase/cold_phase.tsx

…updates-and-timeline-cleanup * 'master' of github.com:elastic/kibana: (44 commits) [Discover] Add grid flyout jest test (elastic#89088) [Search Sessions] Improve session restoration back button (elastic#87635) [TSVB] Remove vis_type_timeseries_enhanced plugin (elastic#89274) [Security Solution] Init Osquery plugin (elastic#87109) [Fleet] Do not defined aliases inside datastream template (elastic#89512) skip flaky suite (elastic#86950) chore(NA): bazel machinery installation on kbn bootstrap (elastic#89469) [build/docker] Add support for centos ARM builds (elastic#84831) Convert default_watch.json to a JS object in order to avoid TS complaints (elastic#89488) [CI] Decrease number of Jest workers (elastic#89504) [Maps] remove maps_oss TS project (elastic#89502) Adds migration settings to Docker (elastic#89501) [Lens] Fix crash in transition from unique count to last value (elastic#88916) [kbn-es] Always use bundled JDK when starting Elasticsearch (elastic#89437) unskip getting_started/shakespeare test elasticsearch 64016 (elastic#89346) [Maps] migrate maps, maps_file_upload, and maps_legacy_licensing to TS projects (elastic#89439) skip flaky suite (elastic#89478) skip flaky suite (elastic#89476) skip flaky suite (elastic#89477) skip flaky suite (elastic#89475) ... # Conflicts: # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/components/phases/hot_phase/hot_phase.tsx # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/lib/absolute_timing_to_relative_timing.ts

odidev · 2021-02-19T12:20:31Z

It will be very helpful if an ARM64 image is released. May I know when you are planning to release a stable ARM64 image?

jbudz · 2021-02-24T19:39:48Z

We're targeting 7.13 for this. The necessary changes on the Kibana side are in 7.12 (here) with the release publishing in a separate repo. I'm swapping the the 7.12 label with 7.13 so this doesn't get picked up the release notes for 7.12.

jbudz added the Team:Operations Kibana-Operations Team label Dec 2, 2020

update snapshots

6f325d2

jbudz marked this pull request as ready for review December 2, 2020 21:51

jbudz requested a review from a team as a code owner December 2, 2020 21:51

jbudz added release_note:enhancement v7.11.0 v8.0.0 labels Dec 2, 2020

Merge branch 'master' of github.com:elastic/kibana into docker/archit…

0d2cce2

…ecture

jbudz added 4 commits December 4, 2020 11:53

Merge branch 'master' into docker/architecture

3b565b5

Merge branch 'master' into docker/architecture

8bc4add

Merge branch 'master' into docker/architecture

09163aa

always generate docker bundle, only build image for native host

c10c26d

update logs to specify package architecture

a2946b0

Merge branch 'master' into docker/architecture

353a305

tylersmalley reviewed Dec 9, 2020

View reviewed changes

jbudz added 5 commits December 9, 2020 12:22

node architecture first

4cccd58

Merge branch 'master' into docker/architecture

a126007

fix types

791ad36

set image architecture name

0369613

fix context

9212860

Merge branch 'master' into docker/architecture

4ca411b

squash architecture specific contexts

034d478

kibanamachine and others added 2 commits January 21, 2021 11:30

Merge branch 'master' into docker/architecture

6326411

fix types

33eab33

kibanamachine and others added 2 commits January 25, 2021 10:45

Merge branch 'master' into docker/architecture

1a6f2cd

Merge remote-tracking branch 'upstream/master' into docker/architecture

6ab6178

tylersmalley approved these changes Jan 27, 2021

View reviewed changes

jbudz added 2 commits January 27, 2021 15:56

Merge branch 'master' into docker/architecture

88c1705

fix ubi context

288710e

jbudz merged commit 2572cd2 into elastic:master Jan 28, 2021

jbudz deleted the docker/architecture branch January 28, 2021 00:25

jbudz added a commit that referenced this pull request Jan 28, 2021

[build/docker] Add support for centos ARM builds (#84831)

77a3d5c

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Tyler Smalley <tyler.smalley@elastic.co>

jbudz added the backported label Jan 28, 2021

jbudz mentioned this pull request Jan 28, 2021

[build] Remove file architecture from docker tag #89605

Merged

antoineco mentioned this pull request Feb 19, 2021

Error when deploying in ARM64 architecture deviantony/docker-elk#576

Closed

jbudz added v7.13.0 and removed v7.12.0 labels Feb 24, 2021

vncloudsco mentioned this pull request Oct 8, 2021

wazuh docker install on arm wazuh/wazuh-docker#532

Closed

jbudz mentioned this pull request Aug 25, 2022

The kibana images for ARM64 Machine #139451

Closed

Conversation

jbudz commented Dec 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jbudz commented Dec 2, 2020

Uh oh!

elasticmachine commented Dec 2, 2020

Uh oh!

tylersmalley commented Dec 4, 2020

Uh oh!

jbudz commented Dec 7, 2020

Uh oh!

jbudz commented Dec 8, 2020

Uh oh!

jbudz commented Dec 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jbudz commented Dec 8, 2020

Uh oh!

tylersmalley Dec 9, 2020

Choose a reason for hiding this comment

Uh oh!

jbudz Dec 9, 2020

Choose a reason for hiding this comment

Uh oh!

jbudz commented Dec 14, 2020

Uh oh!

jmlrt commented Jan 21, 2021

Uh oh!

jbudz commented Jan 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jbudz commented Jan 21, 2021

Uh oh!

jbudz commented Jan 25, 2021

Uh oh!

tylersmalley commented Jan 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tylersmalley commented Jan 27, 2021

Uh oh!

tylersmalley left a comment

Choose a reason for hiding this comment

Uh oh!

kibanamachine commented Jan 27, 2021

💚 Build Succeeded

Metrics [docs]

History

Uh oh!

jbudz commented Jan 28, 2021

Uh oh!

odidev commented Feb 19, 2021

Uh oh!

jbudz commented Feb 24, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

jbudz commented Dec 2, 2020 •

edited

Loading

jbudz commented Dec 8, 2020 •

edited

Loading

jbudz commented Jan 21, 2021 •

edited

Loading

tylersmalley commented Jan 27, 2021 •

edited

Loading