Skip to content

GH-45307: [CI] Use GitHub hosted arm runner #45308

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
raulcd merged 4 commits into from
Jan 21, 2025
Merged

GH-45307: [CI] Use GitHub hosted arm runner #45308

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
173d510
GH-45307: [CI] Use GitHub hosted arm runner
kou Jan 20, 2025
127dabd
Remove unused job
kou Jan 20, 2025
cbdb3e0
Remove garbage
kou Jan 20, 2025
450a344
Remove jobs for self-hosted
kou Jan 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 23 additions & 51 deletions .github/workflows/cpp.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,65 +69,37 @@ env:
DOCKER_VOLUME_PREFIX: ".docker/"

jobs:
docker-targets:
name: Docker targets
runs-on: ubuntu-latest
outputs:
targets: ${{ steps.detect-targets.outputs.targets }}
steps:
- name: Detect targets
id: detect-targets
run: |
echo "targets<<JSON" >> "$GITHUB_OUTPUT"
echo "[" >> "$GITHUB_OUTPUT"
cat <<JSON >> "$GITHUB_OUTPUT"
{
"arch": "amd64",
"clang-tools": "14",
"image": "conda-cpp",
"llvm": "14",
"runs-on": "ubuntu-latest",
"simd-level": "AVX2",
"title": "AMD64 Conda C++ AVX2",
"ubuntu": "22.04"
},
{
"arch": "amd64",
"clang-tools": "14",
"image": "ubuntu-cpp-sanitizer",
"llvm": "14",
"runs-on": "ubuntu-latest",
"title": "AMD64 Ubuntu 22.04 C++ ASAN UBSAN",
"ubuntu": "22.04"
}
JSON
if [ "$GITHUB_REPOSITORY_OWNER" = "apache" ]; then
echo "," >> "$GITHUB_OUTPUT"
cat <<JSON >> "$GITHUB_OUTPUT"
{
"arch": "arm64v8",
"clang-tools": "10",
"image": "ubuntu-cpp",
"llvm": "10",
"runs-on": ["self-hosted", "arm", "linux"],
"title": "ARM64 Ubuntu 20.04 C++",
"ubuntu": "20.04"
}
JSON
fi
echo "]" >> "$GITHUB_OUTPUT"
echo "JSON" >> "$GITHUB_OUTPUT"

docker:
name: ${{ matrix.title }}
needs: docker-targets
runs-on: ${{ matrix.runs-on }}
if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
timeout-minutes: 75
strategy:
fail-fast: false
matrix:
include: ${{ fromJson(needs.docker-targets.outputs.targets) }}
include:
- arch: amd64
clang-tools: 14
image: conda-cpp
llvm: 14
runs-on: ubuntu-latest
simd-level: AVX2
title: AMD64 Conda C++ AVX2
ubuntu: 22.04
- arch: amd64
clang-tools: 14
image: ubuntu-cpp-sanitizer
llvm: 14
runs-on: ubuntu-latest
title: AMD64 Ubuntu 22.04 C++ ASAN UBSAN
ubuntu: 22.04
- arch: arm64v8
clang-tools: 10
image: ubuntu-cpp
llvm: 10
runs-on: ubuntu-24.04-arm
title: ARM64 Ubuntu 20.04 C++
ubuntu: 20.04
env:
ARCH: ${{ matrix.arch }}
ARROW_SIMD_LEVEL: ${{ matrix.simd-level }}
Expand Down
4 changes: 2 additions & 2 deletions dev/tasks/linux-packages/github.linux.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,9 @@ jobs:
package:
name: Package
{% if architecture == "amd64" %}
runs-on: ubuntu-22.04
runs-on: ubuntu-latest
{% else %}
runs-on: ["self-hosted", "Linux", "arm64"]
runs-on: ubuntu-24.04-arm
{% endif %}
env:
ARCHITECTURE: {{ architecture }}
Expand Down
38 changes: 3 additions & 35 deletions dev/tasks/macros.jinja
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,20 +57,11 @@ env:
{% endmacro %}

{%- macro github_install_archery() -%}
- name: Set up Python by actions/setup-python
if: |
!(runner.os == 'Linux' && runner.arch != 'X64')
- name: Set up Python
uses: actions/setup-python@v4
with:
cache: 'pip'
python-version: 3.12
- name: Set up Python by apt
if: runner.os == 'Linux' && runner.arch != 'X64'
run: |
sudo apt update
sudo apt-get install -y python3-pip
pip install -U pip
echo "$HOME/.local/bin" >>"$GITHUB_PATH"
- name: Install Archery
shell: bash
run: pip install -e arrow/dev/archery[all]
Expand All @@ -85,21 +76,10 @@ env:
{% endmacro %}

{%- macro github_upload_releases(pattern) -%}
- name: Set up Python by actions/setup-python
if: |
!(runner.os == 'Linux' && runner.arch != 'X64')
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: 3.12
- name: Set up Python by apt
if: runner.os == 'Linux' && runner.arch != 'X64'
run: |
sudo apt update
sudo apt install -y \
libgit2-dev \
libpython3-dev \
python3-pip
sudo python3 -m pip install --upgrade pip
- name: Checkout Crossbow
uses: actions/checkout@v4
with:
Expand Down Expand Up @@ -145,23 +125,11 @@ env:

{%- macro github_upload_gemfury(pattern) -%}
{%- if arrow.is_default_branch() -%}
- name: Set up Ruby by apt
if: runner.os == 'Linux' && runner.arch != 'X64'
run: |
sudo apt update
sudo apt install -y ruby-full
- name: Set up Ruby by GitHub Actions
if: runner.arch == 'X64' && runner.os != 'macOS'
- name: Set up Ruby
uses: ruby/setup-ruby@v1
with:
ruby-version: "ruby"
- name: Install gemfury client on ARM self-hosted
if: runner.arch != 'X64'
run: |
gem install --user-install gemfury
ruby -r rubygems -e 'puts("#{Gem.user_dir}/bin")' >> $GITHUB_PATH
- name: Install gemfury client
if: runner.arch == 'X64'
run: |
gem install gemfury
- name: Upload package to Gemfury
Expand Down
2 changes: 1 addition & 1 deletion dev/tasks/python-wheels/github.linux.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ jobs:
{% if arch == "amd64" %}
runs-on: ubuntu-latest
{% else %}
runs-on: ["self-hosted", "Linux", "arm64"]
runs-on: ubuntu-24.04-arm
{% endif %}
env:
# archery uses these environment variables
Expand Down