Skip to content

.github: Add reusable workflow for generic linux_job#783

Merged
seemethere merged 45 commits intomainfrom
seemethere/add_build_test_linux_workflow
Sep 29, 2022
Merged

.github: Add reusable workflow for generic linux_job#783
seemethere merged 45 commits intomainfrom
seemethere/add_build_test_linux_workflow

Conversation

@seemethere
Copy link
Copy Markdown
Member

@seemethere seemethere commented Sep 26, 2022 •
edited
Loading

Context

Adds a reusable workflow for running generic scripts on linux. Should cover cpu as well as all of the current CUDA versions we support in our binary images.

Basic idea behind this is that it gives users an easy interface to create a generic workflow with all of the niceties of our current setup like:

  • Access to ECR / S3 by default
  • NVIDIA Drivers by default (if a GPU is present)
  • with-ssh access by default
  • Proper cleanup / teardown

Users will give a simple script to the reusable workflow that can either be a multiline string defined in the YAML or can just reference a local bash script from the repository itself.

Usage:

name: Run simple workflow

on:
  pull_request:
  workflow_dispatch:

jobs:
  test-cpu:
    uses: pytorch/test-infra/.github/workflows/linux_job.yml@main
    with:
      runner: linux.2xlarge
      gpu-arch-type: cpu
      gpu-arch-version: ""
      script: |
        export PATH="/opt/python/cp38-cp38/bin:${PATH}"
        python3 -m pip install --extra-index-url https://download.pytorch.org/whl/nightly/cpu --pre torch
        # Can import pytorch
        python3 -c 'import torch'
  test-gpu:
    uses: pytorch/test-infra/.github/workflows/linux_job.yml@main
    with:
      runner: linux.4xlarge.nvidia.gpu
      gpu-arch-type: cuda
      gpu-arch-version: 11.6
      script: |
        export PATH="/opt/python/cp38-cp38/bin:${PATH}"
        python3 -m pip install --extra-index-url https://download.pytorch.org/whl/nightly/cpu --pre torch
        # Can import pytorch
        python3 -c 'import torch;assert(torch.cuda.is_available())'

depends on changes from #782

@vercel
Copy link
Copy Markdown

vercel bot commented Sep 26, 2022 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Updated
torchci ⬜️ Ignored (Inspect) Sep 28, 2022 at 11:22PM (UTC)

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 26, 2022
@seemethere seemethere changed the title seemethere/add build test linux workflow github: Add workflow for build/test on linux Sep 26, 2022
@seemethere seemethere force-pushed the seemethere/add_build_test_linux_workflow branch from c8b56d9 to fc98c0c Compare September 26, 2022 05:56
@seemethere
.github: Add workflow for build/test on linux
564f7ff 
Adds a workflow for build/test on linux, could potentially use some
improvements to support GPU but this should get _most_ users off the
ground quickly

Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
_ -> -
051e1de 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
bump container image
f7b182c 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
correct path for setup-miniconda
82e2d14 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
switch up the build / test scripts
23251da 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
fix artifact movement / upload
3487328 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
fix /dev/null
bb2e5ca 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
fix dist/
ba10fc4 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
use mv instead of cp
2b45b73 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
remove trailing slash
c28b581 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
mkdir now
e886999 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
fix RUNNER_ARTIFACT_DIR, remove container
3440d31 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
big refactor, focus more on linux_job
a788fb5 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere seemethere force-pushed the seemethere/add_build_test_linux_workflow branch from abd9bc9 to a788fb5 Compare September 27, 2022 23:53
@seemethere
fix download-artifact
d5e3d9c 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
further cleanups
b98ee2b 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
set correct default value for download-artifact
3ae4a9c 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
fix names, fix runner_temp for env
469770f 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
add chown
2170b07 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
remove ec2-user, add always to teardown
b64df8e 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
add pull action, chmod script, exec right
4c8f019 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
fix docker iamge
a836953 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
add environment variables for pip
d322f6e 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
move env up one
05fc5d2 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
remove env altogether
991abcc 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
add path
c970820 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
move env up one more time
4c9ff5d 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
add CU_VERSION for cpu
7dcb22d 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
set UPLOAD_CHANNEL
a5a0750 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
add bigger shm size
d4c9850 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
bump linux.4xlarge
aa3b5b3 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
pin ref for torchvision, increase timeout, bump runner
10b30ab 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
add chown, back to main
64be776 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
switch to not use torchvision directly
1e12408 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
nightly -> ""
dc920be 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
needs-nvidia-driver
852f3f6 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere
add shell
2d6b032 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere seemethere marked this pull request as ready for review September 28, 2022 23:09
@seemethere seemethere requested a review from a team September 28, 2022 23:12
@seemethere seemethere changed the title github: Add workflow for build/test on linux github: Add reusable workflow for generic linux_job Sep 28, 2022
@seemethere seemethere changed the title github: Add reusable workflow for generic linux_job .github: Add reusable workflow for generic linux_job Sep 28, 2022
@seemethere
appease the shellcheck gods
153b295 
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere seemethere added the Nova Tooling Tooling Related to Project Nova (CI Tooling) label Sep 28, 2022
malfet
malfet approved these changes Sep 29, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@malfet malfet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, can it be perhaps pair it with similar change to pytorch/pytorch that uses those actions rather than have 2 slightly different copies of those actions.

@seemethere seemethere merged commit 69e7c4f into main Sep 29, 2022
@seemethere seemethere deleted the seemethere/add_build_test_linux_workflow branch September 29, 2022 21:56
@seemethere seemethere mentioned this pull request Sep 30, 2022
Draft
@malfet malfet mentioned this pull request Sep 30, 2022
Closed
malfet added a commit to pytorch/pytorch that referenced this pull request Oct 1, 2022
@malfet
[BE] Use reusable workflows from test-infra
3755ccc 
Instead of local copies, use workflows checked into test-infra by pytorch/test-infra#783
malfet added a commit to pytorch/pytorch that referenced this pull request Oct 1, 2022
@malfet
[BE] Use reusable workflows from test-infra
5885e17 
Instead of local copies, use workflows checked into test-infra by pytorch/test-infra#783
pytorchmergebot pushed a commit to pytorch/pytorch that referenced this pull request Oct 1, 2022
@malfet @pytorchmergebot
[BE] Use reusable workflows from test-infra (#86035)
fe89cd6 
Instead of local copies, use workflows checked into test-infra by pytorch/test-infra#783

Thought about deleting the actions later, but if I understand how GHA merges work, older PRs merged onto this changes should not cause any problems as it will immediately reference actions from test-infra
Pull Request resolved: #86035
Approved by: https://github.com/kit1980
mehtanirav pushed a commit to pytorch/pytorch that referenced this pull request Oct 4, 2022
@malfet @mehtanirav
[BE] Use reusable workflows from test-infra (#86035)
04a4e18 
Instead of local copies, use workflows checked into test-infra by pytorch/test-infra#783

Thought about deleting the actions later, but if I understand how GHA merges work, older PRs merged onto this changes should not cause any problems as it will immediately reference actions from test-infra
Pull Request resolved: #86035
Approved by: https://github.com/kit1980
@seemethere seemethere mentioned this pull request Oct 17, 2022
Merged
@lequytra lequytra mentioned this pull request Oct 26, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@malfet malfet malfet approved these changes

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. Nova Tooling Tooling Related to Project Nova (CI Tooling)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@seemethere @malfet @facebook-github-bot