Skip to content

Commit f456748

Browse files
author
Iurii Zdebskyi
committed
Update on "[wip] Replace optimizers in torch.optim with the ones from torch.optim._multi_tensor"
 Differential Revision: [D25406490](https://our.internmc.facebook.com/intern/diff/D25406490) ------ ### Benchmark results SGD (lr=1e-3, momentum=1, dampening=0, weight_decay=1, nesterov=True) Current: 201.63 ms Foreach: 56.99 ms Adam (weight_decay=1., amsgrad=True) Current: 233.27 ms Foreach: 46.89 ms AdamW (weight_decay=1., amsgrad=True) Current: 371.18 ms Foreach: 121.04 ms RMSprop (weight_decay=1, momentum=1, centered=True) Current: 364.88 ms Foreach: 47.52 ms Rprop (lr=1e-2, etas=(0.5, 1.2), step_sizes=(1e-6, 50)) Current: 1.43 s Foreach: 1.26 s ASGD (weight_decay=1) Current: 165.39 ms Foreach: 40.61 ms Adamax (weight_decay=1) Current: 374.42 ms Foreach: 291.06 ms Adadelta (weight_decay=1) Current: 252.64 ms Foreach: 29.62 ms ### Benchmark script ``` import torch import torch.optim as optim import torch.nn as nn import torchvision import torch.utils.benchmark as benchmark_utils model = torchvision.models.resnet.resnet101(pretrained=True).to("cuda") targets = torch.randint(0, 1000, (100, 100), device="cuda") criterion = nn.CrossEntropyLoss() # optimizers params = dict(weight_decay=1) optimizer = optim.Adadelta(model.parameters(), **params) optimizer_mta = optim._multi_tensor.Adadelta(model.parameters(), **params) running_loss = 0.0 target = torch.empty(128, dtype=torch.long, device="cuda").random_(5) optimizer.zero_grad() inputs = torch.rand(128, 3, 100, 100, device="cuda" , requires_grad=True) outputs = model(inputs) loss = criterion(outputs, target) loss.backward() optimizer.step() running_loss += loss.item() def main(): timer = benchmark_utils.Timer( stmt="torch.cuda.synchronize(); optimizer.step()", globals=globals(), label="str(optimizer)", ) print(f"autorange:\n{timer.blocked_autorange()}\n\n") timer_mta = benchmark_utils.Timer( stmt="torch.cuda.synchronize(); optimizer_mta.step()", globals=globals(), label="str(optimizer_mta)", ) print(f"autorange:\n{timer_mta.blocked_autorange()}\n\n") if __name__ == "__main__": main() ``` [ghstack-poisoned]
2 parents b348206 + aa03124 commit f456748

954 files changed

Lines changed: 31435 additions & 12244 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.azure_pipelines/build-pipeline.yml‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# PyTorch CI Builds Pipeline on Azure DevOps
2+
#
3+
# This pipeline:
4+
# 1) builds PyTorch on select configurations
5+
# 2) runs only TestTorch unit tests.
6+
7+
stages:
8+
- stage: 'Build'
9+
displayName: 'Build PyTorch'
10+
jobs:
11+
- template: job_templates/build-verify-publish-template-unix.yml
12+
parameters:
13+
name: ubuntu_1804_CPU_docker
14+
pool: 'PyTorch-Linux-CPU'
15+
container_endpoint: pytorchms.azurecr.io
16+
build_stage: True
17+
is_ci_build: True
18+
os: ubuntu
19+
cuda: cpu
20+
customMatrixes:
21+
Py_38:
22+
configuration: ubuntu_1804_py_38_cpu
23+
container_image: pytorchms.azurecr.io/ubuntu_1804_py_38_cpu_dev
24+
25+
- template: job_templates/build-verify-publish-template-unix.yml
26+
parameters:
27+
name: ubuntu_1804_GPU_docker
28+
pool: 'PyTorch-Linux-GPU'
29+
container_endpoint: pytorchms.azurecr.io
30+
build_stage: True
31+
is_ci_build: True
32+
os: ubuntu
33+
cuda: gpu
34+
customMatrixes:
35+
Py_39_CUDA_112_cuDNN_810:
36+
configuration: ubuntu_1804_py_39_cuda_112_cudnn_810
37+
container_image: pytorchms.azurecr.io/ubuntu_1804_py_39_cuda_112_cudnn_8_dev
38+
CUDA_VERSION: 112
39+
40+
- template: job_templates/build-verify-publish-template-win.yml
41+
parameters:
42+
name: windows_2019_CPU
43+
pool: 'PyTorch-Win-CPU'
44+
build_stage: True
45+
is_ci_build: True
46+
os: windows
47+
cuda: cpu
48+
customMatrixes:
49+
Py_37:
50+
configuration: windows_2019_py_37_cpu
51+
52+
- template: job_templates/build-verify-publish-template-win.yml
53+
parameters:
54+
name: windows_2019_GPU
55+
pool: 'PyTorch-Win-GPU'
56+
build_stage: True
57+
is_ci_build: True
58+
os: windows
59+
cuda: gpu
60+
customMatrixes:
61+
Py_38_CUDA_102_cuDNN_765:
62+
configuration: windows_2019_py_38_cuda_102_cudnn_765
63+
CUDA_VERSION: 102

‎.azure_pipelines/daily-pipeline.yml‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# PyTorch Daily Builds Pipeline on Azure DevOps
2+
#
3+
# This pipeline:
4+
# 1) builds PyTorch on all available configurations
5+
# 2) runs all PyTorch unit tests
6+
7+
stages:
8+
- stage: 'BuildTest'
9+
displayName: 'Build and Test PyTorch'
10+
jobs:
11+
- template: job_templates/build-verify-publish-template-unix.yml
12+
parameters:
13+
name: ubuntu_1804_CPU_docker
14+
pool: 'PyTorch-Linux-CPU'
15+
container_endpoint: pytorchms.azurecr.io
16+
build_stage: True
17+
is_daily_build: True
18+
os: ubuntu
19+
cuda: cpu
20+
customMatrixes:
21+
Py_38:
22+
configuration: ubuntu_1804_py_38_cpu
23+
container_image: pytorchms.azurecr.io/ubuntu_1804_py_38_cpu_dev
24+
Py_37:
25+
configuration: ubuntu_1804_py_37_cpu
26+
container_image: pytorchms.azurecr.io/ubuntu_1804_py_37_cpu_dev
27+
28+
- template: job_templates/build-verify-publish-template-unix.yml
29+
parameters:
30+
name: ubuntu_1804_GPU_docker
31+
pool: 'PyTorch-Linux-GPU'
32+
container_endpoint: pytorchms.azurecr.io
33+
build_stage: True
34+
is_daily_build: True
35+
os: ubuntu
36+
cuda: gpu
37+
customMatrixes:
38+
Py_39_CUDA_112_cuDNN_810:
39+
configuration: ubuntu_1804_py_39_cuda_112_cudnn_810
40+
container_image: pytorchms.azurecr.io/ubuntu_1804_py_39_cuda_112_cudnn_8_dev
41+
CUDA_VERSION: 112
42+
Py_38_CUDA_102_cuDNN_810:
43+
configuration: ubuntu_1804_py_38_cuda_102_cudnn_810
44+
container_image: pytorchms.azurecr.io/ubuntu_1804_py_38_cuda_102_cudnn_8_dev
45+
CUDA_VERSION: 102
46+
Py_37_CUDA_101_cuDNN_765:
47+
configuration: ubuntu_1804_py_37_cuda_101_cudnn_765
48+
container_image: pytorchms.azurecr.io/ubuntu_1804_py_37_cuda_101_cudnn_7_dev
49+
CUDA_VERSION: 101
50+
51+
- template: job_templates/build-verify-publish-template-win.yml
52+
parameters:
53+
name: windows_2019_CPU
54+
pool: 'PyTorch-Win-CPU'
55+
build_stage: True
56+
is_daily_build: True
57+
os: windows
58+
cuda: cpu
59+
customMatrixes:
60+
Py_38:
61+
configuration: windows_2019_py_38_cpu
62+
Py_37:
63+
configuration: windows_2019_py_37_cpu
64+
65+
- template: job_templates/build-verify-publish-template-win.yml
66+
parameters:
67+
name: windows_2019_GPU
68+
pool: 'PyTorch-Win-GPU'
69+
build_stage: True
70+
is_daily_build: True
71+
os: windows
72+
cuda: gpu
73+
customMatrixes:
74+
Py_39_CUDA_112_cuDNN_810:
75+
configuration: windows_2019_py_39_cuda_112_cudnn_810
76+
CUDA_VERSION: 112
77+
Py_38_CUDA_102_cuDNN_765:
78+
configuration: windows_2019_py_38_cuda_102_cudnn_765
79+
CUDA_VERSION: 102
80+
Py_37_CUDA_101_cuDNN_764:
81+
configuration: windows_2019_py_37_cuda_101_cudnn_764
82+
CUDA_VERSION: 101

‎.azure_pipelines/job_templates/build-verify-publish-template-unix.yml‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# PyTorch build steps template with Unix images Azure DevOps Instances
2+
#
3+
# This build depends on 3 parameters set as environment variables in the pipeline:
4+
# - AZURE_DEVOPS_CLI_PAT: Secret var for authenticating to Azure DevOps
5+
# - AZURE_DEVOPS_ARTIFACTS_ORGANIZATION: Azure Artifacts Organization name to publish artifacts
6+
# - AZURE_DEVOPS_ARTIFACTS_PROJECT: Azure Artifacts Project name to publish artifacts
7+
8+
parameters:
9+
name: ''
10+
pool: ''
11+
container_endpoint: ''
12+
os: ''
13+
cuda: ''
14+
is_ci_build: False
15+
is_official_build: False
16+
is_daily_build: False
17+
build_stage: False
18+
verify_stage: False
19+
publish_stage: False
20+
customMatrixes: ''
21+
22+
jobs:
23+
- job: ${{parameters.name}}
24+
timeoutInMinutes: 300
25+
strategy:
26+
matrix:
27+
${{ insert }}: ${{parameters.customMatrixes}}
28+
pool:
29+
name: ${{ parameters.pool}}
30+
variables:
31+
DECODE_PERCENTS: false
32+
container:
33+
image: $[variables['container_image']]
34+
endpoint: ${{parameters.container_endpoint}}
35+
36+
steps:
37+
# Build stage
38+
- ${{ if eq(parameters.build_stage, 'True') }}:
39+
# Set up environment variables for specific pipeline build
40+
- template: set-environment-variables.yml
41+
parameters:
42+
os: ${{ parameters.os}}
43+
cuda: ${{ parameters.cuda}}
44+
is_official_build: ${{ parameters.is_official_build}}
45+
46+
# Sync and update PyTorch submodules
47+
- bash: git submodule update --init --recursive
48+
displayName: Update PyTorch submodules
49+
50+
# Build PyTorch and run unit tests - no packaging
51+
- ${{ if or(eq(parameters.is_ci_build, 'True'), eq(parameters.is_daily_build, 'True')) }}:
52+
# Build PyTorch from source in develop mode
53+
- bash: python setup.py develop
54+
displayName: Build PyTorch from source
55+
56+
- ${{ if eq(parameters.is_ci_build, 'True') }}:
57+
# Run TestTorch unit tests to demonstrate successful PyTorch build
58+
- bash: python test/test_torch.py TestTorch
59+
displayName: Run TestTorch unit tests
60+
61+
- ${{ if eq(parameters.is_daily_build, 'True') }}:
62+
# Run all unit tests to demonstrate successful PyTorch build
63+
- bash: python test/run_test.py --continue-through-error --exclude-jit-executor --verbose
64+
displayName: Run all unit tests
65+
66+
# Run ComponentGovernance
67+
- task: ComponentGovernanceComponentDetection@0
68+
inputs:
69+
scanType: 'Register'
70+
verbosity: 'Verbose'
71+
alertWarningLevel: 'High'
72+
73+
# Build PyTorch and produce artifacts for verification stage
74+
- ${{ if eq(parameters.is_official_build, 'True') }}:
75+
# Build PyTorch from source in install mode and exclude test binaries
76+
- bash: python setup.py install
77+
displayName: Build PyTorch from source without test binaries
78+
79+
# Package PyTorch Wheel
80+
- bash: python setup.py bdist_wheel
81+
displayName: Package PyTorch Wheel
82+
83+
# Publish PyTorch Wheel
84+
- task: PublishPipelineArtifact@1
85+
inputs:
86+
targetPath: $(Build.SourcesDirectory)/dist/
87+
artifactName: Build_$(Build.BuildNumber)_$(configuration)
88+
displayName: Publish PyTorch Wheel to Pipeline Artifacts
89+
90+
# Verification stage
91+
- ${{ if eq(parameters.verify_stage, 'True') }}:
92+
# Download PyTorch Wheel
93+
- task: DownloadPipelineArtifact@2
94+
inputs:
95+
artifact: Build_$(Build.BuildNumber)_$(configuration)
96+
path: $(Build.SourcesDirectory)/verify
97+
displayName: Download PyTorch Wheel
98+
99+
# Install PyTorch Wheel on Windows
100+
- bash: python -m pip install $(Build.SourcesDirectory)/verify/torch*linux*.whl
101+
displayName: Install PyTorch Wheel
102+
103+
# Ensure PyTorch installed correctly from produced wheel
104+
- bash: |
105+
cd $(Build.SourcesDirectory)/verify
106+
python -c "import torch; print('Installed Torch version: ' + torch.__version__)"
107+
displayName: Check PyTorch correctly installed from wheel
108+
109+
# Publishing stage
110+
- ${{ if eq(parameters.publish_stage, 'True') }}:
111+
# Download PyTorch Wheel
112+
- task: DownloadPipelineArtifact@2
113+
inputs:
114+
artifact: Build_$(Build.BuildNumber)_$(configuration)
115+
path: $(Build.SourcesDirectory)/publish
116+
displayName: Download PyTorch Wheel
117+
118+
# Publish wheel to Azure Artifacts
119+
# The flag continueOnError=true is needed as the artifact to be published
120+
# may already exist, because the artifact is differentiated based on the
121+
# last commit date.
122+
- bash: |
123+
export TORCH_VERSION=$(head -c 5 ./version.txt)
124+
export LAST_COMMIT=$(git rev-parse --short HEAD)
125+
export LAST_COMMIT_DATE=$(git log -1 --pretty=%ad --date=format:%Y%m%d)
126+
cd $(Build.SourcesDirectory)/publish
127+
export TORCH_WHEEL=$(echo torch*linux*whl)
128+
az extension add -n azure-devops
129+
echo $ADOTOKEN | az devops login
130+
az artifacts universal publish --organization $AZURE_DEVOPS_ARTIFACTS_ORGANIZATION --project $AZURE_DEVOPS_ARTIFACTS_PROJECT --scope project --feed "PyTorch" --name $TORCH_WHEEL --description "PyTorch Official Build Artifact" --version $TORCH_VERSION-$LAST_COMMIT_DATE-$LAST_COMMIT --path .
131+
env:
132+
ADOTOKEN: $(AZURE_DEVOPS_CLI_PAT)
133+
continueOnError: true
134+
displayName: Upload PyTorch Official Build package to Azure Artifacts

0 commit comments

Comments
 (0)