Skip to content

.circleci: Add simple backup and restore solution for RCs#38690

Closed
seemethere wants to merge 2 commits intopytorch:masterfrom
seemethere:add_binary_backup
Closed

.circleci: Add simple backup and restore solution for RCs#38690
seemethere wants to merge 2 commits intopytorch:masterfrom
seemethere:add_binary_backup

Conversation

@seemethere
Copy link
Copy Markdown
Member

@seemethere seemethere commented May 19, 2020

  • Does a basic upload of release candidates to an extra folder within our
    S3 bucket.
  • Refactors AWS promotion to allow for easier development of restoration
    of backups

Backup restoration usage:

RESTORE_FROM=v1.6.0-rc3 restore-backup.sh

Requires:

  • AWS credentials to upload / download stuff
  • Anaconda credentials to upload

@dr-ci
Copy link
Copy Markdown

dr-ci Bot commented May 19, 2020

💊 CI failures summary and remediations

As of commit 3a5fc34 (more details on the Dr. CI page):


  • 6/6 failures introduced in this PR

🕵️ 5 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

See CircleCI build pytorch_linux_xenial_py3_6_gcc5_4_build (1/5)

Step: "(Optional) Merge target branch" (full log | diagnosis details | 🔁 rerun)

Automatic merge failed; fix conflicts and then commit the result.
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/simple/android_gradle.py 
Auto-merging .circleci/cimodel/data/simple/android_gradle.py 
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/pytorch_build_definitions.py 
Auto-merging .circleci/cimodel/data/pytorch_build_definitions.py 
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/pytorch_build_data.py 
Auto-merging .circleci/cimodel/data/pytorch_build_data.py 
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/caffe2_build_definitions.py 
Auto-merging .circleci/cimodel/data/caffe2_build_definitions.py 
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/binary_build_data.py 
Auto-merging .circleci/cimodel/data/binary_build_data.py 
Automatic merge failed; fix conflicts and then commit the result. 

See CircleCI build pytorch_xla_linux_bionic_py3_6_clang9_build (2/5)

Step: "(Optional) Merge target branch" (full log | diagnosis details | 🔁 rerun)

Automatic merge failed; fix conflicts and then commit the result.
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/simple/android_gradle.py 
Auto-merging .circleci/cimodel/data/simple/android_gradle.py 
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/pytorch_build_definitions.py 
Auto-merging .circleci/cimodel/data/pytorch_build_definitions.py 
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/pytorch_build_data.py 
Auto-merging .circleci/cimodel/data/pytorch_build_data.py 
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/caffe2_build_definitions.py 
Auto-merging .circleci/cimodel/data/caffe2_build_definitions.py 
CONFLICT (add/add): Merge conflict in .circleci/cimodel/data/binary_build_data.py 
Auto-merging .circleci/cimodel/data/binary_build_data.py 
Automatic merge failed; fix conflicts and then commit the result. 

See CircleCI build pytorch_windows_vs2019_py36_cpu_build (3/5)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

..\third_party\fbgemm\include\fbgemm/FbgemmFP16.h(100): error C3861: 'runtime_error': identifier not found

de -I..\cmake\..\third_party\googletest\googlemock\include -I..\cmake\..\third_party\googletest\googletest\include -I..\third_party\protobuf\src -Iwin_tmp\mkl\include /DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -openmp:experimental /wd4244 /wd4267 /wd4305 /wd4309 /MD /O2 /Ob2 /DNDEBUG /w /bigobj   -std:c++14 /showIncludes /Fothird_party\fbgemm\CMakeFiles\fbgemm_generic.dir\src\ExecuteKernel.cc.obj /Fdthird_party\fbgemm\CMakeFiles\fbgemm_generic.dir\ /FS -c ..\third_party\fbgemm\src\ExecuteKernel.cc 
Microsoft (R) C/C++ Optimizing Compiler Version 19.26.28805 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

\include -I..\cmake\..\third_party\googletest\googlemock\include -I..\cmake\..\third_party\googletest\googletest\include -I..\third_party\protobuf\src -Iwin_tmp\mkl\include /DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -openmp:experimental /wd4244 /wd4267 /wd4305 /wd4309 /MD /O2 /Ob2 /DNDEBUG /w /bigobj   -std:c++14 /showIncludes /Fothird_party\fbgemm\CMakeFiles\fbgemm_generic.dir\src\FbgemmFP16.cc.obj /Fdthird_party\fbgemm\CMakeFiles\fbgemm_generic.dir\ /FS -c ..\third_party\fbgemm\src\FbgemmFP16.cc 
FAILED: third_party/fbgemm/CMakeFiles/fbgemm_generic.dir/src/FbgemmFP16.cc.obj  
\include -I..\cmake\..\third_party\googletest\googlemock\include -I..\cmake\..\third_party\googletest\googletest\include -I..\third_party\protobuf\src -Iwin_tmp\mkl\include /DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -openmp:experimental /wd4244 /wd4267 /wd4305 /wd4309 /MD /O2 /Ob2 /DNDEBUG /w /bigobj   -std:c++14 /showIncludes /Fothird_party\fbgemm\CMakeFiles\fbgemm_generic.dir\src\FbgemmFP16.cc.obj /Fdthird_party\fbgemm\CMakeFiles\fbgemm_generic.dir\ /FS -c ..\third_party\fbgemm\src\FbgemmFP16.cc 
..\third_party\fbgemm\include\fbgemm/FbgemmFP16.h(100): error C2039: 'runtime_error': is not a member of 'std'
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\include\vector(24): note: see declaration of 'std'
..\third_party\fbgemm\include\fbgemm/FbgemmFP16.h(100): error C3861: 'runtime_error': identifier not found
Microsoft (R) C/C++ Optimizing Compiler Version 19.26.28805 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

ake\..\third_party\googletest\googlemock\include -I..\cmake\..\third_party\googletest\googletest\include -I..\third_party\protobuf\src -Iwin_tmp\mkl\include /DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -openmp:experimental /wd4244 /wd4267 /wd4305 /wd4309 /MD /O2 /Ob2 /DNDEBUG /w /bigobj   -std:c++14 /showIncludes /Fothird_party\fbgemm\CMakeFiles\fbgemm_generic.dir\src\EmbeddingSpMDMNBit.cc.obj /Fdthird_party\fbgemm\CMakeFiles\fbgemm_generic.dir\ /FS -c ..\third_party\fbgemm\src\EmbeddingSpMDMNBit.cc 
Microsoft (R) C/C++ Optimizing Compiler Version 19.26.28805 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

 -I..\cmake\..\third_party\googletest\googlemock\include -I..\cmake\..\third_party\googletest\googletest\include -I..\third_party\protobuf\src -Iwin_tmp\mkl\include /DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -openmp:experimental /wd4244 /wd4267 /wd4305 /wd4309 /MD /O2 /Ob2 /DNDEBUG /w /bigobj   -std:c++14 /showIncludes /Fothird_party\fbgemm\CMakeFiles\fbgemm_generic.dir\src\EmbeddingSpMDM.cc.obj /Fdthird_party\fbgemm\CMakeFiles\fbgemm_generic.dir\ /FS -c ..\third_party\fbgemm\src\EmbeddingSpMDM.cc 
Microsoft (R) C/C++ Optimizing Compiler Version 19.26.28805 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

See CircleCI build pytorch_windows_vs2019_py36_cuda10.1_build (4/5)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

..\third_party\fbgemm\include\fbgemm/FbgemmFP16.h(100): error C3861: 'runtime_error': identifier not found

ake\..\third_party\googletest\googlemock\include -I..\cmake\..\third_party\googletest\googletest\include -I..\third_party\protobuf\src -Iwin_tmp\mkl\include /DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -openmp:experimental /wd4244 /wd4267 /wd4305 /wd4309 /MD /O2 /Ob2 /DNDEBUG /w /bigobj   -std:c++14 /showIncludes /Fothird_party\fbgemm\CMakeFiles\fbgemm_generic.dir\src\PackAWithRowOffset.cc.obj /Fdthird_party\fbgemm\CMakeFiles\fbgemm_generic.dir\ /FS -c ..\third_party\fbgemm\src\PackAWithRowOffset.cc 
Microsoft (R) C/C++ Optimizing Compiler Version 19.26.28805 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

\include -I..\cmake\..\third_party\googletest\googlemock\include -I..\cmake\..\third_party\googletest\googletest\include -I..\third_party\protobuf\src -Iwin_tmp\mkl\include /DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -openmp:experimental /wd4244 /wd4267 /wd4305 /wd4309 /MD /O2 /Ob2 /DNDEBUG /w /bigobj   -std:c++14 /showIncludes /Fothird_party\fbgemm\CMakeFiles\fbgemm_generic.dir\src\FbgemmFP16.cc.obj /Fdthird_party\fbgemm\CMakeFiles\fbgemm_generic.dir\ /FS -c ..\third_party\fbgemm\src\FbgemmFP16.cc 
FAILED: third_party/fbgemm/CMakeFiles/fbgemm_generic.dir/src/FbgemmFP16.cc.obj  
\include -I..\cmake\..\third_party\googletest\googlemock\include -I..\cmake\..\third_party\googletest\googletest\include -I..\third_party\protobuf\src -Iwin_tmp\mkl\include /DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -openmp:experimental /wd4244 /wd4267 /wd4305 /wd4309 /MD /O2 /Ob2 /DNDEBUG /w /bigobj   -std:c++14 /showIncludes /Fothird_party\fbgemm\CMakeFiles\fbgemm_generic.dir\src\FbgemmFP16.cc.obj /Fdthird_party\fbgemm\CMakeFiles\fbgemm_generic.dir\ /FS -c ..\third_party\fbgemm\src\FbgemmFP16.cc 
..\third_party\fbgemm\include\fbgemm/FbgemmFP16.h(100): error C2039: 'runtime_error': is not a member of 'std'
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\include\vector(24): note: see declaration of 'std'
..\third_party\fbgemm\include\fbgemm/FbgemmFP16.h(100): error C3861: 'runtime_error': identifier not found
Microsoft (R) C/C++ Optimizing Compiler Version 19.26.28805 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

ake\..\third_party\googletest\googlemock\include -I..\cmake\..\third_party\googletest\googletest\include -I..\third_party\protobuf\src -Iwin_tmp\mkl\include /DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -openmp:experimental /wd4244 /wd4267 /wd4305 /wd4309 /MD /O2 /Ob2 /DNDEBUG /w /bigobj   -std:c++14 /showIncludes /Fothird_party\fbgemm\CMakeFiles\fbgemm_generic.dir\src\EmbeddingSpMDMNBit.cc.obj /Fdthird_party\fbgemm\CMakeFiles\fbgemm_generic.dir\ /FS -c ..\third_party\fbgemm\src\EmbeddingSpMDMNBit.cc 
Microsoft (R) C/C++ Optimizing Compiler Version 19.26.28805 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

nclude -I..\cmake\..\third_party\googletest\googlemock\include -I..\cmake\..\third_party\googletest\googletest\include -I..\third_party\protobuf\src -Iwin_tmp\mkl\include /DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -openmp:experimental /wd4244 /wd4267 /wd4305 /wd4309 /MD /O2 /Ob2 /DNDEBUG /w /bigobj   -std:c++14 /showIncludes /Fothird_party\fbgemm\CMakeFiles\fbgemm_generic.dir\src\PackBMatrix.cc.obj /Fdthird_party\fbgemm\CMakeFiles\fbgemm_generic.dir\ /FS -c ..\third_party\fbgemm\src\PackBMatrix.cc 
Microsoft (R) C/C++ Optimizing Compiler Version 19.26.28805 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

See CircleCI build pytorch_macos_10_13_py3_test (5/5)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

May 21 12:48:28 AssertionError: False is not true : Tensors failed to compare as equal! Found 1 different element(s) (out of 9), with the greatest difference of 32768 (-32768 vs. 0) occuring at index 0.
May 21 12:48:28   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/common_device_type.py", line 234, in instantiated_test 
May 21 12:48:28     result = test(self, device_arg, dtype) 
May 21 12:48:28   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/common_device_type.py", line 501, in only_fn 
May 21 12:48:28     return fn(self, device, *args, **kwargs) 
May 21 12:48:28   File "test_torch.py", line 16621, in test_float_to_int_conversion_finite 
May 21 12:48:28     self._float_to_int_conversion_helper(vals, device, dtype) 
May 21 12:48:28   File "test_torch.py", line 16597, in _float_to_int_conversion_helper 
May 21 12:48:28     self.assertEqual(torch.from_numpy(a), t.cpu()) 
May 21 12:48:28   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py", line 1083, in assertEqual 
May 21 12:48:28     self.assertTrue(result, msg=message) 
May 21 12:48:28 AssertionError: False is not true : Tensors failed to compare as equal! Found 1 different element(s) (out of 9), with the greatest difference of 32768 (-32768 vs. 0) occuring at index 0. 
May 21 12:48:28  
May 21 12:48:28 ---------------------------------------------------------------------- 
May 21 12:48:28 Ran 2305 tests in 113.609s 
May 21 12:48:28  
May 21 12:48:28 FAILED (failures=1, skipped=135) 
May 21 12:48:28  
May 21 12:48:28 Generating XML reports... 
May 21 12:48:28 Generated XML report: test-reports/dist-gloo/TEST-TestTensorDeviceOpsCPU-20200521124634.xml 
May 21 12:48:28 Generated XML report: test-reports/dist-gloo/TEST-TestTorch-20200521124634.xml 
May 21 12:48:29 Generated XML report: test-reports/dist-gloo/TEST-TestTorchDeviceTypeCPU-20200521124634.xml 

1 failure not recognized by patterns:

Job Step Action
CircleCI binary_windows_libtorch_3_7_cpu_release_build Build 🔁 rerun

This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 31 times.

Comment thread scripts/release/promote/common_utils.sh Outdated
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Simplified this to do a recursive cp and only include files we actually want to copy over.

Comment thread scripts/release/promote/s3_to_s3.sh Outdated
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is actually just a modified version of the old wheel_to_s3.sh script

* Does a basic upload of release candidates to an extra folder within our
S3 bucket.
* Refactors AWS promotion to allow for easier development of restoration
of backups

Backup restoration usage:

  RESTORE_FROM=v1.6.0-rc3 restore-backup.sh

Requires:
  * AWS credentials to upload / download stuff
  * Anaconda credentials to upload

Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere seemethere force-pushed the add_binary_backup branch from b9913ea to 6adb5b9 Compare May 20, 2020 23:45
@seemethere seemethere marked this pull request as ready for review May 20, 2020 23:46
@seemethere seemethere requested review from kostmo and malfet May 20, 2020 23:47
@seemethere seemethere changed the title .circleci: Add simple backup solution for RCs .circleci: Add simple backup and restore solution for RCs May 20, 2020
@seemethere seemethere added the oncall: releng In support of CI and Release Engineering label May 20, 2020
@seemethere seemethere requested a review from ezyang May 21, 2020 18:14
Comment thread .circleci/scripts/binary_linux_upload.sh Outdated
Co-authored-by: Nikita Shulga <nikita.shulga@gmail.com>
Signed-off-by: Eli Uriegas <eliuriegas@fb.com>
@seemethere seemethere force-pushed the add_binary_backup branch from 824aac1 to 3a5fc34 Compare May 21, 2020 18:39
Copy link
Copy Markdown
Contributor

@facebook-github-bot facebook-github-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@seemethere is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Copy Markdown
Contributor

@seemethere merged this pull request in 5dd65ba.

laurentdupin pushed a commit to laurentdupin/pytorch that referenced this pull request Apr 24, 2026
)

Summary:
* Does a basic upload of release candidates to an extra folder within our
S3 bucket.
* Refactors AWS promotion to allow for easier development of restoration
of backups

Backup restoration usage:
```
RESTORE_FROM=v1.6.0-rc3 restore-backup.sh
```
Requires:
  * AWS credentials to upload / download stuff
  * Anaconda credentials to upload
Pull Request resolved: pytorch#38690

Differential Revision: D21691033

Pulled By: seemethere

fbshipit-source-id: 31118814db1ca701c55a3cb0bc32caa1e77a833d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Merged oncall: releng In support of CI and Release Engineering

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants