Enabled pytorch-linux-backward-compatibility-check-test in GHA by MaxMotovilov · Pull Request #61973 · pytorch/pytorch

MaxMotovilov · 2021-07-21T17:48:04Z

Summary:

Test Plan: Execute GHA, verify results

Reviewers: Zhuojie Zhou

Subscribers:

Tasks: T96112383

Tags:

facebook-github-bot · 2021-07-21T17:48:10Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/61973
↩️ [fb-only] Re-run with SSH instructions
🔧 Opt-in to CIFlow to control what jobs run on your PRs

💊 CI failures summary and remediations

As of commit 21013ce (more details on the Dr. CI page):

✅ None of the CI failures appear to be your fault 💚

1/1 tentatively recognized as flaky ❄️
- Click here to rerun these jobs

❄️ 1 failure tentatively classified as flaky

but reruns have not yet been triggered to confirm:

pytorch_linux_xenial_cuda11_1_cudnn8_py3_gcc7_test2 (1/1)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun) ❄️

Aug 05 23:00:15 unknown file: Failure

Aug 05 23:00:15 frame #9: build/bin/test_api() [0xbdb39a]
Aug 05 23:00:15 frame #10: build/bin/test_api() [0xbdbbed]
Aug 05 23:00:15 frame #11: testing::internal::UnitTestImpl::RunAllTests() + 0xe27 (0xbe47b7 in build/bin/test_api)
Aug 05 23:00:15 frame #12: testing::UnitTest::Run() + 0x8f (0xbe4b6f in build/bin/test_api)
Aug 05 23:00:15 frame #13: main + 0xc8 (0x586cb8 in build/bin/test_api)
Aug 05 23:00:15 frame #14: __libc_start_main + 0xf0 (0x7f22493d0840 in /lib/x86_64-linux-gnu/libc.so.6)
Aug 05 23:00:15 frame #15: _start + 0x29 (0x5c7b29 in build/bin/test_api)
Aug 05 23:00:15 " thrown in the test body.
Aug 05 23:00:15 [  FAILED  ] IMethodTest.CallMethod (3 ms)
Aug 05 23:00:15 [ RUN      ] IMethodTest.GetArgumentNames
Aug 05 23:00:15 unknown file: Failure
Aug 05 23:00:15 C++ exception with description "open file failed, file path: torch/csrc/deploy/example/generated/simple_jit
Aug 05 23:00:15 Exception raised from RAIIFile at /var/lib/jenkins/workspace/caffe2/serialize/file_adapter.cc:13 (most recent call first):
Aug 05 23:00:15 frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f2249ae64ab in /var/lib/jenkins/workspace/build/lib/libc10.so)
Aug 05 23:00:15 frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xce (0x7f2249ae1c6e in /var/lib/jenkins/workspace/build/lib/libc10.so)
Aug 05 23:00:15 frame #2: caffe2::serialize::FileAdapter::RAIIFile::RAIIFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xdc (0x7f2265cea12c in /var/lib/jenkins/workspace/build/lib/libtorch_cpu.so)
Aug 05 23:00:15 frame #3: caffe2::serialize::FileAdapter::FileAdapter(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x41 (0x7f2265cea7b1 in /var/lib/jenkins/workspace/build/lib/libtorch_cpu.so)
Aug 05 23:00:15 frame #4: torch::jit::load(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, c10::optional<c10::Device>, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&) + 0x40 (0x7f2267345460 in /var/lib/jenkins/workspace/build/lib/libtorch_cpu.so)
Aug 05 23:00:15 frame #5: torch::jit::load(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, c10::optional<c10::Device>) + 0x6f (0x7f22673455ef in /var/lib/jenkins/workspace/build/lib/libtorch_cpu.so)
Aug 05 23:00:15 frame #6: IMethodTest_GetArgumentNames_Test::TestBody() + 0x7e (0xb9f5be in build/bin/test_api)
Aug 05 23:00:15 frame #7: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) + 0x4a (0xbebafa in build/bin/test_api)

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

facebook-github-bot · 2021-07-22T16:33:15Z

@MaxMotovilov has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

zhouzhuojie

it still triggers a full build, is it possible that the backward tests can be included in the existing linux CI workflow? For example, just like the build_docs job, it's only enabled for pytorch-linux-xenial-py3.6-gcc5.4, and reuse the build artifacts

zhouzhuojie · 2021-07-22T19:38:18Z

+        docker_image_base=f"{DOCKER_REGISTRY}/pytorch/pytorch-linux-xenial-py3.6-gcc5.4",
+        test_runner_type=LINUX_CPU_TEST_RUNNER,
+        on_pull_request=True,
+        num_test_shards=2,


I don't think test_shards is handled for backward tests, can you help check the logs from both shards - I guess they are testing same things, if yes might want to set it to 1 or remove this field

I don't think test_shards is handled for backward tests, can you help check the logs from both shards - I guess they are testing same things, if yes might want to set it to 1 or remove this field

Pretty sure they are... I was wondering where the /0 and /1 are coming from.

Shards does not really work for backward compat tests, so let's keep them at one

Is an entirely new workflow really needed for this? Seems like this is just an extension of pytorch-linux-xenial-py3.6-gcc5.4 in general

@samestep did a lot of work to enable separate configurations within that workflow

As far as I could understand, the script for the bw compat test installs a nightly build of PyTorch anyway -

pytorch/.jenkins/pytorch/test.sh

Line 369 in d6e776d

pip_install --pre torch -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html

. Wonder if it is possible to run it against some manifestly pre-existing Docker image rather than require a fresh build that will get overwritten anyway?

the main goal of bc check is to build with current commit and compare against latest nightly API FunctionSchemas.

so it is not "overwritten" --> once the FunctionSchema from nightly wheel got extracted it is actually not useful anymore.
That being said, it would be nice to just have some way to host the API schema instead of having to download the entire nightly build.

MaxMotovilov · 2021-07-22T19:47:43Z

it still triggers a full build, is it possible that the backward tests can be included in the existing linux CI workflow? For example, just like the build_docs job, it's only enabled for pytorch-linux-xenial-py3.6-gcc5.4, and reuse the build artifacts

Given that the first thing it does for the actual test step is installs a nightly, perhaps it doesn't even need the build dependency? Strange that CircleCI version seems to have it...

zhouzhuojie · 2021-07-22T21:47:30Z

it still triggers a full build, is it possible that the backward tests can be included in the existing linux CI workflow? For example, just like the build_docs job, it's only enabled for pytorch-linux-xenial-py3.6-gcc5.4, and reuse the build artifacts

Given that the first thing it does for the actual test step is installs a nightly, perhaps it doesn't even need the build dependency? Strange that CircleCI version seems to have it...

oh wait, so basically it just test the backward compatibility for nightly builds? if so, then yes, probably can skip the build step and just go for the test

MaxMotovilov · 2021-07-22T21:55:09Z

oh wait, so basically it just test the backward compatibility for nightly builds? if so, then yes, probably can skip the build step and just go for the test

But what docker image should I use? Need something that has [a fresh enough] .jenkins/ and probably some dependencies. What it doesn't need is a PyTorch build.

zhouzhuojie · 2021-07-22T21:59:27Z

oh wait, so basically it just test the backward compatibility for nightly builds? if so, then yes, probably can skip the build step and just go for the test

But what docker image should I use? Need something that has [a fresh enough] .jenkins/ and probably some dependencies. What it doesn't need is a PyTorch build.

That's a base docker image we rebuilt not frequently, e.g. some low-level dependencies will be put there to prepare the pytorch build. The image can be found from the calculate-docker-image step

...in hopes that resolves the submodule issues during CI

Merging upstream to resolve CI issue w/submodules

walterddr

in order to achieve what @zhouzhuojie mentioned I think you also need to change generate_pytorch_test_matrix.py to generate additional test step for backward_compatibility.

walterddr · 2021-08-04T17:39:41Z

it still triggers a full build, is it possible that the backward tests can be included in the existing linux CI workflow? For example, just like the build_docs job, it's only enabled for pytorch-linux-xenial-py3.6-gcc5.4, and reuse the build artifacts

Given that the first thing it does for the actual test step is installs a nightly, perhaps it doesn't even need the build dependency? Strange that CircleCI version seems to have it...

oh wait, so basically it just test the backward compatibility for nightly builds? if so, then yes, probably can skip the build step and just go for the test

this statement is not true, it is comparing against nightly as baseline (it is test for the current commit)

MaxMotovilov · 2021-08-04T17:46:18Z

in order to achieve what @zhouzhuojie mentioned I think you also need to change generate_pytorch_test_matrix.py to generate additional test step for backward_compatibility.

Wait - why? Current code in test.sh just runs the bwc check after the regular test step in pytorch-linux-xenial-py3.6-gcc5.4 because the BUILD_ENVIRONMENT passed into the test step (and the test step only!) is set to pytorch-linux-xenial-py3.6-gcc5.4-backward in the template. Or is this too much of a hack?

walterddr · 2021-08-04T19:11:13Z

Yes I am just reflecting on the original comment from test.sh

# Do NOT run this test before any other tests, like test_python_shard1, etc.
# Because this function uninstalls the torch built from branch, and install
# nightly version.

and I am sure there's a reason why it needs to be on its own job instead of being ran as part of other tests. also this will run bc check on both test1 and test2 right?

MaxMotovilov · 2021-08-04T20:09:55Z

and I am sure there's a reason why it needs to be on its own job instead of being ran as part of other tests

Says "before" - which makes sense given that it overwrites current build with a nightly. Shouldn't have a problem with "after" though?

also this will run bc check on both test1 and test2 right?

Ah! Yes, I meant to fix that and forgot. Will do - just need to pull this all back on a devserver because the one I was using has already lapsed after I finished bootcamp.

github-actions · 2022-04-13T11:34:53Z

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

MaxMotovilov requested review from driazati, janeyx99, seemethere and zhouzhuojie as code owners July 21, 2021 17:48

facebook-github-bot added the cla signed label Jul 21, 2021

zhouzhuojie reviewed Jul 22, 2021

View reviewed changes

Alternate approach - run bwc test after regular tests on a select build

b7ec4be

MaxMotovilov force-pushed the enable-ci-action branch from e17c636 to b7ec4be Compare July 23, 2021 17:15

MaxMotovilov added 4 commits July 23, 2021 10:20

flake8 fix

49cd051

Modified BUILD_ENVIRONMENT goes only into docker run

91bd3fc

Merge remote-tracking branch 'pytorch/master' into enable-ci-action

4e0f255

...in hopes that resolves the submodule issues during CI

Merge remote-tracking branch 'pytorch/master' into enable-ci-action

c1ab3e0

Merging upstream to resolve CI issue w/submodules

MaxMotovilov requested a review from zhouzhuojie July 28, 2021 21:18

zhouzhuojie mentioned this pull request Aug 4, 2021

[BE] fix bc check #62687

Closed

walterddr reviewed Aug 4, 2021

View reviewed changes

MaxMotovilov added 3 commits August 5, 2021 12:58

Only run bwc check in shard 1 (if sharded)

34eb32a

Merge remote-tracking branch 'pytorch/master' into enable-ci-action

c776a9d

Updated the generated yaml

21013ce

driazati removed their request for review September 29, 2021 18:30

github-actions Bot added the Stale label Apr 13, 2022

github-actions Bot closed this May 13, 2022

Conversation

MaxMotovilov commented Jul 21, 2021

Uh oh!

facebook-github-bot commented Jul 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful links

💊 CI failures summary and remediations

❄️ 1 failure tentatively classified as flaky

pytorch_linux_xenial_cuda11_1_cudnn8_py3_gcc7_test2 (1/1)

Uh oh!

facebook-github-bot commented Jul 22, 2021

Uh oh!

zhouzhuojie left a comment

Choose a reason for hiding this comment

Uh oh!

zhouzhuojie Jul 22, 2021

Choose a reason for hiding this comment

Uh oh!

MaxMotovilov Jul 22, 2021

Choose a reason for hiding this comment

Uh oh!

malfet Jul 22, 2021

Choose a reason for hiding this comment

Uh oh!

seemethere Jul 22, 2021

Choose a reason for hiding this comment

Uh oh!

MaxMotovilov Jul 22, 2021

Choose a reason for hiding this comment

Uh oh!

walterddr Aug 4, 2021

Choose a reason for hiding this comment

Uh oh!

MaxMotovilov commented Jul 22, 2021

Uh oh!

zhouzhuojie commented Jul 22, 2021

Uh oh!

MaxMotovilov commented Jul 22, 2021

Uh oh!

zhouzhuojie commented Jul 22, 2021

Uh oh!

walterddr left a comment

Choose a reason for hiding this comment

Uh oh!

walterddr commented Aug 4, 2021

Uh oh!

MaxMotovilov commented Aug 4, 2021

Uh oh!

walterddr commented Aug 4, 2021

Uh oh!

MaxMotovilov commented Aug 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Apr 13, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

facebook-github-bot commented Jul 21, 2021 •

edited

Loading

MaxMotovilov commented Aug 4, 2021 •

edited

Loading