Skip to content

[test-infra] Switch RBE linux build to a new custom image rbe_ubuntu2004.#32748

Merged
veblush merged 7 commits intogrpc:masterfrom
jtattermusch:rbe2004
May 1, 2023
Merged

[test-infra] Switch RBE linux build to a new custom image rbe_ubuntu2004.#32748
veblush merged 7 commits intogrpc:masterfrom
jtattermusch:rbe2004

Conversation

@jtattermusch
Copy link
Copy Markdown
Contributor

@jtattermusch jtattermusch commented Mar 29, 2023 •
edited
Loading

  • Add a new docker image "rbe_ubuntu2004" that is built in a way that's analogous to how our other testing docker images are built (this gives us control over what exactly is contained in the docker image and ability to fine-tune our RBE configuration)
  • Switch RBE on linux to the new image (which gives us ubuntu20.04-based builds)

For some reason, RBE seems to have trouble pulling the docker image from Google Artifact Registry (GAR), which is where our public testing images normally live, so for now, I used a workaround and I upload a copy of the rbe_ubuntu2004 docker image to GCR as well, and that makes RBE works just fine (see comment in the renerate_linux_rbe_configs.sh script).

More followup items (config cleanup, getting local sanitizer builds working etc.) are in go/grpc-rbe-tech-debt-2023

@jtattermusch jtattermusch changed the title Rbe2004 Switch RBE linux build to a new custom image rbe_ubuntu2004 Apr 14, 2023
@jtattermusch jtattermusch changed the title Switch RBE linux build to a new custom image rbe_ubuntu2004 [test-infra] Switch RBE linux build to a new custom image rbe_ubuntu2004 Apr 14, 2023
@jtattermusch jtattermusch marked this pull request as ready for review April 14, 2023 08:01
@jtattermusch jtattermusch requested a review from apolcyn as a code owner April 14, 2023 08:01
@jtattermusch jtattermusch added lang/c++ release notes: no Indicates if PR should not be in release notes and removed lang/Python labels Apr 14, 2023
@jtattermusch jtattermusch requested review from ctiller and veblush April 14, 2023 08:03
jtattermusch
jtattermusch commented Apr 14, 2023
View reviewed changes
tools/remote_build/linux.bazelrc
# use MSAN-instrumented version of libc++
# TODO(jtattermusch): include the modified LD_LIBRARY_PATH in an msan-specific
# toolchain configuration, instead of needing to explicitly set it here.
build:msan --action_env=LD_LIBRARY_PATH=/usr/local/libcxx-msan/lib/x86_64-unknown-linux-gnu
Copy link
Copy Markdown
Contributor Author

@jtattermusch jtattermusch Apr 14, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

qq: until now, we haven't been setting any special flavor of libc++ for TSAN. Since the rbe_ubuntu2004 image now has also the TSAN-instrumented version of libstdc++ installed, it might be a good idea to use it (in a similar way we're setting MSAN-instrumented version of libstdc++ here). I'm leaving that as a possible followup (I don't want to change more stuff than necessary in this PR).

@jtattermusch
Copy link
Copy Markdown
Contributor Author

jtattermusch commented Apr 14, 2023 •
edited
Loading

Issues to resolve:
The examine_stack_test is failing with ASAN (for unknown reason, perhaps something missing in the RBE docker image?)

[symbolize_elf.inc : 1403] RAW: /b/f/w/bazel-out/k8-fastbuild/bin/test/core/gprpp/examine_stack_test.runfiles/com_github_grpc_grpc/_solib_k8/libsrc_Score_Slibexamine_Ustack.so: unable to find LOAD segment for pc: 0x7fc55d6b8ecb, start_addr: 7fc55d6b7000
test/core/gprpp/examine_stack_test.cc:71: Failure
Value of: stack_trace->find("GetCurrentStackTrace") != std::string::npos
  Actual: false
Expected: true
[  FAILED  ] ExamineStackTest.AbseilStackProvider (15 ms)

https://source.cloud.google.com/results/invocations/b07c5649-3972-476d-a581-06d76b5d67d1/targets;collapsed=/%2F%2Ftest%2Fcore%2Fgprpp:examine_stack_test/tests

@jtattermusch jtattermusch requested a review from gnossen April 14, 2023 13:57
@jtattermusch
Copy link
Copy Markdown
Contributor Author

jtattermusch commented Apr 14, 2023

@veblush @ctiller I'd be grateful for tips how to solve the examine_stack_test failure on ASAN (since it seems to be only thing that prevents merging).

@ctiller ctiller mentioned this pull request Apr 19, 2023
Merged
ctiller added a commit that referenced this pull request Apr 19, 2023
@ctiller
[gprpp] Better test for examine_stack (#32897)
5da7cbb 
In order to help #32748, change the
test so that it tells us what the problem is in the logs.

<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
@ctiller ctiller mentioned this pull request Apr 30, 2023
Merged
ctiller added a commit that referenced this pull request May 1, 2023
@ctiller
[examine-stack] Try to unblock Ubuntu 20.04 (#32975)
9f00eda 
Try a different approach to this test and check some non-leaf functions
in the returned text - looks like we're running into problems getting
the leaf function out of the stack trace on that platform (which is
probably fine):
https://source.cloud.google.com/results/invocations/09e8e1ea-df48-4fdb-96dd-916bd5014f90/targets/%2F%2Ftest%2Fcore%2Fgprpp:examine_stack_test/tests

Needed to unblock #32748

<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
@veblush
Copy link
Copy Markdown
Contributor

veblush commented May 1, 2023

It looks like it's now passing all Bazel RBE tests so it's worth being merged.

@ctiller ctiller removed their request for review May 1, 2023 17:24
@ctiller ctiller changed the title [test-infra] Switch RBE linux build to a new custom image rbe_ubuntu2004 [test-infra] Switch RBE linux build to a new custom image rbe_ubuntu2004. May 1, 2023
@veblush veblush merged commit 30b3d50 into grpc:master May 1, 2023
@copybara-service copybara-service bot added the imported Specifies if the PR has been imported to the internal repository label May 1, 2023
XuanWang-Amos pushed a commit to XuanWang-Amos/grpc that referenced this pull request May 1, 2023
@ctiller @XuanWang-Amos
[gprpp] Better test for examine_stack (grpc#32897)
d29dc5d 
In order to help grpc#32748, change the
test so that it tells us what the problem is in the logs.

<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
@jtattermusch
Copy link
Copy Markdown
Contributor Author

jtattermusch commented May 3, 2023

Thanks @ctiller for unblocking this!

paulosjca pushed a commit to paulosjca/grpc that referenced this pull request May 4, 2023
@ctiller @paulosjca
[examine-stack] Try to unblock Ubuntu 20.04 (grpc#32975)
4303cdd 
Try a different approach to this test and check some non-leaf functions
in the returned text - looks like we're running into problems getting
the leaf function out of the stack trace on that platform (which is
probably fine):
https://source.cloud.google.com/results/invocations/09e8e1ea-df48-4fdb-96dd-916bd5014f90/targets/%2F%2Ftest%2Fcore%2Fgprpp:examine_stack_test/tests

Needed to unblock grpc#32748

<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
paulosjca pushed a commit to paulosjca/grpc that referenced this pull request May 4, 2023
@jtattermusch @paulosjca
[test-infra] Switch RBE linux build to a new custom image rbe_ubuntu2…
8f65b4b 
…004. (grpc#32748)

- Add a new docker image "rbe_ubuntu2004" that is built in a way that's
analogous to how our other testing docker images are built (this gives
us control over what exactly is contained in the docker image and
ability to fine-tune our RBE configuration)
- Switch RBE on linux to the new image (which gives us ubuntu20.04-based
builds)

For some reason, RBE seems to have trouble pulling the docker image from
Google Artifact Registry (GAR), which is where our public testing images
normally live, so for now, I used a workaround and I upload a copy of
the rbe_ubuntu2004 docker image to GCR as well, and that makes RBE works
just fine (see comment in the `renerate_linux_rbe_configs.sh` script).

More followup items (config cleanup, getting local sanitizer builds
working etc.) are in go/grpc-rbe-tech-debt-2023
wanlin31 pushed a commit that referenced this pull request May 18, 2023
@ctiller @wanlin31
[gprpp] Better test for examine_stack (#32897)
c713159 
In order to help #32748, change the
test so that it tells us what the problem is in the logs.

<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
wanlin31 pushed a commit that referenced this pull request May 18, 2023
@ctiller @wanlin31
[examine-stack] Try to unblock Ubuntu 20.04 (#32975)
06c06f1 
Try a different approach to this test and check some non-leaf functions
in the returned text - looks like we're running into problems getting
the leaf function out of the stack trace on that platform (which is
probably fine):
https://source.cloud.google.com/results/invocations/09e8e1ea-df48-4fdb-96dd-916bd5014f90/targets/%2F%2Ftest%2Fcore%2Fgprpp:examine_stack_test/tests

Needed to unblock #32748

<!--

If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.

If your pull request is for a specific language, please add the
appropriate
lang label.

-->
wanlin31 pushed a commit that referenced this pull request May 18, 2023
@jtattermusch @wanlin31
[test-infra] Switch RBE linux build to a new custom image rbe_ubuntu2…
3c73fbc 
…004. (#32748)

- Add a new docker image "rbe_ubuntu2004" that is built in a way that's
analogous to how our other testing docker images are built (this gives
us control over what exactly is contained in the docker image and
ability to fine-tune our RBE configuration)
- Switch RBE on linux to the new image (which gives us ubuntu20.04-based
builds)

For some reason, RBE seems to have trouble pulling the docker image from
Google Artifact Registry (GAR), which is where our public testing images
normally live, so for now, I used a workaround and I upload a copy of
the rbe_ubuntu2004 docker image to GCR as well, and that makes RBE works
just fine (see comment in the `renerate_linux_rbe_configs.sh` script).

More followup items (config cleanup, getting local sanitizer builds
working etc.) are in go/grpc-rbe-tech-debt-2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@apolcyn apolcyn Awaiting requested review from apolcyn

@gnossen gnossen Awaiting requested review from gnossen

1 more reviewer

@veblush veblush veblush approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@veblush veblush

Labels

bloat/none imported Specifies if the PR has been imported to the internal repository lang/c++ per-call-memory/neutral per-channel-memory/neutral release notes: no Indicates if PR should not be in release notes title needs formatting

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@jtattermusch @veblush @ctiller @grpc-kokoro