Skip to content

Workaround the ExternalDnsWithTracingClientServerTest flake on Linux#21179

Merged
jtattermusch merged 2 commits intogrpc:masterfrom
jtattermusch:external_dns_with_tracing_deflake
Nov 14, 2019
Merged

Workaround the ExternalDnsWithTracingClientServerTest flake on Linux#21179
jtattermusch merged 2 commits intogrpc:masterfrom
jtattermusch:external_dns_with_tracing_deflake

Conversation

@jtattermusch
Copy link
Copy Markdown
Contributor

Fixes #18126

I've done quite a bit of experimenting with the test on mono on linux:
The test itself always finishes fine, but the mono binary gets stuck on shutdown. I attached gdb and didn't see anything suspicious and it seemed that all the managed threads have finished. My theory is that the binary fails to exit because of this:

  • mono runtime keeps track of "managed" and "native" threads and the main thread won't exit until all the managed threads have finished.
  • when a managed callback is invoked on an otherwise "native" thread (this happens when the log handler callback runs), mono promotes the native thread to a "managed" thread.
  • when the test's Main returns, some of the native threads (e.g. timer thread) that usually don't prevent the process from exiting have been promoted to "managed" and they keep the process alive. (forever, which eventually leads to the test timeout).

The workaround is to disable the logging/tracing part of the test on Linux and Mac, because we are interested in testing on Windows only anyway (where the test passes reliably, according to the test result history)

The PR also fixes a few typos.

@jtattermusch jtattermusch added lang/C# release notes: no Indicates if PR should not be in release notes labels Nov 13, 2019
@jtattermusch jtattermusch requested a review from apolcyn November 13, 2019 17:09
@jtattermusch jtattermusch merged commit abdf470 into grpc:master Nov 14, 2019
@lock lock bot locked as resolved and limited conversation to collaborators Feb 12, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

lang/C# release notes: no Indicates if PR should not be in release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

csharp.Grpc.IntegrationTesting.ExternalDnsWithTracingClientServerTest flake

2 participants