Codecov Report

Merging #497 into master will increase coverage by 3.09%.
The diff coverage is 63.32%.

@@            Coverage Diff             @@
##           master     #497      +/-   ##
==========================================
+ Coverage   73.09%   76.19%   +3.09%     
==========================================
  Files          99       99              
  Lines        4115     3701     -414     
  Branches      859      672     -187     
==========================================
- Hits         3008     2820     -188     
+ Misses        871      631     -240     
- Partials      236      250      +14

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4dd1751...4832607. Read the comment docs.

I'm still investigating why tests were hanging on CI's Linux environment. It doesn't happen in this PR after I introduced NoopCentralConfigFetcher to be used in the tests that don't test Central Configuration feature (which at the moment are all the tests since tests for Central Configuration feature were deferred to a future PR, see #496).
I'm going to use a temporary PR (#499) for experiments since the issue of hanging tests doesn't reproduce in my environment (which is Windows - it's not clear at the moment if it's a Linux vs Windows issue).
Let's consider this PR as a draft. I will remove draft status after I find the root cause. If the root cause is related to Central Configuration feature I'll add a fix.

After quite a few experiments I found quite a few issues (most are only relevant for test use cases so I'll submit fixes for them later as separate PRs) but the main root causes for tests hanging on Linux were:

1. HttpClient's SendAsync under certain circumstances not respecting cancellation (even though it accepts CancellationToken). It seems that those problematic circumstances occur when there are multiple simultaneously running tests instantiate CentralConfigFetcher which tries to connect to APM Server using HttpClient.SendAsync(... GET request ...) although I didn't succeed in creating a minimal reproducing example (i.e., the using only HttpClient without any of Agent's code). I solved the issue by wrapping HttpClient.SendAsync into AwaitOrTimeout which is cancellable,
2. CentralConfigFetcher's Dispose signalling cancellation to HttpClient's SendAsync. It turns out that CancellationTokenSource.Cancel invokes callbacks synchronously which in our cases caused deadlock because the thread calling CancellationTokenSource.Cancel after that tries to Join with the thread on which HttpClient.SendAsync is running that the first thread tries to cancel. Following the advice at the abov e link and wrapping CancellationTokenSource.Cancel into Task.Run solved the deadlock.

Are we sure we merge all this 5000+ lines without review?