Skip to content

reenable tests to start debugging in the pipeline (they run fine local…#4618

Merged
mitchdenny merged 52 commits intomainfrom
mitchdenny/fix-4613
Jun 25, 2024
Merged

reenable tests to start debugging in the pipeline (they run fine local…#4618
mitchdenny merged 52 commits intomainfrom
mitchdenny/fix-4613

Conversation

@mitchdenny
Copy link
Member

@mitchdenny mitchdenny commented Jun 21, 2024

Tests were disabled in #4614

This PR should go in after #4444 (its merged with it). This will reenable the tests that were disabled and use the new testing APIs so that if/when this test fails again we will get a clearer picture of whether we are getting failures during build, or whether the app is starting and the container is unresponsive for reason.

Fixes #4613

DamianEdwards and others added 30 commits June 12, 2024 13:38
- Add ResourceNotificationService.WaitForResource
…ice.cs

Co-authored-by: Ankit Jain <radical@gmail.com>
@mitchdenny
Copy link
Member Author

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@eerhardt
Copy link
Member

#4444 is now merged. Can this PR be cleaned up to just contain the reenable test changes?

@radical
Copy link
Member

radical commented Jun 21, 2024

#4444 is now merged. Can this PR be cleaned up to just contain the reenable test changes?

done

@mitchdenny
Copy link
Member Author

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@mitchdenny
Copy link
Member Author

@radical this is ready to go in. There is some flakiness in an unrelated test case which I've added a WaitForResource(...) on so we can get a clearer picture of root cause.

@mitchdenny mitchdenny changed the title Renable tests to start debugging in the pipeline (they run fine local… reenable tests to start debugging in the pipeline (they run fine local… Jun 24, 2024
@radical
Copy link
Member

radical commented Jun 24, 2024

Thank you for fixing this, @mitchdenny !!

using var app = builder.Build();
await app.StartAsync();

var runningCts = new CancellationTokenSource(TimeSpan.FromSeconds(180));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it easier to say 3 minutes? Repeated elswhere.

@eerhardt
Copy link
Member

When we merge this, please make sure the 50 commits on this PR don't show up in the merge/squash commit message.

@@ -42,6 +42,10 @@ public void CanGetResources()
[RequiresDocker]
public async Task HttpClientGetTest()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test is still failing!

System.Net.Http.HttpRequestException : Connection refused (localhost:5150)
---- System.Net.Sockets.SocketException : Connection refused

  at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.<SendCoreAsync>g__Core|5_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Resilience.ResilienceHandler.<>c.<<SendAsync>b__3_0>d.MoveNext()
--- End of stack trace from previous location ---
   at Polly.Outcome`1.ThrowIfException()
   at Microsoft.Extensions.Http.Resilience.ResilienceHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.<SendCoreAsync>g__Core|5_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at System.Net.Http.Json.HttpClientJsonExtensions.<FromJsonAsyncCore>g__Core|12_0[TValue,TJsonOptions](HttpClient client, Task`1 responseTask, Boolean usingResponseHeadersRead, CancellationTokenSource linkedCTS, Func`4 deserializeMethod, TJsonOptions jsonOptions, CancellationToken cancellationToken)
   at Aspire.Hosting.Testing.Tests.TestingFactoryTests.HttpClientGetTest() in /_/tests/Aspire.Hosting.Testing.Tests/TestingFactoryTests.cs:line 50
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed that if starting the apphost fails for some reason like dev-certs are not installed (from the log):

mywebapp1[0] 1: 2024-06-25T00:36:33.7093120 �[41m�[30mfail�[39m�[22m�[49m: Microsoft.Extensions.Hosting.Internal.Host[11]
mywebapp1[0] 2:       Hosting failed to start
mywebapp1[0] 3:       System.InvalidOperationException: Unable to configure HTTPS endpoint. No server certificate was specified, and the default developer certificate could not be found or is out of date.
mywebapp1[0] 4:       To generate a developer certificate run 'dotnet dev-certs https'. To trust the certificate (Windows and macOS only) run 'dotnet dev-certs https --trust'.
mywebapp1[0] 5:       For more information on configuring HTTPS see https://go.microsoft.com/fwlink/?linkid=848054.
mywebapp1[0] 6:          at Microsoft.AspNetCore.Hosting.ListenOptionsHttpsExtensions.UseHttps(ListenOptions listenOptions, Action`1 configureOptions)
mywebapp1[0] 7:          at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.AddressesStrategy.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
mywebapp1[0] 8:          at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindAsync(ListenOptions[] listenOptions, AddressBindContext context, Func`2 useHttps, CancellationToken cancellationToken)
mywebapp1[0] 9:          at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.BindAsync(CancellationToken cancellationToken)
mywebapp1[0] 10:          at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
mywebapp1[0] 11:          at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
mywebapp1[0] 12:          at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__15_1(IHostedService service, CancellationToken token)

.. that doesn't prevent the test from running. And these two lines in the test run successfully:

        var rns = _app.Services.GetRequiredService<ResourceNotificationService>();
        await rns.WaitForResourceAsync("mywebapp1").WaitAsync(TimeSpan.FromSeconds(60));

        var httpClient = _app.CreateHttpClientWithResilience("mywebapp1");
        var result1 = await httpClient.GetFromJsonAsync<WeatherForecast[]>("/weatherforecast");

.. and the failure happens on the 4th line.

  Failed Aspire.Hosting.Testing.Tests.TestingFactoryTests.HttpClientGetTest [8 s]
  Error Message:
   System.Net.Http.HttpRequestException : Connection refused (localhost:5150)
---- System.Net.Sockets.SocketException : Connecti  Failed Aspire.Hosting.Testing.Tests.TestingFactoryTests.HttpClientGetTest [8 s]
  Error Message:
   System.Net.Http.HttpRequestException : Connection refused (localhost:5150)
---- System.Net.Sockets.SocketException : Connection refused
  Stack Trace:
     at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(QueueItem queueItem)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.<SendCoreAsync>g__Core|5_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Resilience.ResilienceHandler.<>c.<<SendAsync>b__3_0>d.MoveNext()
--- End of stack trace from previous location ---
   at Polly.Outcome`1.ThrowIfException()
   at Microsoft.Extensions.Http.Resilience.ResilienceHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.<SendCoreAsync>g__Core|5_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at System.Net.Http.Json.HttpClientJsonExtensions.<FromJsonAsyncCore>g__Core|12_0[TValue,TJsonOptions](HttpClient client, Task`1 responseTask, Boolean usingResponseHeadersRead, CancellationTokenSource linkedCTS, Func`4 deserializeMethod, TJsonOptions jsonOptions, CancellationToken cancellationToken)
   at Aspire.Hosting.Testing.Tests.TestingFactoryTests.HttpClientGetTest() in /Users/ankj/dev/aspire/tests/Aspire.Hosting.Testing.Tests/TestingFactoryTests.cs:line 50
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTas

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note that the dev-certs not being installed is an issue I caused locally and got this error. On CI I think it is failing because the webapp fails due to the port being in use.

But we should be failing all the tests with that fixture if host fails to start. And if it hasn't started then should .WatchResourceAsync succeeding?

Copy link
Member

@radical radical Jun 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the webapp show up as Running as soon as it is dotnet run? If so, then this doesn't handle failures. And also if the webapp's host takes time to start up then WatchResourceAsync returning wouldn't mean that the resource is actually usable. Is there something else needed to check for that?

Edit: note that I'm new to this, and might be asking non-sensical questions!

@radical
Copy link
Member

radical commented Jun 24, 2024

And
Screenshot 2024-06-24 at 16 45 27

@radical
Copy link
Member

radical commented Jun 25, 2024

I think it would be useful to add [assembly: CollectionBehavior(CollectionBehavior.CollectionPerAssembly)] to Aspire.Hosting.Testing.Tests. It would help with any tests that use the testproject and happen to run in parallel, for example, if they are listening on the default ports from the project.

@mitchdenny mitchdenny merged commit 7c73b81 into main Jun 25, 2024
@mitchdenny mitchdenny deleted the mitchdenny/fix-4613 branch June 25, 2024 11:55
@mitchdenny
Copy link
Member Author

@radical I think the TestingFactoryTests might need a bit more investigation. It passed this time around and CI and since they aren't related to my specific change I went ahead and merged, but the ports on the http endpoint for the mywebapp1 resource should be randomly assigned but aren't being (the DCP option for port randomization is set).

I'd say its something that perhaps once worked but was broken somehow.

@github-actions github-actions bot locked and limited conversation to collaborators Jul 27, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-app-model Issues pertaining to the APIs in Aspire.Hosting, e.g. DistributedApplication

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[tests] Failing WithDockerfileTests

4 participants