/lgtm
did you look a bit why it is taking so long ? maybe there is something wrong ?

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ldemailly

The full list of commands accepted by this bot can be found here.

@ldemailly it looks like prow is running really slow for some reason. Also #3516 (merged on 2/16) fixed the fact that we were not previously running the auth tests. Now that we're running the full suite of pilot tests, its clear that the auth tests are taking a ridiculously long time on prow. When I run them locally on a GKE cluster, all of the tests take ~5min (with the same parameters as prow).

/test istio-presubmit

/test all [submit-queue is verifying that this PR is safe to merge]

can you raise this with @sebastienvas @chxchx

cc @hklai

Automatic merge from submit-queue.

@sebastienvas @chxchx @hklai

I've just managed to replicate the slowness of the pilot e2e tests from my machine on a gke cluster. The problem appears to be the fact that our test cluster has only 1 node. Previously, I was using a cluster with 4 nodes (nodeType=n1-standard-4) and the test completed in ~5 min. With an identical cluster running a single node, the same tests took 1 h 12 min and failed several tests.

As a short-term solution we could we bump up the cluster size for these tests to something reasonable (e.g. 4).

Going forward, we might consider using a large shared cluster, separating the test resources by namespace.

WDYT?

I've created a short-term fix in #3663

@nmittler: The following test failed, say /retest to rerun them all:

thanks for the investigation. in the past we've had the issue where some tasks were run on prow itself instead of being scheduled on the target cluster, is this another example of that ?
(though I guess 1 node is not much either way)

@ldemailly I think so ... the logs showed that the cluster had a single node.

@ldemailly I've added details in #3663 (comment)

ty!