This is a combination of several factors:

• A checkout with cold caches taking 14mins during peak hours
• This is impacting mostly presubmits
• 5X increase in checkouts because of the sharding of recipes v2
• Checkouts finding cache problems taking twice the execution time because of deleting and recreating

The main culprit of this is throttling of checkouts from GoB.