Purpose

This issue tracks the arcade-services repository rollout. On top of the Rollout instructions described on the wiki, it provides the person responsible for the rollout checklist of the steps that need to be performed to rollout services in this repository. All relevant information, including the rollout PR, issues encountered during the rollout and steps taken to resolve them should be linked or added to this issue to keep full audit trail of changes rolled out to production.

Process

Build status check (Monday)

• Check the status of the dotnet-arcade-services-weekly pipeline
• Rotate any secrets that need manual rotation
• In case the build is failing, try to fix it and ensure the arcade-services-internal-ci pipeline is green

Rollout preparation (Tuesday)

• Check the vendor prepared the rollout:
  • Thread on the Rollout channel
  • Rollout issue in AzDO
  • Rollout PR in arcade-services
• In case there is a problem with the CI build, notify the Rollout channel
• Link the rollout PR to the Rollout data section of this issue
• Double-check that the release notes contain all information
• Merge the already prepared rollout PR
• Ensure the build is green and stops at the "Approval" phase

Rollout day (Wednesday)

• Approve the Approval stage of the rollout build (that has been already started the day before)
• Monitor the rollout build for failures.
  • Note: this Maestro exceptions query might help in diagnosing issues.
• Keep track of any issues encountered during the rollout either directly in this issue, or in a dedicated issue linked to this issue
• Update the rollout stats in the Stats section below
• Notify the Rollout channel
• Close this issue with closing comment describing a high-level summary of issues encountered during the rollout

Rollout data

Rollout PRs

• The main PR: [Rollout] Production rollout, 2023-07-12 #2739

Rollout times

Use the following Kusto query to gather data about rollout times:

• Pre-Approval run time: 36 minutes
• Post-Approval run time: 1 hour 6 minutes

Useful links

• AzDO pipelines
  • arcade-services-internal-ci
  • dotnet-arcade-services-weekly
• Rollout channel
• Maestro exceptions
• Deployment Policy