• Creating a new release
• Out of schedule releases
• Undoing a release

See Upgrading to a New Release for an overview of CiviForm release practices.

Releases are drafted by the on-call engineer during the first week of their shift on Tuesday before 12pm Pacific Time and released on Tuesday (or Wednesday if delayed) during the second week of their shift. See Out-of-schedule releases for information about cutting a release at other times and other ways to release hotfixes.

New releases go through a QA process that takes them from a draft state to fully published. The QA process is best-effort for official release images, and may be skipped if resources are not available.

Releases are ideally published on Tuesdays, but may be pushed to Wednesdays if QA is not available in time for a Tuesday release.

Check the #eng-ci Slack channel for the latest successful SNAPSHOT id that was deployed in:

• Seattle Deploy with Env: staging
• AWS Staging Deploy
• AWS QA Deploy
• AWS e2e test

Extract the commit SHA from it. The SHA is the middle item in the SNAPSHOT-sha-hash ID. For example for SNAPSHOT-fa971bb-1666156765 ID the commit SHA is fa971bb.

NOTE: If the last successful snapshot for ALL environments is not the most recent one, contact a Seattle software engineer to request that they deploy the release candidate to their staging environment. If changes are introduced after Monday night that should be included in the release, we'll want to re-deploy QA.

Before proceding you'll need to determine the Commit SHA and the next Release number.

This is the middle item in the SNAPSHOT-sha-hash ID from Step 1.

• The release version string looks like "vX.Y.Z". Previously, this followed semver. Now, while this still looks like semver, it isn't semver. Generally, we'll always increment Y on a release and set Z to 0. If we do an unscheduled release in order to fix a bug in the last release, we'll increment Z.
• Incrementing X will only happen after discussion with the wider team, and is reserved for major changes that require IT admins to deploy the release with care (e.g. we've updated the PostgreSQL version and there's a risk of data loss if things go wrong deploying the release), or perhaps a major change in how CiviForm works.
• Docs explanation about versioning
• Review the version number from the last release and determine the new release number. To see which commits are going into the release, you can compare the main branch to the last tag (example). Note that we aren't actually releasing HEAD of main, so only look at commits up to the SHA determined in step 1.
• Make sure to include the 'v' prefix.

With those values create the draft release with one of the following options.

• [Step 1 & 2] Navigate to the Create Release GitHub Action.
• [Step 3] Click the "Run workflow" button
  • Supply the Commit SHA and Release number values
• [Step 4] Click the "Run workflow" button

Setup:

• Login to docker. If you're an Exygy employee, login as the civiformexygy user. Otherwise, login with your DockerHub account if you're a part of the DockerHub civiform organization. Contact an Exygy engineer if you need to be added.
• Generate a GH access token and provide it as documented.

When ready run create-release script:

bin/create-release SHA vNumber

If the script fails due to missing requests library. Try the following:

python3 -m ensurepip
pip3 install requests --user

The release script will then:

1. Validates the commit SHA is a commit on main and RELEASE_NUMBER matches the convention
2. Tags the git commit with the release number, annotated with description that includes the email address of the release caller, and pushes it to github
3. Pulls the existing server image from Docker Hub (identified by the commit SHA), tags it with the release number and pushes it to Docker Hub
4. Calls the GitHub API to create a new draft release
5. Prints the GH URL to the draft release

Ensure both the civiform and cloud-deploy-infra repos and civiform Docker are tagged appropriately after running the Create Release action.

If you used the GitHub action in Step 1: Navigate to the repo's releases page and find the new draft release.

If you did the manual option: Follow the GitHub URL to the draft release printed at the end of the bin/create-release script step.

Next, edit the release notes to include the following:

• Description of fixed bugs
• Description of new features
• Database schema changes
• Other stateful changes (such as stored file key name schema)
• API version changes and deprecation notices
• New configuration values
• Infrastructure changes
• Dependency changes (for the server only) including new, removed, and version updates

Check every PR that is not Under Development for a Release Notes section, if it's present use that instead of the title.

Additionally, add a sentence or two to the top of the release notes summarizing what is in the release. It should be friendly to non-technical readers (think CiviForm and Program Admins). Feel free to use phrases like "small usability improvements" rather than listing each improvement.

NOTE: When drafting the summary blurb, you can run bin/summarize-release-messages locally to output PR titles, authors, and descriptions for all changes that have gone in since the latest release. This saves you the time and effort of clicking through each PR in the release to read their summaries. This script accepts an optional release version tag if you want a summary of commits since an older release.

Message Matthew Sprenke (City of Seattle) in the #eng-release-discussion channel on Slack, notifying QA that a new release candidate is ready for review. Make sure to do this AFTER Seattle has deployed the release candidate to staging if it is not the most recent snapshot.

Also tag any engineers who made changes that are going out in the release so they can review the summary you wrote at the top of the release notes.

A new draft release vX.X.X is ready for testing. Release notes can be found here: [INSERT LINK TO GITHUB DRAFT RELEASE HERE].
Draft release is available on the following stagings:
https://civiformstage.seattle.gov/
https://qa.civiform.dev/
https://staging-aws.civiform.dev/

Please perform necessary QA checks before EOD Tuesday [INSERT DATE] PT time. If you find any issues with the release - reply to this announcement.

Folks to notify: @Matthew Sprenke

Wait until Matthew provides feedback on the release to proceed to step 6, do not proceed if he reports any blocking issues with the release candidate.

If Matthew is not available, recruit engineers, PMs, and other contributors to evaluate the release candidate. Direct them to specific areas of interest based on the release notes. You can also use Seattle's QA test plan as a guide for baseline functionality to test.

On Tuesday (or Wednesday if delayed) of your second week on call, you will publish the draft release. Make sure QA has been completed and release notes look good before publishing. Publish the draft release (click the edit button and then "Publish" at the bottom of the screen) and then email the release notes to civiform-announce@googlegroups.com and civiform-technical@googlegroups.com. Ensure the sentence or two summary you added to the release notes is also present in the emails, in addition to the link pointing to the release notes on GitHub.

If you need to release a change for one or more individual Governments, you can provide them with a Docker image tag that contains your changes. A tagged image is automatically created when changes are merged to main. If you have already merged your changes, you will need to find the correct tag in our Docker Hub. The correct SNAPSHOT tag will have your commit hash in the middle eg: SNAPSHOT-[your commit hash]-12345.

If you want to release changes from a branch, you can use this action to build and push an image to Docker Hub from your branch. The created image will have a tag prepended with DEV with your commit hash in the middle eg. DEV-[your commit hash]-12345.

Once you have the correct image tag, share it with the Government and instruct them to add it to their civiform_config.sh file (or whatever file controls the settings for their deployment) in their local copy of the civiform-deploy repo as the value for the CIVIFORM_VERSION var (eg. export CIVIFORM_VERSION="SNAPSHOT-[your commit hash]-12345).

Cutting an official release mid-cycle is generally discouraged and should only be done if it is a significant issue that affects all Governments using CiviForm (eg. if there is risk of data loss or exposure). If this is the case, the preference is to cut a release from the tip of main the way we normally do. If there are already changes in main that you do not want to go out in this release, talk with the rest of the engineering team to figure out the best approach. It is possible to cut a release from a branch, but we have not had to do this yet 🤜 🪵 and would need to ensure it wouldn't affect subsequent releases.

Sometimes, we find problems with a release after we've run the automation to create a release draft. If we decide we can't do a release this week, we need to undo some of the artifacts created by the release automation so that they don't collide with the release next week.

1. Go to https://github.com/civiform/civiform/releases and delete the draft release.
2. Go to https://github.com/civiform/civiform/tags and delete the release tag.
3. Go to https://github.com/civiform/cloud-deploy-infra/tags and delete the release tag.
4. Go to https://hub.docker.com/repository/docker/civiform/civiform/tags and delete the release tag. An Exygy engineer will need to do this with the civiformexygy account.
5. Revert the docs commit created by civiform-github-automation for the release at https://github.com/civiform/docs/commits/main/.