Looks like release-35-arm64/test incorrectly fallbacks to downloading v3.3.0. https://screenshot.googleplex.com/Bq6FbvbfkeuvXLx

release pass downloads a older etcd binary. Either picked by providing a script, or tries to guess which to download.
Some tests like downgrade/upgrade tests, depend on this binary being present and are skipped otherwise.

If we want to improve periodics with the release pass we should think about how to validate that each branch downloaded a proper etcd binary and run the conditional tests.

Looks like release-35-arm64/test incorrectly fallbacks to downloading v3.3.0. https://screenshot.googleplex.com/Bq6FbvbfkeuvXLx

release pass downloads a older etcd binary. Either picked by providing a script, or tries to guess which to download. Some tests like downgrade/upgrade tests, depend on this binary being present and are skipped otherwise.

If we want to improve periodics with the release pass we should think about how to validate that each branch downloaded a proper etcd binary and run the conditional tests.

Great spotting. I was trying to avoid passing the MANUAL_VER=<version> parameter and instead rely on https://github.com/etcd-io/etcd/blob/release-3.5/test.sh#L639. Reviewing the shell script again I have a hunch what might be going wrong. When I get back this weekend I will do some more testing and update this pr 🙏🏻

Hey @serathius I've resolved the release e2e issue. It wasn't limited to the arm64 workflows either. The main e2e workflow was also hitting the fallback conditional, it just wasn't as problematic because that one was falling back to 3.5.0.

So long story short we have this snippet in test.sh

  UPGRADE_VER=$(git tag -l --sort=-version:refname "v3.5.*" | head -1 | cut -d- -f1)
  if [ -n "${MANUAL_VER:-}" ]; then
    # in case, we need to test against different version
    UPGRADE_VER=$MANUAL_VER
  fi
  if [[ -z ${UPGRADE_VER} ]]; then
    UPGRADE_VER="v3.5.0"
    log_warning "fallback to" ${UPGRADE_VER}
  fi

When running locally this works great and UPGRADE_VER would be defined, however by default actions/checkout does a shallow clone so running git tag doesn't list anything meaning UPGRADE_VER is empty and the fallback conditional is hit. To fix this we just need to pass the fetch-depth parameter to ensure we do a full clone and make tags available.

With this pr it's fixed now for both arm64 nightly e2e for main and release-3.5 and also standard e2e for main.

Note: For release-3.5 standard e2e we should probably backport the fix to https://github.com/etcd-io/etcd/blob/release-3.5/.github/workflows/e2e.yaml#L28 to avoid passing a static MANUAL_VER and instead just do full clone so that version will always be dynamic for previous release. If we have agreement on this I will create a good first issue for it.