Skip to content

release-25.2: publish-artifacts: change GCS latest key prefix for Workload binary#156855

Closed
williamchoe3 wants to merge 1 commit intocockroachdb:release-25.2from
williamchoe3:backport25.2-156092
Closed

release-25.2: publish-artifacts: change GCS latest key prefix for Workload binary#156855
williamchoe3 wants to merge 1 commit intocockroachdb:release-25.2from
williamchoe3:backport25.2-156092

Conversation

@williamchoe3
Copy link
Copy Markdown
Contributor

@williamchoe3 williamchoe3 commented Nov 4, 2025

Backport 1/2 commits from #156092.

/cc @cockroachdb/release

Original PR that was reverted: #155789

Changes split into 2 separate commits to allow for backporting only the required changes for easier merge conflict resolution

Informs #154274

In gcs gs://cockroach-edge-artifacts-prod, change latest Workload binary key prefix from cockroach to workload
i.e.

gs://cockroach-edge-artifacts-prod/cockroach/workload.release-24.1
to
gs://cockroach-edge-artifacts-prod/workload/workload.release-24.1

This will allow latest workload binary to not be deleted after 30 days so tests can download and use workload binaries from unsupported versions (that no longer have nightlies running against them)

Notes:

  • Will need to backfill currently unsupported versions
  • ".../workload/workload...." while descriptive also sounds redundant, open to suggestions if folks don't like that prefix
  • The non latest copy of the obj i.e. gs://cockroach-edge-artifacts-prod/cockroach/workload.ee80eab77ca6434f4be7c33c0f6d29dd0b9d07b1 is still prefixed with cockroach so these objects will get deleted after 30 days with the current lifecycle policy

Edited PutNonRelease's docstring to remove "redirect" terminology, looks to be from a previous s3 implementation which is not a feature of GCS.

  • Note: I don't feel that strongly about the docstring, it just confused me a bit when reading the code and just wanted it to match what the function is actually doing

Also noticed there's no unit test coverage for non master branches, added 1 release branch test with simple coverage

Also noticed there's a slight difference between mockGCSProvider.PutObject & release.PutObject

# obj latest key names on gs://cockroach-edge-artifacts-prod
gs://cockroach-edge-artifacts-prod/cockroach/workload.release-23.2

# obj latest key names in unit tests
gs://edge-binaries-bucket/workload/workload.release-25.4/no-cache

Actual release.PutObject does not modify the key name based on the PutObjectInput.CacheControl field, but in mockGCSProvider.PutObject the "/no-cache" gets appended
I changed the behavior in the mock to match the actual function, and modified the unit tests
FWIW PutObjectInput.CacheControl is only set to False currently

  • It looks like this might be a hold over from the S3 implementation? I didn't read that diff but let me know if I should revert this change

This PR is a dependency for these roachtest changes: #155397
Also discussed other options like using a new bucket or using the release bucket here: https://cockroachlabs.slack.com/archives/C03SG8QKYRJ/p1760977289072049?thread_ts=1760976858.868889&cid=C03SG8QKYRJ
Decided on this approach for simplicity

For verification, I'm relying on the unit tests. Just assumed this wouldn't work on my local, but if there's a simple way to verify this E2E let me know and i'll do it

Will squash and write a proper commit msg after review, leaving commit separate incase i need to revert anything

Release justification: ci test artifact only changes

@blathers-crl
Copy link
Copy Markdown

blathers-crl bot commented Nov 4, 2025

Thanks for opening a backport.

Before merging, please confirm that it falls into one of the following categories (select one):

  • Non-production code changes. Includes test-only changes, build system changes, etc.
  • Fixes for serious issues. Defined in the policy as correctness, stability, or security issues, data corruption/loss, significant performance regressions, breaking working and widely used functionality, or an inability to detect and debug production issues.
  • Other approved changes. These changes must be gated behind a disabled-by-default feature flag unless there is a strong justification not to.

Add a brief release justification to the PR description explaining your selection.

Also, confirm that the change does not break backward compatibility and complies with all aspects of the backport policy.

All backports must be reviewed by the TL and EM for the owning area.

@blathers-crl blathers-crl bot added backport Label PR's that are backports to older release branches T-testeng TestEng Team labels Nov 4, 2025
@cockroach-teamcity
Copy link
Copy Markdown
Member

cockroach-teamcity commented Nov 4, 2025

This change is Reviewable

@blathers-crl
Copy link
Copy Markdown

blathers-crl bot commented Nov 4, 2025 •
edited
Loading

✅ PR #156855 is compliant with backport policy

Confidence: high
Backward compatible: true
Explanation: The pull request changes files associated with the publication of non-production artifacts to a storage bucket and a modification to the unit test file for the related 'publish-artifacts' command. The modifications in 'upload.go' and 'main_test.go' are clearly tied to changing how a 'workload' binary is uploaded and how its 'latest' key is redirected in Google Cloud Storage (GCS). The changes in 'upload.go' specifically involve adjusting the prefix for binaries and handling keys differently based on whether the binary is a workload binary or not, which does not affect the database's operational stability, data integrity, performance, or correctness. Furthermore, the change in the test file 'main_test.go' covers renamed keys in the test strings to reflect the new path behavior. These files and changes do not impact production code and match non-production file changes as they deal with the publication of artifacts, which contribute to the exempt status under the current backport policy for non-production changes. There is no critical bug fix or removal of version gates, thus maintaining backward compatibility.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@williamchoe3
publish-artifacts: change GCS latest key prefix for Workload binary
10ddcd6 
Change latest key prefix from 'cockroach' to 'workload' for workload binary to
allow for separate lifecycle policies for workload binary objects.

Informs cockroachdb#154274

Epic: None
Release note: None
This was referenced Nov 4, 2025
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

backport Label PR's that are backports to older release branches T-testeng TestEng Team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@williamchoe3 @cockroach-teamcity