Skip to content

publish-artifacts: change GCS latest key prefix for Workload binary#156092

Closed
williamchoe3 wants to merge 2 commits intocockroachdb:masterfrom
williamchoe3:wchoe/154274-publish-artifacts-change-gcs-key-for-workload-binary-2
Closed

publish-artifacts: change GCS latest key prefix for Workload binary#156092
williamchoe3 wants to merge 2 commits intocockroachdb:masterfrom
williamchoe3:wchoe/154274-publish-artifacts-change-gcs-key-for-workload-binary-2

Conversation

@williamchoe3
Copy link
Copy Markdown
Contributor

@williamchoe3 williamchoe3 commented Oct 24, 2025

Original PR that was reverted: #155789

Changes split into 2 separate commits to allow for backporting only the required changes for easier merge conflict resolution

Informs #154274

In gcs gs://cockroach-edge-artifacts-prod, change latest Workload binary key prefix from cockroach to workload
i.e.

gs://cockroach-edge-artifacts-prod/cockroach/workload.release-24.1
to
gs://cockroach-edge-artifacts-prod/workload/workload.release-24.1

This will allow latest workload binary to not be deleted after 30 days so tests can download and use workload binaries from unsupported versions (that no longer have nightlies running against them)

Notes:

  • Will need to backfill currently unsupported versions
  • ".../workload/workload...." while descriptive also sounds redundant, open to suggestions if folks don't like that prefix
  • The non latest copy of the obj i.e. gs://cockroach-edge-artifacts-prod/cockroach/workload.ee80eab77ca6434f4be7c33c0f6d29dd0b9d07b1 is still prefixed with cockroach so these objects will get deleted after 30 days with the current lifecycle policy

Edited PutNonRelease's docstring to remove "redirect" terminology, looks to be from a previous s3 implementation which is not a feature of GCS.

  • Note: I don't feel that strongly about the docstring, it just confused me a bit when reading the code and just wanted it to match what the function is actually doing

Also noticed there's no unit test coverage for non master branches, added 1 release branch test with simple coverage

Also noticed there's a slight difference between mockGCSProvider.PutObject & release.PutObject

# obj latest key names on gs://cockroach-edge-artifacts-prod
gs://cockroach-edge-artifacts-prod/cockroach/workload.release-23.2

# obj latest key names in unit tests
gs://edge-binaries-bucket/workload/workload.release-25.4/no-cache

Actual release.PutObject does not modify the key name based on the PutObjectInput.CacheControl field, but in mockGCSProvider.PutObject the "/no-cache" gets appended
I changed the behavior in the mock to match the actual function, and modified the unit tests
FWIW PutObjectInput.CacheControl is only set to False currently

  • It looks like this might be a hold over from the S3 implementation? I didn't read that diff but let me know if I should revert this change

This PR is a dependency for these roachtest changes: #155397
Also discussed other options like using a new bucket or using the release bucket here: https://cockroachlabs.slack.com/archives/C03SG8QKYRJ/p1760977289072049?thread_ts=1760976858.868889&cid=C03SG8QKYRJ
Decided on this approach for simplicity

For verification, I'm relying on the unit tests. Just assumed this wouldn't work on my local, but if there's a simple way to verify this E2E let me know and i'll do it

Will squash and write a proper commit msg after review, leaving commit separate incase i need to revert anything

@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@williamchoe3 williamchoe3 force-pushed the wchoe/154274-publish-artifacts-change-gcs-key-for-workload-binary-2 branch from 0165611 to 93299c1 Compare October 24, 2025 17:14
Change latest key prefix from 'cockroach' to 'workload' for workload binary to
allow for separate lifecycle policies for workload binary objects.

Informs cockroachdb#154274

Epic: None
Release note: None
@williamchoe3 williamchoe3 force-pushed the wchoe/154274-publish-artifacts-change-gcs-key-for-workload-binary-2 branch from 93299c1 to 3c9254b Compare October 24, 2025 17:18
Mocked PutObject no longer appends /no-cache which matches real function
behavior. Added unit test for release branch case. Changed docstring to remove
s3 terminology

Informs cockroachdb#154274

Epic: None
Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants