-
Notifications
You must be signed in to change notification settings - Fork 4.1k
cdc/bank roachtest pull 260MB off a 3rd party vendor upon every CI run, and fails if upstream unavailable #51543
Copy link
Copy link
Open
Labels
A-cdcChange Data CaptureChange Data CaptureA-roachprodA-testingTesting tools and infrastructureTesting tools and infrastructureS-3-productivitySevere issues that impede the productivity of CockroachDB developers.Severe issues that impede the productivity of CockroachDB developers.T-cdcbranch-masterFailures and bugs on the master branch.Failures and bugs on the master branch.
Description
Describe the problem
The cdc/bank roachtest runs the following command every time it runs:
curl -s https://packages.confluent.io/archive/4.0/confluent-oss-4.0.0-2.11.tar.gz | tar -xz -C /tmp/confluent
I went and checked and that is a 262MB archive to download (compressed).
The archive is not cached, unlike the builder image, so that's a mandatory ingress cost on every CI run.
Moreover, today the upstream HTTP server is saying "no" and is causing all the CI runs to fails.
Expected behavior
The archive should be embedded in the builder image, and/or the fetch should use a cached copy if it was already downloaded earlier on the TC agent.
(At the very least we should be fetching from a proxy cache inside the CRL infra so that the CI downloads are internal to GCP).
Epic DEVINF-109
Jira issue: CRDB-4033
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
A-cdcChange Data CaptureChange Data CaptureA-roachprodA-testingTesting tools and infrastructureTesting tools and infrastructureS-3-productivitySevere issues that impede the productivity of CockroachDB developers.Severe issues that impede the productivity of CockroachDB developers.T-cdcbranch-masterFailures and bugs on the master branch.Failures and bugs on the master branch.