Skip to content

cdc/bank roachtest pull 260MB off a 3rd party vendor upon every CI run, and fails if upstream unavailable #51543

@knz

Description

@knz

Describe the problem

The cdc/bank roachtest runs the following command every time it runs:

 curl -s https://packages.confluent.io/archive/4.0/confluent-oss-4.0.0-2.11.tar.gz | tar -xz -C /tmp/confluent

I went and checked and that is a 262MB archive to download (compressed).

The archive is not cached, unlike the builder image, so that's a mandatory ingress cost on every CI run.

Moreover, today the upstream HTTP server is saying "no" and is causing all the CI runs to fails.

Expected behavior

The archive should be embedded in the builder image, and/or the fetch should use a cached copy if it was already downloaded earlier on the TC agent.

(At the very least we should be fetching from a proxy cache inside the CRL infra so that the CI downloads are internal to GCP).

cc @jlinder @tbg for triage.

Epic DEVINF-109

Jira issue: CRDB-4033

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-cdcChange Data CaptureA-roachprodA-testingTesting tools and infrastructureS-3-productivitySevere issues that impede the productivity of CockroachDB developers.T-cdcbranch-masterFailures and bugs on the master branch.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions