Skip to content

ENH, TST: improvements to CI testing #2108

@tylerjereddy

Description

@tylerjereddy

There have been some discussions downstream of OpenBLAS about how we might help catch issues sooner, perhaps by contributing to your CI. I have a few thoughts here, which you may very well reject, but better to know that before I try to put time in or open a PR.

  1. OpenBLAS currently has access to 10 free parallel CI slots it isn't using, since OpenBLAS doesn't currently use Azure pipelines on PRs. We've had an excellent experience using Azure downstream alongside Travis.
  2. If you do adopt Azure and shift some of your jobs from your very-busy Travis matrix over there you may also benefit from freeing up a Travis slot for a ppc64le / POWER8 build, which we use downstream for native testing on that platform in CI.
  3. I also noticed a number of emulated ARM jobs in your travis matrix. You may also benefit from running natively, which is now available for free in CI on ARMv8 via Shippable.
  4. Adding an older OS test on Azure, like CentOS 6, to match i.e., manylinux1 / old build libs. This might need to be coordinated with R, Julia, and other downstream communities to see if there's some overlap in what is most suitable to test mutual interests.

Are you open to some of these ideas--could it be helpful? The worst part is probably that a maintainer would likely have to set up the app integrations for those CI services, but that's not too bad if we can submit the PRs to help build up the testing a bit more.

This may also help change the dialogue from us just showing up and asking for bug fixes to a way to help out a little bit so that both sides avoid rushing new releases for suddenly-discovered issues at or just after release time.

There's definitely some contentious debate about what the appropriate medium for the testing is. There's also the concern about being able to test newer archs like SkylakeX with the recent issues--that seems harder / not something more CI can easily solve unless the VMs will support the instructions somehow.

I think there's also a buildbot that tests OpenBLAS on some archs, though I'm not sure how regularly, and I believe i.e., Skylake may not be available there.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions