Edit by bsipocz to hash out some ideas e.g. to include in CI templating, etc:

• job that tests nightly wheels/dev versions. Maybe a separate job that tests pre-releases.
• job that tests the oldest supported versions of declared dependencies
• cron jobs: notification are not working properly on github, we should implement a small tool, workaround to get notifications to maintainers
• filterwarnings to error out on warnings: expected warnings should be handled in CI, or ignored. Erroring out by default provides enormous help to notice new deprecations while upstream dependencies are still in their dev cycle (assuming there is a job that uses the dev versions/nightly builds)

Also, this could include documentation about the available CI tooling, though there is a risk involved here with keeping this up-to-date.