Related to #43539 (comment)

The current requirements-wheel-test.txt file has quite complex and detailed version pinning, varying per architecture. I think this can be simplified because we just want to test with some older version of numpy and pandas (and the exact version isn't that important).

The file is pinning to the most recent version of numpy for a given platform / python version. In the past, that made sense because we were building against the oldest supported numpy (and so testing with a different, more recent version). Nowadays we are actually building with the most recent version of numpy, and so then it might make sense to test with an older version of numpy.