We are not allowed to modify the array, its hard to handle NAN's correctly in avx512_argsort routines for float and double.