partition_quantiles finds incorrect minimum with large unsigned integers #7304
Description
What happened:
dask.dataframe.partitionquantiles.partition_quantiles finds an incorrect minimum and maximum value with large integer inputs.
What you expected to happen:
For partition_quantiles to find correct minimum and maximum.
Minimal Complete Verifiable Example:
In [1]: import numpy as np
...: import pandas as pd
...: import dask.dataframe as dd
...: from dask.dataframe.partitionquantiles import partition_quantiles
In [2]: pandas_df = pd.DataFrame({
...: 'a': np.array([612509347682975743, 616762138058293247], dtype=np.uint64)
...: })
...: dask_df = dd.from_pandas(pandas_df, npartitions=1)
In [3]: partition_quantiles(dask_df.a, npartitions=1).compute()
Out[3]:
0.0 612509347682975744
1.0 616762138058293248
Name: a, dtype: uint64
In [4]: dask_df.a.min().compute()
Out[4]: 612509347682975743Anything else we need to know?:
As more commentary on this bug: this is the simplest example I could come up with, but I've been having related issues with set_index on a uint64 column. Specifically, if I load data from an unsorted CSV, and the minimum value is towards the end of the dataset, then calling set_index leaves the minimum value in the last partition!
I can't share the full dataset but this is really interesting:
df = dd.read_csv(path, dtype={'uint_col': np.uint64})
df = df.set_index('uint_col')
df_min_value = df.index.min().compute()
# 612509347682975743
partition_min_value = df.partitions[-1].compute().index.min()
# 612509347682975743
divisions_min_value = df.divisions[0]
# 612509347682975744
df_min_value < divisions_min_value
# True
partition_min_value < divisions_min_value
# False
partition_min_value < np.uint64(divisions_min_value)
# True
df_min_value == partition_min_value
# True
type(df_min_value)
# numpy.int64
type(partition_min_value)
# numpy.uint64
type(divisions_min_value)
# int🤯
So I'm guessing that there are some buffer overflow issues with uints?
Edit: It seems like it might be helpful to store more things in the same datatype as the index, and prevent unnecessary coercion? For example, this seems like a bug:
df.index.min().dtype
# dtype('uint64')
df.index.min().compute().dtype
# dtype('int64')I don't understand why the dtype should change there.
Environment:
- Dask version: 2021.2.0
- Python version: 3.8.3
- Operating System: Mac OSX 11.1
- Install method (conda, pip, source): pip