read_parquet filter bug with nulls #9845
Description
Describe the issue:
When using dd.read_parquet on a column with nulls: filtering on null equality doesn't work, != on another value also ends up removing nulls
Minimal Complete Verifiable Example:
dd.read_parquet("test.parquet", filters=[("a", "==", np.nan)]).compute() # empty
dd.read_parquet("test.parquet", filters=[("a", "!=", np.nan)]).compute() # works
dd.read_parquet("test.parquet", filters=[("a", "!=", 1)]).compute() # empty
dd.read_parquet("test.parquet", filters=[("a", "==", None)]).compute() # empty
dd.read_parquet("test.parquet", filters=[("b", "!=", 2 )]).compute() # 13 rows instead of 14
dd.read_parquet("test.parquet", filters=[("c", "!=", "a")]).compute() # empty
dd.read_parquet("test.parquet", filters=[("c", "!=", None)]).compute() # empty
dd.read_parquet("test.parquet", filters=[("c", "!=", "")]).compute() # worksFor table creation:
import pandas as pd
import numpy as np
import pyarrow as pa
import pyarrow.parquet as pq
from dask import dataframe as dd
df = pd.DataFrame()
df["a"] = [1, None]*5 + [None]*5
df["b"] = np.arange(14).tolist() + [None]
df["c"] = ["a", None]*2 + [None]*11
pa_table = pa.Table.from_pandas(df)
pq.write_table(pa_table, "test.parquet", row_group_size=10)Anything else we need to know?:
Environment:
- Dask version: 2023.1.1
- Python version: 3.9
- Operating System: ubuntu 20.04
- Install method (conda, pip, source): pip