Is the thought that if the index is not a default rangeindex then it must be intentional and so we must not reset it? How are we to know that the "default" rangeindex is not intentional?

I don't have strong feeling about how we handle the default index. I just know that pandas/pyarrow avoids carrying around a filtered index if/when the original DataFrame would have had a default index anyway.

and_ has type bool -> bool -> bool so this only works if you have exactly two conjunctions.

You want (handles length-1 conjunctions as well):

disjunctions.append(functools.reduce(operator.and_, conjunctions))

Ah right - I did completely forget that operator.and_/or_ are binops

Again here, wants to be functools.reduce(operator.or_, disjunctions)

Can I propose a rewrite like this, which I think separates the handler logic from the conjunction/disjunction a little more clearly? WDYT?

(Probably needs some minor modifications for py 3.9 compat (with the type-annotations)).

from functools import reduce, partial
import operator
import numpy as np


def handle_in(column, value):
    if not isinstance(value, (list, set, tuple)):
        raise TypeError("Value of 'in' filter must be a " "list, set, or tuple.")
    return reduce(operator.or_, (operator.eq(column, value) for v in value))


def handle_is(column, value, *, negate):
    if value not in {np.nan, None}:
        raise TypeError("Value of 'is' or 'is not' filter " "must be np.nan or None.")
    return ~column.isna() if negate else column.isna()


def _apply_post_filters(df, filters: list[tuple | list[tuple]]):
    # Apply DNF filters to an in-memory DataFrame
    #
    # Disjunctive normal form (DNF) means that the inner-most
    # tuple describes a single column predicate. These inner
    # predicates are combined with an AND conjunction into a
    # larger predicate. The outer-most list then combines all
    # of the combined filters with an OR disjunction.

    if not filters:
        # No filters to apply
        return df

    handlers = {
        "==": operator.eq,
        "!=": operator.ne,
        "<": operator.lt,
        "<=": operator.le,
        ">": operator.gt,
        ">=": operator.ge,
        "in": handle_in,
        "is": partial(handle_is, negate=False),
        "is not": partial(handle_is, negate=True),
    }

    try:
        # Disjunction loop
        #
        # All elements of `disjunctions` shall be combined with
        # an `OR` disjunction (operator.or_)
        disjunctions = []
        expressions = [f if isinstance(f, list) else [f] for f in filters]
        for expr in expressions:
            conjunction = reduce(
                operator.and_,
                (handlers[op](df[column], value) for (column, op, value) in expr),
            )
            disjunctions.append(conjunction)

        return reduce(operator.or_, disjunctions)
    except (KeyError, TypeError):
        warnings.warn(...)
        return df

The reduction over the disjunctions could be merged in (so no looping) but I think it's a little less readable (would be something like):

        return reduce(
            operator.or_,
            (
                reduce(
                    operator.and_,
                    (handlers[op](df[column], value) for (column, op, value) in expr),
                )
                for expr in expressions
            ),
        )

Thanks for the suggestion. I revised the code to look more like your example (there were some minor bugs, and I could't quite get the type annotations right yet - so left them out). Wouldn't mind going with the loop-free code, but didn't get a chance to try it yet.

Can you add some tests with more than two predicates being anded together (and being ored together)?

I am not fully sure this is correct handling (my suggestion might also have been wrong).

AIUI, these are possible inputs as filters:

• (A, op, B) => (A op B)
• [(A, op, B)] => (A op B)
• [(A, op, B), (C, op, D)] => (A op B) v (C op D)
• [[(A, op, B), (C, op, D)], (E, op, F)] => ((A op B) ^ (C op D)) v (E op F)
• [[(A, op, B), (C, op, D)], [(E, op, F), (G, op H)]] => ((A op B) ^ (C op D)) v ((E op F) ^ (G op H))

So the input type is tuple | list[tuple | list[tuple]]

But this code only handles tuple | list[list[tuple]].

TBF, my code only handled list[tuple | list[tuple]].

To rephrase, who should do the sanitisation of the filters argument to this function? It would be much easier if, by the time we got here, we always just had list[list[tuple]]. That sanitisation could either be pushed up to read_parquet or else here but a little bit earlier, so we would say something like:

def _apply_filters(df, filters : tuple | list[tuple | list[tuple]]):
    if isinstance(filters, tuple): # singleton (A op B)
        filters = [filters]
    filters = [f if isinstance(f, list) else [f] for f in filters]
    ...
    for expr in filters: # Now everything is pre-processed correctly.

Oops, sorry!

Is it easier to accept empty containers as "no filters" and normalise them to None (rather than requiring specifically None as empty filters).

So:

if filters:
   ... validate
else:
   return None

OK, so now we definitively have a list-of-lists.

OK, so each entry must be a non-empty list.

And each entry in that non-empty list must be a 3-tuple of appropriate type.

tl;dr: nothing to do here.

Could write:

if not all(isinstance(item, tuple) and len(item) == 3 for item in conjunction):
   raise TypeError("Every predicate must be ...")

But I guess then it's hard to point at the bad one, unless one abuses the walrus operator like so:

if not all(isinstance(item := pred, tuple) and len(pred) == 3 for pred in conjunctions):
   raise TypeError(... item)

Which is kind of ugly.

Right - I don't expect the number of predicates to get very large here. It seems reasonable to just call _validate_predicate in a loop.