I think dd.core.Series and dd.core.DataFrame already have _meta, no? In that case this may be redundant and could be removed in favor of the condition immediately below: if hasattr(x, "_meta").

Right, make_meta_object does something like the following to capture these cases:

    if hasattr(x, "_meta"):
        return x._meta
    elif is_arraylike(x) and x.shape:
        return x[:0]

However, if that logic is already captured in make_meta_object (which is now registered with make_meta_obj), it doesn't seem like this check is necessary at all. Is this correct?

Good catch. I think your assessment is right, Rick. Unless I'm missing something, we could indeed remove both checks here.

Yes, this is a redundant check. I got rid of it in make_meta_object. The reason being since make_meta dispatch was registered against generic python object type, all objects would simply pass through the make_meta, but with the change being made this won't happen so it will be the task of make_meta_util to handle this. The is_arraylike check however shouldn't be done in make_meta_util as objects like pd.Series/cudf.Series need to go through the dispatch mechanism below.

Does this mean we still need to remove the make_meta.register(object) definition in dask-cudf before the original issue is resolved? Not a problem if so, I am just trying to understand everything.

Correct. We'll need dask-cudf side changes to accommodate this new dispatch - make_meta_object.

Right, make_meta_object does something like the following to capture these cases:

    if hasattr(x, "_meta"):
        return x._meta
    elif is_arraylike(x) and x.shape:
        return x[:0]

However, if that logic is already captured in make_meta_object (which is now registered with make_meta_obj), it doesn't seem like this check is necessary at all. Is this correct?

Doesn't look like this change is necessary (I don't think parent_meta is used in new_dd_object)

Needed for this: https://github.com/dask/dask/pull/7586/files#r637579242

Are there expected cases where a user would need the option of passing this in. Is there an expected reason that the _meta of the first _Frame in args is not a good assumption?

This would be similar to the above one, when _Frame list can be all empty or the args has no _Frame at all and the meta at hand is way too generic.

I don't think we need this, but I may be missing something.

Similar issue here, meta can be too generic :

> /nvme/0/pgali/cudf/dask/dask/dataframe/core.py(5535)apply_concat_apply()
(Pdb) dfs[0]._meta
Series([], dtype: int64)
(Pdb) meta
1

It would be nice if we didn't need to pass parent metadata into map_partitions since we could easily use the _meta on the first _Frame object within that function.

The reason map_partitions would need a parent_meta is dfs frame objects list can be empty too:

Another case where I'd like to avoid passing parent_meta if we can avoid it.

This case is an example where we have meta is (q, 'f8') and we cannot really know what the parents meta is.

Can we not pass in parent_meta=self._meta here? Is the Scalar meta still too "general"?

Yes, for some scalars the meta seems to be too generic like int, hence the need to pass self._parent_meta here :

for example

> /nvme/0/pgali/cudf/dask/dask/dataframe/core.py(269)_scalar_binary()
(Pdb) self._meta
1
(Pdb) self._parent_meta
Series([], dtype: float64)

Is there a good reason to pass parent_meta if a meta object is already provided? That is, shouldn't meta already be an appropriate DataFrame type? My intuition tells me that the above changes should just be swapping make_meta with make_meta_util.

That is, shouldn't meta already be an appropriate DataFrame type?

Sadly, nope. For example:

> /nvme/0/pgali/cudf/dask/dask/dataframe/io/io.py(603)from_delayed()
(Pdb) meta
[('a', 'f8'), ('b', 'f8'), ('c', 'f8'), ('d', 'f8')]
(Pdb) parent_meta
Empty DataFrame
Columns: [a, b, c, d]
Index: []