Thanks this is looking nice. Are you wanting to investigate the index.name = None case here too?

Are you wanting to investigate the index.name = None case here too?

I'm still looking into this - It is proving to be pretty tricky.

@TomAugspurger - I spent more time than I expected trying to investigate the best way to preserve an index labelled as None. This PR implements the null-preservation feature by reserving the column-name "__null_dask_index__" for any index with the original name None. That is, rather than letting pandas change None to "index" during the write, I am using a more obscure name, and assuming that any index with that name should become None during a read.

We should be able to do this in a "cleaner" way by working with the pandas metadata only. However, the fact that we are starting the to_parquet logic by resetting the index (along with the need to support multiple engines, partitioning, and appending) turns this into a bit of a headache. Note that I do have a rough/messy version working for pyarrow only with this approach, but I prefer the "__null_dask_index__"-based approach for now.

Most recent commits allow us to handle None index from pandas/pyarrow. I needed to modify the read_metadata API to make this happen.

@rjzamora are there any changes we could make to the pandas metadata that might have made this whole process easier, or perhaps avoided the need for the special NONE_LABEL?

I'd like to think about this a bit more, but I get the feeling that the challenge with the label of None is all on the dask/fastparquet side.

@TomAugspurger - I may be able to work out a way to avoid avoid the special NONE_LABEL, but it seems tricky enough for fastparquet that I'd prefer to prioritize my read_parquet-dev time on other performance-related improvements. Do you get the feeling that the current state of this PR is a reasonable stopping point, or is the NONE_LABEL change too much of a "sideways" move?

Agreed with your assessment of priorities. Let me skim through here one more time, but I think we're good.

Thanks @rjzamora!