Skip to content

change zarr_read_kwargs to mode argument#12205

Merged
TomAugspurger merged 16 commits intodask:mainfrom
melonora:correct_warning
Jan 24, 2026
Merged

change zarr_read_kwargs to mode argument#12205
TomAugspurger merged 16 commits intodask:mainfrom
melonora:correct_warning

Conversation

@melonora
Copy link
Copy Markdown
Contributor

@melonora melonora commented Dec 18, 2025

With #12153 I forgot to update one warning. Based on issue #12204 I went back to the code state before this PR to check whether zarr_read_kwargs is indeed the correct argument to use from now on. The problem was that **kwargs was not only used for zarr.create as stated in the docstring back then. It was also use when creating a store from a url. This could involve both read_only and mode arguments. Having read_only does not make sense to me when writing so in this PR I added an argument mode and remove zarr_read_kwargs.

I also added some checks for mode. Mode r will throw an error. If mode=="w" then the overwrite argument for zarr.create_array or zarr.create will be set to True. Interestingly with a sharded array and if an array is already stored at the location, this will cause a checksum error when lock==False. I am not certain yet whether this is a bug in zarr. For now lock==True when mode=="w" prevents the issue.

@jacobtomlinson @CSSFrancis @thewtex @jo-mueller

@melonora
Copy link
Copy Markdown
Contributor Author

I did remove zarr_read_kwargs. Not sure if I should add a deprecation for it already even though it was there just for a very short time.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Dec 18, 2025

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

0 tests  ±0   0 ✅ ±0   0s ⏱️ ±0s
0 suites ±0   0 💤 ±0 
0 files   ±0   0 ❌ ±0 

Results for commit e6bb507. ± Comparison against base commit 6dff73f.

♻️ This comment has been updated with latest results.

@melonora melonora changed the title correct warning change zarr_read_kwargs to mode argument Dec 18, 2025
Copy link
Copy Markdown
Member

@jacobtomlinson jacobtomlinson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up. I'm not concerned about a deprecation given this was only merged recently, the impact will be very low.

I'd appreciate a review from someone with zarr knowledge like @dcherian @d-v-b @TomAugspurger

@CSSFrancis
Copy link
Copy Markdown

@melonora I think that this change is good but I wonder if you actually need to deprecate kwargs?

It seems like now the kwargs are actually being used as documented. That and just passing keyword arguments is cleaner than creating a dict.

@melonora
Copy link
Copy Markdown
Contributor Author

That would indeed not be required. It could be unpacked with that name though just for more clarity and I wiuld also update the docstring with the actual documentation links. I do not have access to my pc at the moment, but will do so later

1 similar comment
@melonora

This comment was marked as duplicate.

@melonora
Copy link
Copy Markdown
Contributor Author

@CSSFrancis changes have been performed

Copy link
Copy Markdown

@CSSFrancis CSSFrancis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! @melonora thanks!

@melonora
Copy link
Copy Markdown
Contributor Author

melonora commented Jan 6, 2026

@jacobtomlinson Do you see any further action points for me to do in this PR?

@melonora
Copy link
Copy Markdown
Contributor Author

melonora commented Jan 8, 2026

FYI @will-moore

@melonora
Copy link
Copy Markdown
Contributor Author

@TomAugspurger Do you think this PR is good to go so it could be included in the first release of 2026?

@TomAugspurger
Copy link
Copy Markdown
Member

Let's get this or #12233 in today (cc @joshua-gould).

Anybody mind summarizing the differences? My main goal is to ensure that code from dask<=2025.11.0 continues to work. If we can also make code from dask==2025.12.0 then great, but IMO that's secondary.

@melonora
Copy link
Copy Markdown
Contributor Author

@TomAugspurger in #12333 read_kwargs specified in **kwargs, get reassigned to zarr_array_kwargs. This means that mode argument can now be in zarr_array_kwargs but then wouldn't be passed on to arr_store = _setup_zarr_store(url, storage_options, **zarr_read_kwargs), which has not been changed. I don't think that would be correct behaviour but I could be wrong. It isn't in the spirit of **kwargs in the PR here where these are the arguments to Group.create_array alone.

@TomAugspurger
Copy link
Copy Markdown
Member

Sorry, I'm finding it a bit hard to follow all the changes :/

Does this come down to fundamental ambiguity / disagreements over what **kwargs used to mean?

@melonora
Copy link
Copy Markdown
Contributor Author

melonora commented Jan 16, 2026

@TomAugspurger if you would like we could have a quick minicall (15 min would be enough).

But in short yes, in the othere PR the read arguments get mixed with the actual Group.create_array arguments. In this PR given that the only valid argument is mode I just added that parameter so now **kwargs are really only arguments for create_array. Also in the other PR if you pass mode in **kwargs it will have no effect because it is not passed on where it should be, which is _setup_zarr_store(url, storage_options, **zarr_read_kwargs).

However, this PR is not fully backward compatible as far as I am aware if somebody was passing on read_only in **kwargs. The reason why I removed it is because I don't know why somebody would make it read_only when writing. The mode argument I kept because there are multiple modes allowing for writing.

@melonora
Copy link
Copy Markdown
Contributor Author

@TomAugspurger I will try to be there with the next community meeting if that is more suitable. I saw that 2026.1.0 got released, but I think it would be good to get this in sooner rather than later given the API change. We don't want people to adopt the current API too much if it gets switched back to **kwargs.

@will-moore
Copy link
Copy Markdown

The change that required usage of zarr_array_kwargs to pass dimension_names when writing to zarr caused ome-zarr-py to write invalid OME-Zarr (since dimension_names were missing).

This was fixed in ome/ome-zarr-py#511 but hasn't been released yet, so it's possible that some users could be writing invalid OME-Zarr now.
I'm not sure if zarr_array_kwargs is likely to change again (or are you only discussing zarr_read_kwargs?) but either way, we should release ome-zarr-py soon so as to avoid users writing invalid data. cc @joshmoore

@melonora
Copy link
Copy Markdown
Contributor Author

@will-moore yes, working on ome-zarr-py right now but I would have liked to skip these dask versions if this PR was to be merged given the different API. After this **zarr_array_kwargs is not looking to be changed. The contents within **zarr_array_kwargs could change if it is changed upstream in zarr-python. Zarr_array_kwargs are directly the arguments of Group.create_array.

@melonora
Copy link
Copy Markdown
Contributor Author

melonora commented Jan 19, 2026

@will-moore note that that change regarding "dimension_names" is only when writing zarr_format v3 arrays, not v2.

@TomAugspurger
Copy link
Copy Markdown
Member

Thanks @melonora. Last request: can you add a note to the changelog with a summary of the changes here. I want to provide clear guidance to users about what they need to do, depending on their version of dask.

@melonora
Copy link
Copy Markdown
Contributor Author

Of course, will add it when I get to my laptop

@melonora
Copy link
Copy Markdown
Contributor Author

@TomAugspurger Done, can you please let me know if it is clear to you?

mydask.html Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this (I should be able to, after reviewing).

@TomAugspurger
Copy link
Copy Markdown
Member

Thanks @melonora.

@joshua-gould are you able to check whether this fixes the issue you ran into?

@joshua-gould
Copy link
Copy Markdown

Thanks @melonora.

@joshua-gould are you able to check whether this fixes the issue you ran into?

This fixes my issue, thanks.

@melonora
Copy link
Copy Markdown
Contributor Author

Thanks @melonora.
@joshua-gould are you able to check whether this fixes the issue you ran into?

This fixes my issue, thanks.

Great!

@TomAugspurger TomAugspurger merged commit c3784ec into dask:main Jan 24, 2026
27 checks passed
@TomAugspurger
Copy link
Copy Markdown
Member

Thanks I’ll try to get this released next week.

@melonora
Copy link
Copy Markdown
Contributor Author

Thanks @TomAugspurger !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

.to_zarr function zarr_store_kwargs not a parameter

6 participants