Skip to content

Conversation

@alamb
Copy link
Contributor

@alamb alamb commented Aug 18, 2025

Which issue does this PR close?

Rationale for this change

It was not clear to me what the expected behavior for round trip through row converter was for DictionaryArrays, so let's document what @tustvold says here: #8067 (comment)

I think the issue is that Datafusion is not handling the fact that row encoding "hydrates" dictionaries. It should be updated to understand that List<Dictionary<...>> will be converted to List<...>, much like it already handles this for the non-nested case. Converting back to a dictionary is expensive, and likely pointless, not to mention a breaking change.

What changes are included in this PR?

Document expected behavior with english comments and doc test

Are these changes tested?

Yes (doctests)

Are there any user-facing changes?

More docs, no behavior change

@alamb alamb marked this pull request as ready for review August 18, 2025 19:31
@alamb alamb added the documentation Improvements or additions to documentation label Aug 18, 2025
@github-actions github-actions bot added the arrow Changes to the arrow crate label Aug 18, 2025
@alamb alamb requested a review from tustvold August 18, 2025 19:55
@alamb
Copy link
Contributor Author

alamb commented Aug 20, 2025

@ding-young and @LiaCastaneda, do you have a few minutes to review this PR and ensure this documentation also matches your understanding?

Copy link
Contributor

@ding-young ding-young left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code in the docs works fine, and I think the description matches my understanding. Thank you.

@alamb
Copy link
Contributor Author

alamb commented Aug 21, 2025

Thank you @ding-young for the review

@alamb alamb merged commit 8875504 into apache:main Aug 21, 2025
13 checks passed
@LiaCastaneda
Copy link

This lgmt too thank you again @alamb , sorry for the late reply, didn't have acces to my laptop!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arrow Changes to the arrow crate documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants