Add validated constructors for UnionFields #8891

friendlymatthew · 2025-11-20T17:05:37Z

related to Prevent UnionArray with Repeated Type IDs #3982

Rationale for this change

This PR introduces UnionFields::try_new and UnionFields::from_fields as replacements for the "unsafe" UnionFields::new constructor, which is now deprecated

Previously, UnionFields could be constructed with invalid invariants including negative type ids, duplicate type ids, or duplicate fields. Since UnionArrays index by type id, negative values cause panics at runtimes and duplicates break comparisons.

UnionFields::try_new validates that type ids are non-negative, unique, under 128, and that fields are unique, returning an error if any constraint is violated.

UnionFields::from_fields is a convenience constructor that auto-assigns sequential type ids starting from 0. (Will panic if it exceeds 128 elements)

Note about reviewing

I broke it up into smaller commits. The last commit updates all call sites across the different kernels

friendlymatthew · 2025-11-28T21:32:49Z

cc @alamb

alamb

Thank you so much @friendlymatthew and I am sorry for the delay in review

My only concern about this PR is the potentially new restriction to disallow repeated fields (I worry this could cause regressions for people downstream).

Otherwise this one is 👨‍🍳 👌

alamb · 2025-12-03T18:16:52Z

arrow-ipc/src/convert.rs

            let fields = match union.typeIds() {
-                None => UnionFields::new(0_i8..fields.len() as i8, fields),
-                Some(ids) => UnionFields::new(ids.iter().map(|i| i as i8), fields),
+                None => UnionFields::from_fields(fields),


this is a nice API 👍

alamb · 2025-12-03T18:21:45Z

arrow-schema/src/fields.rs

+    ///
+    /// This function returns an error if:
+    /// - Any type_id appears more than once (duplicate type ids)
+    /// - Any field appears more than once (duplicate fields)


I am not sure the arrow spec requires unique field names for union. I couldn't find any reference to such a restriction in https://arrow.apache.org/docs/format/Columnar.html#union-layout

Since I think previous versions of arrow-rs allow duplicate fields I don't think it is ok to start erroring on them in this release.

Hm, that makes sense to me. I suspect this will cause some subtleties in the way we compare Union types, but I'll make sure to document that in a follow up PR

alamb · 2025-12-03T18:23:23Z

arrow-schema/src/fields.rs

+        );
+    }
+
+    #[test]


very nice tests

kylebarron · 2025-12-03T19:31:11Z

UnionFields::from_fields is a convenience constructor that auto-assigns sequential type ids starting from 0. (Will panic if it exceeds 128 elements)

Perhaps we should error instead of panicking for >128 elements?

friendlymatthew · 2025-12-04T10:08:27Z

UnionFields::from_fields is a convenience constructor that auto-assigns sequential type ids starting from 0. (Will panic if it exceeds 128 elements)

Perhaps we should error instead of panicking for >128 elements?

Hi, that makes sense to me. I do really like from_fields being infallible for the cases when I hard code my UnionFields.

What if we include a fallible version, something like UnionFields::try_from_fields and inform users on which method to use

kylebarron · 2025-12-04T19:12:52Z

I think having from_fields and try_from_fields makes sense 👍

alamb · 2025-12-05T15:20:49Z

So I think this PR is waiting on:

Don't error with duplicated field names
from_fields / try_from_fields

friendlymatthew · 2025-12-05T16:03:34Z

So I think this PR is waiting on:

Don't error with duplicated field names

from_fields / try_from_fields

Hi, I am back from vacation and will resume work on (union data type support, variant work, blog post)

friendlymatthew · 2025-12-05T18:33:49Z

arrow-schema/src/fields.rs

+    pub fn try_from_fields<F>(fields: F) -> Result<Self, ArrowError>
+    where
+        F: IntoIterator,
+        F::Item: Into<FieldRef>,
+    {


cc @kylebarron

friendlymatthew · 2025-12-05T18:34:09Z

arrow-schema/src/fields.rs

+    /// # Panics
+    ///
+    /// Panics if the number of fields exceeds 127 (the maximum value for i8 type IDs).
+    ///
+    /// If you want to avoid panics, use [`UnionFields::try_from_fields`] instead, which
+    /// returns a `Result`.


here's a note about the fallible/infallible ways to construct UnionFields

# Which issue does this PR close? - Closes #8958 # Rationale for this change This PR introduces 2 ways to access elements of `UnionFields` by index. You can now use direct indexing (`fields[idx]`) or the safe accessor `UnionFields::get(index: usize)` to avoid potential panics Note, since we validate `UnionFields` contains only 128 elements upon construction, we can forgo this check upon read - #8891

alamb

Thank you @friendlymatthew -- I went through this PR again and it looks good to me

I left a few small comment suggestions, but I think we could do that as a follow on too

arrow-schema/src/field.rs

arrow-schema/src/fields.rs

alamb · 2025-12-12T18:22:10Z

looks like there is a clippy error

friendlymatthew · 2025-12-12T18:53:19Z

looks like there is a clippy error

That is what I get for relying on Github

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

alamb · 2025-12-15T18:11:15Z

Thanks again @friendlymatthew

Supersede #3884 Although some warnings are issued when upgrading arrow (apache/arrow-rs#8891), if we avoid deprecated functions and use newly introduced functions, we will lose compatibility with previous arrow versions. Therefore, I ignored the warnings here.

github-actions bot added arrow Changes to the arrow crate arrow-flight Changes to the arrow-flight crate parquet-variant parquet-variant* crates arrow-avro arrow-avro crate labels Nov 20, 2025

friendlymatthew mentioned this pull request Nov 20, 2025

Add comparison support for Union arrays #8838

Merged

This was referenced Nov 21, 2025

Andrew Lamb Weekly-ish Open Source plan - 2025-11-17 apache/datafusion#18711

Closed

Andrew Lamb Weekly-ish Open Source plan - 2025-11-24 apache/datafusion#18888

Closed

alamb mentioned this pull request Dec 1, 2025

Andrew Lamb Weekly-ish Open Source plan - 2025-12-01 apache/datafusion#19016

Closed

42 tasks

alamb reviewed Dec 3, 2025

View reviewed changes

friendlymatthew force-pushed the friendlymatthew/unionfields_try_new branch 2 times, most recently from 3499adc to d678756 Compare December 5, 2025 18:02

friendlymatthew mentioned this pull request Dec 5, 2025

Access UnionFields elements by index #8959

Merged

friendlymatthew commented Dec 5, 2025

View reviewed changes

alamb approved these changes Dec 11, 2025

View reviewed changes

arrow-schema/src/field.rs Outdated Show resolved Hide resolved

arrow-schema/src/fields.rs Outdated Show resolved Hide resolved

friendlymatthew added 6 commits December 12, 2025 14:06

UnionFields::try_new

5a7800c

Remove todo

55a7b35

Add UnionFields::from_fields

8e89b6c

Add coverage

885c2bf

Use UnionFields::try_new

131430d

Remove unique field check

d1dea0b

friendlymatthew and others added 4 commits December 12, 2025 14:06

Add try_from_fields

c9bb428

Update arrow-schema/src/field.rs

ef76320

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Update arrow-schema/src/fields.rs

120355b

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Clippy

245e00a

friendlymatthew force-pushed the friendlymatthew/unionfields_try_new branch from 4b5108a to 245e00a Compare December 12, 2025 19:06

alamb merged commit 63212d2 into apache:main Dec 15, 2025
29 checks passed

Jefffrey mentioned this pull request Dec 23, 2025

Fix row slice bug in Union column decoding with many columns #9000

Merged

alamb mentioned this pull request Jan 11, 2026

Upgrade DataFusion to arrow-rs/parquet 57.2.0 apache/datafusion#19355

Merged

eitsupi mentioned this pull request Jan 14, 2026

chore(rust): bump dependent datafusion and arrow apache/arrow-adbc#3894

Merged

+                      );
+                  }
+                  #[test]

Add validated constructors for UnionFields #8891

Add validated constructors for UnionFields #8891

Uh oh!

Conversation

friendlymatthew commented Nov 20, 2025 • edited by alamb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

Note about reviewing

Uh oh!

friendlymatthew commented Nov 28, 2025

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

friendlymatthew Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

kylebarron commented Dec 3, 2025

Uh oh!

friendlymatthew commented Dec 4, 2025

Uh oh!

kylebarron commented Dec 4, 2025

Uh oh!

alamb commented Dec 5, 2025

Uh oh!

friendlymatthew commented Dec 5, 2025

Uh oh!

friendlymatthew Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

friendlymatthew Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

alamb commented Dec 12, 2025

Uh oh!

friendlymatthew commented Dec 12, 2025

Uh oh!

Uh oh!

alamb commented Dec 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

friendlymatthew commented Nov 20, 2025 •

edited by alamb

Loading