Add array/map/fixed schema resolution and default value support to arrow-avro codec #8292

jecsand838 · 2025-09-08T02:37:24Z

Which issue does this PR close?

This work continues arrow-avro schema resolution support and aligns behavior with the Avro spec.

Related to: Add Avro Support #4886 (“Add Avro Support”): ongoing work to round out the reader/decoder, including schema resolution and type promotion.
Follow-ups/Context: Add schema resolution and type promotion support to arrow-avro Decoder #8124 (schema resolution & type promotion for the decoder), Added arrow-avro enum mapping support for schema resolution #8223 (enum mapping for schema resolution). These previous efforts established the foundations that this PR extends to default values and additional resolvable types.

Rationale for this change

Avro’s schema resolution requires readers to reconcile differences between the writer and reader schemas, including:

Using record-field default values when the writer lacks a field present in the reader; defaults must be type-correct (i.e., union defaults match the first union member; bytes/fixed defaults are JSON strings).
Recursively resolving arrays (by item schema) and maps (by value schema).
Resolving fixed types (size and unqualified name must match) and erroring when they do not.

Prior to this change, arrow-avro’s resolution handled some cases but lacked full Codec support for default values and for resolving array/map/fixed shapes between writer and reader. This led to gaps when reading evolved data or datasets produced by heterogeneous systems. This PR implements these missing pieces so the Arrow reader behaves per the spec in common evolution scenarios.

What changes are included in this PR?

This PR modifies arrow-avro/src/codec.rs to extend the schema-resolution path

Default value handling for record fields
- Reads and applies default values when the reader expects a field absent from the writer, including nested defaults.
- Validates defaults per the Avro spec (e.g., union defaults match the first schema; bytes/fixed defaults are JSON strings).
Array / Map / Fixed schema resolution
- Array: recursively resolves item schemas (writer↔reader).
- Map: recursively resolves value schemas.
- Fixed: enforces matching size and (unqualified) name; otherwise signals an error, consistent with the spec.
Codec updates
- Refactors internal codec logic to support the above during decoding, including resolution for record fields and nested defaults. (See commit message for the high-level summary.)

Are these changes tested?

Yes. This PR includes new unit tests in arrow-avro/src/codec.rs covering:

Default validation & persistence
- Null/union‑nullability rules; metadata persistence of defaults (AVRO_FIELD_DEFAULT_METADATA_KEY).
AvroLiteral Parsing
- Range checks for i32/f32; correct literals for i64/f64; Utf8/Utf8View; uuid strings (RFC‑4122).
- Byte‑range mapping for bytes/fixed defaults; Fixed(n) length enforcement; decimal on fixed vs bytes; duration/interval fixed 12‑byte enforcement.
Collections & records
- Array/map defaults shape; enum symbol validity; record defaults for missing fields, required‑field errors, and honoring field‑level defaults; skip‑fields retained for writer‑only fields.
Resolution mechanics
- Element promotion (int to long) for arrays; reader metadata precedence for colliding attributes; fixed name/size match including alias.

Are there any user-facing changes?

N/A

…rrow-avro codec.rs Implements handling of default values, including validation during schema resolution. Adds support for resolving differences in `array`, `map`, and `fixed` schemas between writer and reader. Updates codec logic to handle nested default values and enhances resolution for record fields.

jecsand838 · 2025-09-09T22:24:39Z

@alamb Would you be able to review this PR whenever you get a chance? About 60% of the new code are tests.

nathaniel-d-ef · 2025-09-10T17:00:22Z

arrow-avro/src/codec.rs

+            }
+        }
+
+        // Handle JSON nulls per-spec: allowed only for `null` type or unions with null FIRST


I recall we're supporting first and second positions elsewhere in this crate re: impala - can you clarify why it's not an issue that we're diverging from that here?

That's a really good catch. I was planning to bring in NullSecond support here as a part of the full Dense Union type decoding support I'm finishing up right now. Which is why I left the comment calling it out.

nathaniel-d-ef · 2025-09-10T17:17:29Z

arrow-avro/src/codec.rs

+                            })?;
+                            let lit = f.data_type().parse_default_literal(&v)?;
+                            out.insert(name, lit);
+                        } else if f.data_type().nullability() == Some(Nullability::default()) {


This is probably a cheap check - worth it to move it up in the chain before the others?

@nathaniel-d-ef Good callout. I see what you're getting at. I went ahead and cleaned this code up a bit more.

alamb

Thanks @jecsand838 and @nathaniel-d-ef

As always, I don't really have a huge amount of insight about the avro spec, so I wouldn't be able to catch any avro corner cases here, but the code seems clear to me and well tested. Thank you

arrow-avro/src/codec.rs

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

alamb · 2025-09-11T15:56:30Z

Thank you @jecsand838 and @nathaniel-d-ef

# Which issue does this PR close? This work continues arrow-avro schema resolution support and aligns behavior with the Avro spec. - **Related to**: #4886 (“Add Avro Support”): ongoing work to round out the reader/decoder, including schema resolution and type promotion. - **Follow-ups/Context**: #8292 (Add array/map/fixed schema resolution and default value support to arrow-avro codec), #8124 (schema resolution & type promotion for the decoder), #8223 (enum mapping for schema resolution). These previous efforts established the foundations that this PR extends to default values and additional resolvable types. # Rationale for this change Avro’s specification requires readers to materialize default values when a field exists in the **reader** schema but not in the **writer** schema, and to validate defaults (i.e., union defaults must match the first branch; bytes/fixed defaults must be JSON strings; enums may specify a default symbol for unknown writer symbols). Implementing this behavior makes `arrow-avro` more standards‑compliant and improves interoperability with evolving schemas. # What changes are included in this PR? **High‑level summary** * **Refactor `RecordDecoder`** around a simpler **`Projector`**‑style abstraction that consumes `ResolvedRecord` to: (a) skip writer‑only fields, and (b) materialize reader‑only defaulted fields, reducing branching in the hot path. (See commit subject and record decoder changes.) **Touched files (2):** * `arrow-avro/src/reader/record.rs` - refactor decoder to use precomputed mappings and defaults. * `arrow-avro/src/reader/mod.rs` - add comprehensive tests for defaults and error cases (see below). # Are these changes tested? Yes, new integration tests cover both the **happy path** and **validation errors**: * `test_schema_resolution_defaults_all_supported_types`: verifies that defaults for boolean/int/long/float/double/bytes/string/date/time/timestamp/decimal/fixed/enum/duration/uuid/array/map/nested record and unions are materialized correctly for all rows. * `test_schema_resolution_default_enum_invalid_symbol_errors`: invalid enum default symbol is rejected. * `test_schema_resolution_default_fixed_size_mismatch_errors`: mismatched fixed/bytes default lengths are rejected. These tests assert the Avro‑spec behavior (i.e., union defaults must match the first branch; bytes/fixed defaults use JSON strings). # Are there any user-facing changes? N/A

github-actions bot added arrow Changes to the arrow crate arrow-avro arrow-avro crate labels Sep 8, 2025

jecsand838 mentioned this pull request Sep 8, 2025

Add projection with default values support to RecordDecoder #8293

Merged

Merge branch 'main' into avro-schema-resolution-resolvable-types

1f6cf02

jecsand838 force-pushed the avro-schema-resolution-resolvable-types branch from 57db02d to 1f6cf02 Compare September 8, 2025 17:00

Merge branch 'main' into avro-schema-resolution-resolvable-types

508ef98

jecsand838 force-pushed the avro-schema-resolution-resolvable-types branch 2 times, most recently from 2b58c87 to 2b4cd3d Compare September 8, 2025 22:46

Merge branch 'main' into avro-schema-resolution-resolvable-types

5e4744c

jecsand838 force-pushed the avro-schema-resolution-resolvable-types branch from 2b4cd3d to 5e4744c Compare September 9, 2025 02:39

Merge branch 'main' into avro-schema-resolution-resolvable-types

da299fb

nathaniel-d-ef reviewed Sep 10, 2025

View reviewed changes

alamb approved these changes Sep 10, 2025

View reviewed changes

arrow-avro/src/codec.rs Outdated Show resolved Hide resolved

jecsand838 and others added 3 commits September 10, 2025 16:14

Update arrow-avro/src/codec.rs

b28397d

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Merge branch 'main' into avro-schema-resolution-resolvable-types

41f1c63

Address PR Comments

ea56641

jecsand838 force-pushed the avro-schema-resolution-resolvable-types branch from 692167d to ea56641 Compare September 10, 2025 22:00

nathaniel-d-ef approved these changes Sep 11, 2025

View reviewed changes

alamb merged commit 567f441 into apache:main Sep 11, 2025
23 checks passed

jecsand838 deleted the avro-schema-resolution-resolvable-types branch September 22, 2025 04:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add array/map/fixed schema resolution and default value support to arrow-avro codec #8292

Add array/map/fixed schema resolution and default value support to arrow-avro codec #8292

Uh oh!

jecsand838 commented Sep 8, 2025

Uh oh!

jecsand838 commented Sep 9, 2025

Uh oh!

nathaniel-d-ef Sep 10, 2025

Uh oh!

jecsand838 Sep 10, 2025

Uh oh!

nathaniel-d-ef Sep 10, 2025

Uh oh!

jecsand838 Sep 10, 2025

Uh oh!

alamb left a comment

Uh oh!

Uh oh!

Uh oh!

alamb commented Sep 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add array/map/fixed schema resolution and default value support to arrow-avro codec #8292

Add array/map/fixed schema resolution and default value support to arrow-avro codec #8292

Uh oh!

Conversation

jecsand838 commented Sep 8, 2025

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

jecsand838 commented Sep 9, 2025

Uh oh!

nathaniel-d-ef Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

jecsand838 Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

nathaniel-d-ef Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

jecsand838 Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

alamb commented Sep 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants