feat(Rust): Implementing Type Compatible #2492

urlyy · 2025-08-19T17:48:57Z

What does this PR do?

Modify according to the latest xlang specification. (Implemented but not tested with Java)
Improved the implementation of compatible mode by comparing the field_name and field_type of the sender and receiver to determine whether to assign or skip.

TODO

Haven’t figured out how to serialize and deserialize Box yet.
Handling of built-in unsigned types. (I will add type_id like 125=>u8 to implement se/deserialize for rust to rust)
Special handling of Option field types when extracting the generic structure of field types in macros.
In deserialization, for fields that do not exist locally and need to be skipped, automatically create a deserializer for the corresponding type and consume the byte data.

Related issues

[OSPP] [Rust] Implementing Type Forward/Backward Compatible Serialization For Fury Rust #2145

Does this PR introduce any user-facing change?

Does this PR introduce any public API change?
Does this PR introduce any binary protocol compatibility change?

Benchmark

rust/fory-core/src/buffer.rs

rust/tests/tests/test_complex_struct.rs

rust/fory-core/src/types.rs

rust/fory-core/src/meta/type_meta.rs

rust/fory-core/src/types.rs

rust/fory-core/src/meta/type_meta.rs

rust/fory-core/src/resolver/type_resolver.rs

urlyy · 2025-08-25T17:58:02Z

@theweipeng ready for review.

chaokunyang

LGTM

theweipeng

LGTM

## Why? Rust does not support objects being `null`; instead, it uses the `Option` type to wrap nullable objects. In the current Rust macros, `Option<T>` is extracted as `MetaFieldType { name: "Option", generics: [T] }`, and then compared with the `MetaFieldType { name: T, generics: [] }` received from the sender peer like java. This causes a type mismatch. Therefore, the `Option` type requires special handling. ## What does this PR do? - support `Option` in MetaFieldType se/de. Specifically, during Rust peer serialization, I will serialize `Option<T>` as `(T::type_id, nullable=true)`. During deserialization, `(T::type_id, nullable=true)` will be converted to `Option<T>`, while `(T::type_id, nullable=false)` will be converted to `T`. - correct the previous mistake in #2492, rename `FieldId` to `TypeId`. ## notice It doesn’t support handling adjacent Options, such as Option<Option<T>>. ## Related issues #2526 ## Does this PR introduce any user-facing change? - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark

…field id (#2758) ## Why?  ## What does this PR do? This pr implemented a new fory_read_compatible macro function to use match by assigned field id for better performance. - When read type meta, assign field id based on local field type info. If type info are consistent(excluding nullability), then assign remote field to local field. Otherwise, skip this remote field value when deserializing, use default value instead. - When generating fory_read_compatible code, use match by field id arm, which will compiled to switch. This is much more efficient compared to previous compare field by string - Skip using Option is local field is primitive and not Option. Now the macro generate much more efficient and compact code: ```rust fn fory_read_compatible( fory: &fory_core::fory::Fory, context: &mut fory_core::resolver::context::ReadContext, ) -> Result<Self, fory_core::error::Error> { let remote_type_id = context.reader.read_varuint32(); let meta_index = context.reader.read_varuint32(); let meta = context.get_meta(meta_index as usize); let fields = { let meta = context.get_meta(meta_index as usize); meta.get_field_infos().clone() }; let mut _f7: i16 = 0 as i16; let mut _f5: i8 = 0 as i8; let mut _last: i8 = 0 as i8; let mut _f4: Option<String> = None; let mut _f3: Option<Vec<i8>> = None; let mut _f6: Option<Vec<i16>> = None; let mut _f1: Option<HashMap<i8, Vec<i8>>> = None; let local_type_def = fory .get_type_resolver() .get_type_info(std::any::TypeId::of::<Self>()) .get_type_def(); let high_bytes = &local_type_def[..8]; let local_type_hash = i64::from_le_bytes(high_bytes.try_into().unwrap()); if meta.get_hash() == local_type_hash { <Self as fory_core::serializer::Serializer>::fory_read_data( fory, context, false, ) } else { for _field in fields.iter() { match _field.field_id { 0i16 => { if !&_field.field_type.nullable { _f7 = fory_core::serializer::read_ref_info_data::< i16, >(fory, context, true, true, false)?; } else { if (context.reader.read_bool()) { _f7 = <i16 as fory_core::serializer::ForyDefault>::fory_default(); } else { _f7 = fory_core::serializer::read_ref_info_data::< i16, >(fory, context, true, true, false)?; } } } 1i16 => { if !&_field.field_type.nullable { _f5 = fory_core::serializer::read_ref_info_data::< i8, >(fory, context, true, true, false)?; } else { if (context.reader.read_bool()) { _f5 = <i8 as fory_core::serializer::ForyDefault>::fory_default(); } else { _f5 = fory_core::serializer::read_ref_info_data::< i8, >(fory, context, true, true, false)?; } } } 2i16 => { if !&_field.field_type.nullable { _last = fory_core::serializer::read_ref_info_data::< i8, >(fory, context, true, true, false)?; } else { if (context.reader.read_bool()) { _last = <i8 as fory_core::serializer::ForyDefault>::fory_default(); } else { _last = fory_core::serializer::read_ref_info_data::< i8, >(fory, context, true, true, false)?; } } } 3i16 => { if !&_field.field_type.nullable { _f4 = Some( fory_core::serializer::read_ref_info_data::< String, >(fory, context, true, false, false)?, ); } else { if (context.reader.read_bool()) { _f4 = Some( <String as fory_core::serializer::ForyDefault>::fory_default(), ); } else { _f4 = Some( fory_core::serializer::read_ref_info_data::< String, >(fory, context, true, false, false)?, ); } } } 4i16 => { if !&_field.field_type.nullable { _f3 = Some( fory_core::serializer::read_ref_info_data::< Vec<i8>, >(fory, context, true, false, false)?, ); } else { if (context.reader.read_bool()) { _f3 = Some( <Vec< i8, > as fory_core::serializer::ForyDefault>::fory_default(), ); } else { _f3 = Some( fory_core::serializer::read_ref_info_data::< Vec<i8>, >(fory, context, true, false, false)?, ); } } } 5i16 => { if !&_field.field_type.nullable { _f6 = Some( fory_core::serializer::read_ref_info_data::< Vec<i16>, >(fory, context, true, false, false)?, ); } else { if (context.reader.read_bool()) { _f6 = Some( <Vec< i16, > as fory_core::serializer::ForyDefault>::fory_default(), ); } else { _f6 = Some( fory_core::serializer::read_ref_info_data::< Vec<i16>, >(fory, context, true, false, false)?, ); } } } 6i16 => { if !&_field.field_type.nullable { _f1 = Some( fory_core::serializer::read_ref_info_data::< HashMap<i8, Vec<i8>>, >(fory, context, true, false, false)?, ); } else { if (context.reader.read_bool()) { _f1 = Some( <HashMap< i8, Vec<i8>, > as fory_core::serializer::ForyDefault>::fory_default(), ); } else { _f1 = Some( fory_core::serializer::read_ref_info_data::< HashMap<i8, Vec<i8>>, >(fory, context, true, false, false)?, ); } } } _ => { let field_type = &_field.field_type; let read_ref_flag = fory_core::serializer::skip::get_read_ref_flag( &field_type, ); fory_core::serializer::skip::skip_field_value( fory, context, &field_type, read_ref_flag, ) .unwrap(); } } } Ok(Self { f7: _f7, f5: _f5, last: _last, f4: _f4.unwrap_or_default(), f3: _f3.unwrap_or_default(), f6: _f6.unwrap_or_default(), f1: _f1.unwrap_or_default(), }) } ``` ## Related issues #2492 #2545 Closes #2761 ## Does this PR introduce any user-facing change?  - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark

…field id (apache#2758) ## Why?  ## What does this PR do? This pr implemented a new fory_read_compatible macro function to use match by assigned field id for better performance. - When read type meta, assign field id based on local field type info. If type info are consistent(excluding nullability), then assign remote field to local field. Otherwise, skip this remote field value when deserializing, use default value instead. - When generating fory_read_compatible code, use match by field id arm, which will compiled to switch. This is much more efficient compared to previous compare field by string - Skip using Option is local field is primitive and not Option. Now the macro generate much more efficient and compact code: ```rust fn fory_read_compatible( fory: &fory_core::fory::Fory, context: &mut fory_core::resolver::context::ReadContext, ) -> Result<Self, fory_core::error::Error> { let remote_type_id = context.reader.read_varuint32(); let meta_index = context.reader.read_varuint32(); let meta = context.get_meta(meta_index as usize); let fields = { let meta = context.get_meta(meta_index as usize); meta.get_field_infos().clone() }; let mut _f7: i16 = 0 as i16; let mut _f5: i8 = 0 as i8; let mut _last: i8 = 0 as i8; let mut _f4: Option<String> = None; let mut _f3: Option<Vec<i8>> = None; let mut _f6: Option<Vec<i16>> = None; let mut _f1: Option<HashMap<i8, Vec<i8>>> = None; let local_type_def = fory .get_type_resolver() .get_type_info(std::any::TypeId::of::<Self>()) .get_type_def(); let high_bytes = &local_type_def[..8]; let local_type_hash = i64::from_le_bytes(high_bytes.try_into().unwrap()); if meta.get_hash() == local_type_hash { <Self as fory_core::serializer::Serializer>::fory_read_data( fory, context, false, ) } else { for _field in fields.iter() { match _field.field_id { 0i16 => { if !&_field.field_type.nullable { _f7 = fory_core::serializer::read_ref_info_data::< i16, >(fory, context, true, true, false)?; } else { if (context.reader.read_bool()) { _f7 = <i16 as fory_core::serializer::ForyDefault>::fory_default(); } else { _f7 = fory_core::serializer::read_ref_info_data::< i16, >(fory, context, true, true, false)?; } } } 1i16 => { if !&_field.field_type.nullable { _f5 = fory_core::serializer::read_ref_info_data::< i8, >(fory, context, true, true, false)?; } else { if (context.reader.read_bool()) { _f5 = <i8 as fory_core::serializer::ForyDefault>::fory_default(); } else { _f5 = fory_core::serializer::read_ref_info_data::< i8, >(fory, context, true, true, false)?; } } } 2i16 => { if !&_field.field_type.nullable { _last = fory_core::serializer::read_ref_info_data::< i8, >(fory, context, true, true, false)?; } else { if (context.reader.read_bool()) { _last = <i8 as fory_core::serializer::ForyDefault>::fory_default(); } else { _last = fory_core::serializer::read_ref_info_data::< i8, >(fory, context, true, true, false)?; } } } 3i16 => { if !&_field.field_type.nullable { _f4 = Some( fory_core::serializer::read_ref_info_data::< String, >(fory, context, true, false, false)?, ); } else { if (context.reader.read_bool()) { _f4 = Some( <String as fory_core::serializer::ForyDefault>::fory_default(), ); } else { _f4 = Some( fory_core::serializer::read_ref_info_data::< String, >(fory, context, true, false, false)?, ); } } } 4i16 => { if !&_field.field_type.nullable { _f3 = Some( fory_core::serializer::read_ref_info_data::< Vec<i8>, >(fory, context, true, false, false)?, ); } else { if (context.reader.read_bool()) { _f3 = Some( <Vec< i8, > as fory_core::serializer::ForyDefault>::fory_default(), ); } else { _f3 = Some( fory_core::serializer::read_ref_info_data::< Vec<i8>, >(fory, context, true, false, false)?, ); } } } 5i16 => { if !&_field.field_type.nullable { _f6 = Some( fory_core::serializer::read_ref_info_data::< Vec<i16>, >(fory, context, true, false, false)?, ); } else { if (context.reader.read_bool()) { _f6 = Some( <Vec< i16, > as fory_core::serializer::ForyDefault>::fory_default(), ); } else { _f6 = Some( fory_core::serializer::read_ref_info_data::< Vec<i16>, >(fory, context, true, false, false)?, ); } } } 6i16 => { if !&_field.field_type.nullable { _f1 = Some( fory_core::serializer::read_ref_info_data::< HashMap<i8, Vec<i8>>, >(fory, context, true, false, false)?, ); } else { if (context.reader.read_bool()) { _f1 = Some( <HashMap< i8, Vec<i8>, > as fory_core::serializer::ForyDefault>::fory_default(), ); } else { _f1 = Some( fory_core::serializer::read_ref_info_data::< HashMap<i8, Vec<i8>>, >(fory, context, true, false, false)?, ); } } } _ => { let field_type = &_field.field_type; let read_ref_flag = fory_core::serializer::skip::get_read_ref_flag( &field_type, ); fory_core::serializer::skip::skip_field_value( fory, context, &field_type, read_ref_flag, ) .unwrap(); } } } Ok(Self { f7: _f7, f5: _f5, last: _last, f4: _f4.unwrap_or_default(), f3: _f3.unwrap_or_default(), f6: _f6.unwrap_or_default(), f1: _f1.unwrap_or_default(), }) } ``` ## Related issues apache#2492 apache#2545 Closes apache#2761 ## Does this PR introduce any user-facing change?  - [ ] Does this PR introduce any public API change? - [ ] Does this PR introduce any binary protocol compatibility change? ## Benchmark

urlyy added 2 commits July 11, 2025 22:09

replace 'class_' suffix with 'type_xx' suffix

4834bf3

complete initial compatible version

4ae27ca

urlyy requested review from chaokunyang and theweipeng as code owners August 19, 2025 17:48

urlyy changed the title ~~feat(Rust): Implementing Type Compatible~~ feat(Rust): Implementing Type Compatible(WIP) Aug 19, 2025

urlyy added 2 commits August 20, 2025 02:11

fix fmt check

ac6009f

fix lint check

c36c8b9

chaokunyang reviewed Aug 20, 2025

View reviewed changes

rust/fory-core/src/buffer.rs Outdated Show resolved Hide resolved

urlyy added 2 commits August 20, 2025 12:30

fix Writer::varint32()

e0e7be5

finish field_type compare in meta

0699046

urlyy requested a review from chaokunyang August 25, 2025 08:07