Skip to content

feat(rust): support skipping fields bytes when deserializing in compatible mode #2545

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
chaokunyang merged 1 commit into from
Aug 29, 2025
Merged

feat(rust): support skipping fields bytes when deserializing in compatible mode #2545

chaokunyang merged 1 commit into from
Aug 29, 2025

Conversation

@urlyy
Copy link
Contributor

@urlyy urlyy commented Aug 28, 2025

What does this PR do?

Support skipping unneeded field bytes when deserializing in compatible mode.

Related issues

#2531

@urlyy
Copy link
Contributor Author

urlyy commented Aug 28, 2025 •
edited
Loading

Oh, maybe this filename isn’t quite appropriate? nonexistent.rs should probably be skip.rs instead. Since it skips not only when the type or field is nonexistent but also when it doesn’t match.

urlyy
urlyy commented Aug 28, 2025
View reviewed changes
rust/fory-derive/src/object/read.rs
#(#pattern_items),*
_ => {
// skip bytes
println!("no need to deserialize {:?}:{:?}", _field.field_name.as_str(), _field.field_type);
Copy link
Contributor Author

@urlyy urlyy Aug 28, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe need to be replaced as println!("skip {:?}:{:?}", _field.field_name.as_str(), _field.field_type);. Or just remove this. I just want to print a warning or log to user.

urlyy
urlyy commented Aug 28, 2025
View reviewed changes
rust/fory-core/src/serializer/nonexistent.rs
Err(anyhow!("Unknown ref flag, value:{ref_flag}"))?
}
} else {
unreachable!()
Copy link
Contributor Author

@urlyy urlyy Aug 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Option or Enum or some other will enter here. The implementation and unit tests will be added in other PRs.

@urlyy
Copy link
Contributor Author

urlyy commented Aug 29, 2025 •
edited
Loading

@chaokunyang Please review.

@chaokunyang chaokunyang changed the title feat(Rust): Support skipping unneeded field bytes when deserializing in compatible mode feat(Rust): Support skipping fields bytes when deserializing in compatible mode Aug 29, 2025
@chaokunyang chaokunyang changed the title feat(Rust): Support skipping fields bytes when deserializing in compatible mode feat(rust): support skipping fields bytes when deserializing in compatible mode Aug 29, 2025
@chaokunyang chaokunyang merged commit 707de60 into apache:main Aug 29, 2025
55 checks passed
This was referenced Oct 6, 2025
Open
Merged
chaokunyang added a commit that referenced this pull request Oct 13, 2025
@chaokunyang
feat(rust): fast fory_read_compatible macro to use match by assigned …
bf6ebf9 
…field id (#2758)

## Why?

<!-- Describe the purpose of this PR. -->

## What does this PR do?

This pr implemented a new fory_read_compatible macro function to use
match by assigned field id for better performance.
- When read type meta, assign field id based on local field type info.
If type info are consistent(excluding nullability), then assign remote
field to local field. Otherwise, skip this remote field value when
deserializing, use default value instead.
- When generating fory_read_compatible code, use match by field id arm,
which will compiled to switch. This is much more efficient compared to
previous compare field by string
- Skip using Option is local field is primitive and not Option.

Now the macro generate much more efficient and compact code:
```rust
fn fory_read_compatible(
fory: &fory_core::fory::Fory,
context: &mut fory_core::resolver::context::ReadContext,
) -> Result<Self, fory_core::error::Error> {
let remote_type_id = context.reader.read_varuint32();
let meta_index = context.reader.read_varuint32();
let meta = context.get_meta(meta_index as usize);
let fields = {
    let meta = context.get_meta(meta_index as usize);
    meta.get_field_infos().clone()
};
let mut _f7: i16 = 0 as i16;
let mut _f5: i8 = 0 as i8;
let mut _last: i8 = 0 as i8;
let mut _f4: Option<String> = None;
let mut _f3: Option<Vec<i8>> = None;
let mut _f6: Option<Vec<i16>> = None;
let mut _f1: Option<HashMap<i8, Vec<i8>>> = None;
let local_type_def = fory
    .get_type_resolver()
    .get_type_info(std::any::TypeId::of::<Self>())
    .get_type_def();
let high_bytes = &local_type_def[..8];
let local_type_hash = i64::from_le_bytes(high_bytes.try_into().unwrap());
if meta.get_hash() == local_type_hash {
    <Self as fory_core::serializer::Serializer>::fory_read_data(
        fory,
        context,
        false,
    )
} else {
    for _field in fields.iter() {
        match _field.field_id {
            0i16 => {
                if !&_field.field_type.nullable {
                    _f7 = fory_core::serializer::read_ref_info_data::<
                        i16,
                    >(fory, context, true, true, false)?;
                } else {
                    if (context.reader.read_bool()) {
                        _f7 = <i16 as fory_core::serializer::ForyDefault>::fory_default();
                    } else {
                        _f7 = fory_core::serializer::read_ref_info_data::<
                            i16,
                        >(fory, context, true, true, false)?;
                    }
                }
            }
            1i16 => {
                if !&_field.field_type.nullable {
                    _f5 = fory_core::serializer::read_ref_info_data::<
                        i8,
                    >(fory, context, true, true, false)?;
                } else {
                    if (context.reader.read_bool()) {
                        _f5 = <i8 as fory_core::serializer::ForyDefault>::fory_default();
                    } else {
                        _f5 = fory_core::serializer::read_ref_info_data::<
                            i8,
                        >(fory, context, true, true, false)?;
                    }
                }
            }
            2i16 => {
                if !&_field.field_type.nullable {
                    _last = fory_core::serializer::read_ref_info_data::<
                        i8,
                    >(fory, context, true, true, false)?;
                } else {
                    if (context.reader.read_bool()) {
                        _last = <i8 as fory_core::serializer::ForyDefault>::fory_default();
                    } else {
                        _last = fory_core::serializer::read_ref_info_data::<
                            i8,
                        >(fory, context, true, true, false)?;
                    }
                }
            }
            3i16 => {
                if !&_field.field_type.nullable {
                    _f4 = Some(
                        fory_core::serializer::read_ref_info_data::<
                            String,
                        >(fory, context, true, false, false)?,
                    );
                } else {
                    if (context.reader.read_bool()) {
                        _f4 = Some(
                            <String as fory_core::serializer::ForyDefault>::fory_default(),
                        );
                    } else {
                        _f4 = Some(
                            fory_core::serializer::read_ref_info_data::<
                                String,
                            >(fory, context, true, false, false)?,
                        );
                    }
                }
            }
            4i16 => {
                if !&_field.field_type.nullable {
                    _f3 = Some(
                        fory_core::serializer::read_ref_info_data::<
                            Vec<i8>,
                        >(fory, context, true, false, false)?,
                    );
                } else {
                    if (context.reader.read_bool()) {
                        _f3 = Some(
                            <Vec<
                                i8,
                            > as fory_core::serializer::ForyDefault>::fory_default(),
                        );
                    } else {
                        _f3 = Some(
                            fory_core::serializer::read_ref_info_data::<
                                Vec<i8>,
                            >(fory, context, true, false, false)?,
                        );
                    }
                }
            }
            5i16 => {
                if !&_field.field_type.nullable {
                    _f6 = Some(
                        fory_core::serializer::read_ref_info_data::<
                            Vec<i16>,
                        >(fory, context, true, false, false)?,
                    );
                } else {
                    if (context.reader.read_bool()) {
                        _f6 = Some(
                            <Vec<
                                i16,
                            > as fory_core::serializer::ForyDefault>::fory_default(),
                        );
                    } else {
                        _f6 = Some(
                            fory_core::serializer::read_ref_info_data::<
                                Vec<i16>,
                            >(fory, context, true, false, false)?,
                        );
                    }
                }
            }
            6i16 => {
                if !&_field.field_type.nullable {
                    _f1 = Some(
                        fory_core::serializer::read_ref_info_data::<
                            HashMap<i8, Vec<i8>>,
                        >(fory, context, true, false, false)?,
                    );
                } else {
                    if (context.reader.read_bool()) {
                        _f1 = Some(
                            <HashMap<
                                i8,
                                Vec<i8>,
                            > as fory_core::serializer::ForyDefault>::fory_default(),
                        );
                    } else {
                        _f1 = Some(
                            fory_core::serializer::read_ref_info_data::<
                                HashMap<i8, Vec<i8>>,
                            >(fory, context, true, false, false)?,
                        );
                    }
                }
            }
            _ => {
                let field_type = &_field.field_type;
                let read_ref_flag = fory_core::serializer::skip::get_read_ref_flag(
                    &field_type,
                );
                fory_core::serializer::skip::skip_field_value(
                        fory,
                        context,
                        &field_type,
                        read_ref_flag,
                    )
                    .unwrap();
            }
        }
    }
    Ok(Self {
        f7: _f7,
        f5: _f5,
        last: _last,
        f4: _f4.unwrap_or_default(),
        f3: _f3.unwrap_or_default(),
        f6: _f6.unwrap_or_default(),
        f1: _f1.unwrap_or_default(),
    })
}
```

## Related issues

#2492 #2545

Closes #2761

## Does this PR introduce any user-facing change?

<!--
If any user-facing interface changes, please [open an
issue](https://github.com/apache/fory/issues/new/choose) describing the
need to do so and update the document if necessary.

Delete section if not applicable.
-->

- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?

## Benchmark

<!--
When the PR has an impact on performance (if you don't know whether the
PR will have an impact on performance, you can submit the PR first, and
if it will have impact on performance, the code reviewer will explain
it), be sure to attach a benchmark data here.

Delete section if not applicable.
-->
rajuyadav03 pushed a commit to rajuyadav03/fory that referenced this pull request Oct 20, 2025
@chaokunyang @rajuyadav03
feat(rust): fast fory_read_compatible macro to use match by assigned …
db4ea28 
…field id (apache#2758)

## Why?

<!-- Describe the purpose of this PR. -->

## What does this PR do?

This pr implemented a new fory_read_compatible macro function to use
match by assigned field id for better performance.
- When read type meta, assign field id based on local field type info.
If type info are consistent(excluding nullability), then assign remote
field to local field. Otherwise, skip this remote field value when
deserializing, use default value instead.
- When generating fory_read_compatible code, use match by field id arm,
which will compiled to switch. This is much more efficient compared to
previous compare field by string
- Skip using Option is local field is primitive and not Option.

Now the macro generate much more efficient and compact code:
```rust
fn fory_read_compatible(
fory: &fory_core::fory::Fory,
context: &mut fory_core::resolver::context::ReadContext,
) -> Result<Self, fory_core::error::Error> {
let remote_type_id = context.reader.read_varuint32();
let meta_index = context.reader.read_varuint32();
let meta = context.get_meta(meta_index as usize);
let fields = {
    let meta = context.get_meta(meta_index as usize);
    meta.get_field_infos().clone()
};
let mut _f7: i16 = 0 as i16;
let mut _f5: i8 = 0 as i8;
let mut _last: i8 = 0 as i8;
let mut _f4: Option<String> = None;
let mut _f3: Option<Vec<i8>> = None;
let mut _f6: Option<Vec<i16>> = None;
let mut _f1: Option<HashMap<i8, Vec<i8>>> = None;
let local_type_def = fory
    .get_type_resolver()
    .get_type_info(std::any::TypeId::of::<Self>())
    .get_type_def();
let high_bytes = &local_type_def[..8];
let local_type_hash = i64::from_le_bytes(high_bytes.try_into().unwrap());
if meta.get_hash() == local_type_hash {
    <Self as fory_core::serializer::Serializer>::fory_read_data(
        fory,
        context,
        false,
    )
} else {
    for _field in fields.iter() {
        match _field.field_id {
            0i16 => {
                if !&_field.field_type.nullable {
                    _f7 = fory_core::serializer::read_ref_info_data::<
                        i16,
                    >(fory, context, true, true, false)?;
                } else {
                    if (context.reader.read_bool()) {
                        _f7 = <i16 as fory_core::serializer::ForyDefault>::fory_default();
                    } else {
                        _f7 = fory_core::serializer::read_ref_info_data::<
                            i16,
                        >(fory, context, true, true, false)?;
                    }
                }
            }
            1i16 => {
                if !&_field.field_type.nullable {
                    _f5 = fory_core::serializer::read_ref_info_data::<
                        i8,
                    >(fory, context, true, true, false)?;
                } else {
                    if (context.reader.read_bool()) {
                        _f5 = <i8 as fory_core::serializer::ForyDefault>::fory_default();
                    } else {
                        _f5 = fory_core::serializer::read_ref_info_data::<
                            i8,
                        >(fory, context, true, true, false)?;
                    }
                }
            }
            2i16 => {
                if !&_field.field_type.nullable {
                    _last = fory_core::serializer::read_ref_info_data::<
                        i8,
                    >(fory, context, true, true, false)?;
                } else {
                    if (context.reader.read_bool()) {
                        _last = <i8 as fory_core::serializer::ForyDefault>::fory_default();
                    } else {
                        _last = fory_core::serializer::read_ref_info_data::<
                            i8,
                        >(fory, context, true, true, false)?;
                    }
                }
            }
            3i16 => {
                if !&_field.field_type.nullable {
                    _f4 = Some(
                        fory_core::serializer::read_ref_info_data::<
                            String,
                        >(fory, context, true, false, false)?,
                    );
                } else {
                    if (context.reader.read_bool()) {
                        _f4 = Some(
                            <String as fory_core::serializer::ForyDefault>::fory_default(),
                        );
                    } else {
                        _f4 = Some(
                            fory_core::serializer::read_ref_info_data::<
                                String,
                            >(fory, context, true, false, false)?,
                        );
                    }
                }
            }
            4i16 => {
                if !&_field.field_type.nullable {
                    _f3 = Some(
                        fory_core::serializer::read_ref_info_data::<
                            Vec<i8>,
                        >(fory, context, true, false, false)?,
                    );
                } else {
                    if (context.reader.read_bool()) {
                        _f3 = Some(
                            <Vec<
                                i8,
                            > as fory_core::serializer::ForyDefault>::fory_default(),
                        );
                    } else {
                        _f3 = Some(
                            fory_core::serializer::read_ref_info_data::<
                                Vec<i8>,
                            >(fory, context, true, false, false)?,
                        );
                    }
                }
            }
            5i16 => {
                if !&_field.field_type.nullable {
                    _f6 = Some(
                        fory_core::serializer::read_ref_info_data::<
                            Vec<i16>,
                        >(fory, context, true, false, false)?,
                    );
                } else {
                    if (context.reader.read_bool()) {
                        _f6 = Some(
                            <Vec<
                                i16,
                            > as fory_core::serializer::ForyDefault>::fory_default(),
                        );
                    } else {
                        _f6 = Some(
                            fory_core::serializer::read_ref_info_data::<
                                Vec<i16>,
                            >(fory, context, true, false, false)?,
                        );
                    }
                }
            }
            6i16 => {
                if !&_field.field_type.nullable {
                    _f1 = Some(
                        fory_core::serializer::read_ref_info_data::<
                            HashMap<i8, Vec<i8>>,
                        >(fory, context, true, false, false)?,
                    );
                } else {
                    if (context.reader.read_bool()) {
                        _f1 = Some(
                            <HashMap<
                                i8,
                                Vec<i8>,
                            > as fory_core::serializer::ForyDefault>::fory_default(),
                        );
                    } else {
                        _f1 = Some(
                            fory_core::serializer::read_ref_info_data::<
                                HashMap<i8, Vec<i8>>,
                            >(fory, context, true, false, false)?,
                        );
                    }
                }
            }
            _ => {
                let field_type = &_field.field_type;
                let read_ref_flag = fory_core::serializer::skip::get_read_ref_flag(
                    &field_type,
                );
                fory_core::serializer::skip::skip_field_value(
                        fory,
                        context,
                        &field_type,
                        read_ref_flag,
                    )
                    .unwrap();
            }
        }
    }
    Ok(Self {
        f7: _f7,
        f5: _f5,
        last: _last,
        f4: _f4.unwrap_or_default(),
        f3: _f3.unwrap_or_default(),
        f6: _f6.unwrap_or_default(),
        f1: _f1.unwrap_or_default(),
    })
}
```

## Related issues

apache#2492 apache#2545

Closes apache#2761

## Does this PR introduce any user-facing change?

<!--
If any user-facing interface changes, please [open an
issue](https://github.com/apache/fory/issues/new/choose) describing the
need to do so and update the document if necessary.

Delete section if not applicable.
-->

- [ ] Does this PR introduce any public API change?
- [ ] Does this PR introduce any binary protocol compatibility change?

## Benchmark

<!--
When the PR has an impact on performance (if you don't know whether the
PR will have an impact on performance, you can submit the PR first, and
if it will have impact on performance, the code reviewer will explain
it), be sure to attach a benchmark data here.

Delete section if not applicable.
-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chaokunyang chaokunyang chaokunyang approved these changes

@theweipeng theweipeng Awaiting requested review from theweipeng theweipeng is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@urlyy @chaokunyang