Skip to content

Conversation

@chaokunyang
Copy link
Collaborator

@chaokunyang chaokunyang commented Dec 4, 2025

Why?

The previous Go serialization implementation had several limitations that affected both performance and developer experience:

  1. Performance bottlenecks: Mixed read/write state in a single context, lack of fast paths for common types, and redundant indirections
  2. API usability: Interface{}-based API caused unnecessary allocations and lacked compile-time type safety
  3. Concurrency gaps: No built-in thread-safe support, forcing users to manage instances manually
  4. Code maintainability: Mixed responsibilities and unclear separation of concerns made the codebase hard to navigate

This refactoring addresses these fundamental issues to make Fory Go more performant, safer, and easier to use.

What does this PR do?

This PR comprehensively refactors the Fory Go serialization implementation:

1. New Generic API Design (Closes #2992 )

  • Introduces type-safe generic APIs: Serialize[T] and Deserialize[T]
  • Pointer-based parameters eliminate unnecessary heap allocations and struct copies
  • Separate SerializeAny/DeserializeAny for polymorphic use cases
  • Cleaner distinction between typed and untyped serialization

2. Performance Optimizations (Closes #3016 )

  • Separate ReadContext and WriteContext eliminate mixed state overhead
  • Fast paths for primitives: bool, int8/16/32/64, float32/64, string
  • Optimized serializers for common collections: []int, []string, []bool, etc.
  • Direct memory access for struct field serialization
  • Inlined hot path methods for compiler optimization

3. Thread-Safe Package (Closes #3017 )

  • New fory/threadsafe package using sync.Pool for zero-contention concurrency
  • Safe concurrent use without manual instance management
  • Same API surface as main package (generic and non-generic)
  • Global convenience functions for quick usage

4. Code Organization (Closes #3018 )

  • Separate files: reader.go, writer.go for clear read/write separation
  • Type-specific serializers: array.go, primitive.go, pointer.go, time.go, etc.
  • Renamed files for clarity: resolver.gometa_string_resolver.go, type.gotype_resolver.go
  • Better documentation and code structure

Additional Improvements

  • Updated codegen to align with new API
  • Improved test coverage with new typed tests
  • Updated documentation and examples

Related issues

Closes #2992 - API redesign with generics
Closes #3016 - Performance optimizations
Closes #3017 - Thread-safe package
Closes #3018 - Code refactoring
Closes #3010
Closes #3011

Closes #2611
Closes #2661
Closes #2639

#2982
#1017

Does this PR introduce any user-facing change?

Yes, this PR introduces breaking API changes:

Breaking Changes:

  1. Main API changes:

    • Old: fory.Serialize(value) → New: fory.Serialize(foryInstance, &value) or fory.Marshal(&value)
    • Old: fory.Deserialize(data, &target) → New: fory.Deserialize(foryInstance, data, &target) or fory.Unmarshal(data, &target)
    • Constructor: fory.NewFory(refTracking)fory.New(fory.WithTrackRef(enabled))
  2. Type-safe generic API (recommended):

    // Serialization
    data, err := fory.Marshal(&myStruct)
    
    // Deserialization
    var result MyStruct
    err := fory.Unmarshal(data, &result)
  3. Thread-safe usage:

    import fory "github.com/apache/fory/go/fory/threadsafe"
    
    // Global instance (recommended for concurrent use)
    data, err := fory.Marshal(&myStruct)
    var result MyStruct
    err := fory.Unmarshal(data, &result)
    
    // Or custom instance
    f := fory.New(fory.WithTrackRef(true))
    data, err := fory.Serialize(f, &myStruct)

Migration Guide:

Users need to update their code to:

  • Use pointer parameters instead of values
  • Switch to generic APIs for better performance
  • Use the threadsafe package for concurrent applications
  • Update to functional options pattern for configuration

Does this PR introduce any public API change?

Yes, extensive public API changes as detailed above. The new API is more performant and type-safe, but requires migration from the old interface{}-based API.

Does this PR introduce any binary protocol compatibility change?

No, the binary protocol remains fully compatible. This is purely an implementation and API refactoring. Data serialized with the old implementation can be deserialized with the new one and vice versa.

Go struct values are value types, not reference types, so they
should not participate in reference tracking. Previously, Go
incorrectly used JSON marshaling to deduplicate struct values
by content, which was both semantically wrong and slow.

Changes:
- Go: Struct values now write NotNullValueFlag (-1) instead of
  RefValueFlag (0), correctly treating them as non-referenceable
- Go: Removed unused basicValueCache and json import
- Python: Fixed reference() to handle empty read_ref_ids when
  NotNullValueFlag is received (prevents segfault)

This aligns Go behavior with the xlang spec and C++/Rust
implementations where value types don't use reference tracking.
@chaokunyang chaokunyang marked this pull request as draft December 4, 2025 17:49
@chaokunyang chaokunyang force-pushed the refactor_golang_serializaiton branch 2 times, most recently from 03617c1 to abd762e Compare December 5, 2025 05:26
Resolved go.mod conflict by using newer yaml.v3 version (v3.0.1).
- Use snake_case field names in hash computation for reflection
- Add FORY_STRING_ARRAY and other array types to isPrimitiveArrayType
- Use generic sliceSerializer for struct fields for cross-language compatibility
- Update codegen to use context-based API with WriteContext/ReadContext
- Add ptrToCodegenSerializer wrapper for pointer type serialization
- Fix Unmarshal to handle pointer dereference cases
Update golang.org/x/tools from v0.1.12 to v0.39.0 for compatibility
with Go 1.23+. The old version caused a panic during code generation.
@chaokunyang chaokunyang force-pushed the refactor_golang_serializaiton branch from abd762e to 8eb36d3 Compare December 5, 2025 06:15
@chaokunyang chaokunyang force-pushed the refactor_golang_serializaiton branch from ffe32c1 to 64ed87f Compare December 5, 2025 15:57
@chaokunyang chaokunyang marked this pull request as ready for review December 6, 2025 17:54
@chaokunyang chaokunyang changed the title feat(go): generics based new fory golang serializaiton implementation feat(go): new fory golang serializaiton implementation Dec 6, 2025
@chaokunyang chaokunyang requested a review from urlyy December 6, 2025 17:54
@chaokunyang chaokunyang changed the title feat(go): new fory golang serializaiton implementation refactor(go): redesign serialization implementation with performance and usability improvements Dec 6, 2025
@chaokunyang chaokunyang mentioned this pull request Dec 6, 2025
17 tasks
@chaokunyang chaokunyang merged commit e93fdea into apache:main Dec 7, 2025
63 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants