Skip to content

ffi: Add WIP Serializer class for the next IR stream format.#446

Merged
LinZhihao-723 merged 4 commits into
y-scope:mainfrom
LinZhihao-723:ffi_serializer
Jun 22, 2024
Merged

ffi: Add WIP Serializer class for the next IR stream format.#446
LinZhihao-723 merged 4 commits into
y-scope:mainfrom
LinZhihao-723:ffi_serializer

Conversation

@LinZhihao-723

Copy link
Copy Markdown
Member

Description

When migrating to the next IR stream format, there are two important differences:

  1. The serialization is no longer stateless in the next IR format. The stream has to maintain a schema tree across the entire IR stream.
  2. In the current IR format, bytes can be written into the IR buffer in the same order as encoded: first encoded variables, then logtype, and finally timestamp. This is not true in the next IR format. For structured key-value pairs, the overall packet of a new record would be: [new_schema_tree_nodes, keys, values]. Each sub-packet has to be encoded in a separate buffer and merged into one after the entire record is processed. This means we need more than one internal buffer during the serialization, even though only the final merged buffer is exposed to the upper layer APIs.

To adapt to these changes, we wrap all the necessary data structures into a new class named Serializer to keep track of the states, as well as maintain necessary data structures. In details:

  • It maintains an underlying schema tree for the stream.
  • It maintains all the necessary internal buffers for serialization. These internal buffers are private members. In the future PRs, serialization methods will be added as member functions and they will have direct access to these buffers.
  • It provides an API to access the view of the serialized IR buffer (containing serialized bytes).

Note: this class is designed to provide serialization functionalities only. The upper-level caller should be responsible for writing the serialized bytes into IO streams properly. In addition, it doesn't provide a call to terminate the IR stream. The upper-level caller should decide when to terminate the stream by append clp::ffi::ir_stream::cProtocol::Eof at the end of the stream.

Validation performed

  • Added a unit test case to ensure the serializer can be successfully created with the preamble section serialized properly; also tested to ensure it can properly serialize a UTC offset change.
  • Passsed clang-tidy checks for the Serializer implementation.
  • Didn't execute clang-tidy against the unit test file since it is not synchronized with the latest coding guideline.

@kirkrodrigues kirkrodrigues requested a review from gibber9809 June 17, 2024 18:31
gibber9809
gibber9809 previously approved these changes Jun 21, 2024

@gibber9809 gibber9809 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. How about "ffi: Add Serializer class for new IR stream format." for the title?

Comment thread components/core/src/clp/ffi/ir_stream/Serializer.cpp Outdated
Comment thread components/core/tests/test-ir_encoding_methods.cpp Outdated
Comment thread components/core/tests/test-ir_encoding_methods.cpp Outdated
Comment thread components/core/tests/test-ir_encoding_methods.cpp Outdated
Comment thread components/core/tests/test-ir_encoding_methods.cpp
Comment thread components/core/src/clp/ffi/ir_stream/Serializer.hpp Outdated
Comment thread components/core/src/clp/ffi/ir_stream/Serializer.hpp Outdated
Comment thread components/core/src/clp/ffi/ir_stream/Serializer.hpp Outdated
Comment thread components/core/src/clp/ffi/ir_stream/Serializer.hpp Outdated
Comment thread components/core/src/clp/ffi/ir_stream/Serializer.hpp Outdated
Co-authored-by: kirkrodrigues <2454684+kirkrodrigues@users.noreply.github.com>

@kirkrodrigues kirkrodrigues left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. How about "ffi: Add Serializer class for new IR stream format." for the title?

How about a slight update:

ffi: Add WIP Serializer class for the next IR stream format.

@LinZhihao-723 LinZhihao-723 changed the title ffi: Add support for IR serializer of the next IR stream format. ffi: Add WIP Serializer class for the next IR stream format. Jun 22, 2024
@LinZhihao-723

Copy link
Copy Markdown
Member Author

LGTM. How about "ffi: Add Serializer class for new IR stream format." for the title?

How about a slight update:

ffi: Add WIP Serializer class for the next IR stream format.

lgtm

@LinZhihao-723 LinZhihao-723 merged commit 628ba3d into y-scope:main Jun 22, 2024
jackluo923 pushed a commit to jackluo923/clp that referenced this pull request Dec 4, 2024
…#446)

Co-authored-by: kirkrodrigues <2454684+kirkrodrigues@users.noreply.github.com>
junhaoliao pushed a commit to junhaoliao/clp that referenced this pull request May 17, 2026
…#446)

Co-authored-by: kirkrodrigues <2454684+kirkrodrigues@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants