feat(clp-rust-utils): Add structs to support deserialization of AWS S3 event notifications sent via SQS. by LinZhihao-723 · Pull Request #1537 · y-scope/clp

LinZhihao-723 · 2025-10-31T23:43:02Z

Description

As the title suggests, this PR implements structs for AWS S3 SQS messages following these docs: https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-content-structure.html.

Checklist

The PR satisfies the contribution guidelines.
This is a breaking change and that has been indicated in the PR title, OR this isn't a
breaking change.
Necessary docs have been updated, OR no docs need to be updated.

Validation performed

Ensure all workflows passed.
Tested in the log-ingestor prototype.
Add simple test-cases to parse and deserialize AWS example S3 SQS notification.

Summary by CodeRabbit

New Features
- Added S3 event notification support to parse and handle S3 object events delivered via SQS.
Tests
- Added unit tests validating S3 event deserialization, including successful parsing of object-created events and correct handling of invalid payloads.

coderabbitai · 2025-10-31T23:43:11Z

Walkthrough

Adds an SQS event submodule exposing S3 event types and deserialization logic, introduces strongly typed S3-related structs, re-exports the S3 API, and adds unit tests validating deserialization behaviour.

Changes

Cohort / File(s)	Summary
SQS event entry `components/clp-rust-utils/src/sqs.rs`	Adds `pub mod event` to expose SQS event-related submodule.
Event module re-export `components/clp-rust-utils/src/sqs/event.rs`	Declares `mod s3;` and `pub use s3::*;` to expose S3 event API from the event module.
S3 event types `components/clp-rust-utils/src/sqs/event/s3.rs`	Adds deserializable public types modelling AWS S3 event schema v2.1: `S3` (top-level with `records`), `Record` (maps `eventName` → `event_name`), `Entity`, `Bucket` (`name`), and `Object` (`key`, `size`).
Unit tests `components/clp-rust-utils/tests/sqs_test.rs`	Adds tests: successful deserialization of a PUT event (asserts record count, event_name, bucket name, object key/size) and a test that an invalid payload fails deserialization.

Sequence Diagram

sequenceDiagram
    participant SQS as SQS JSON payload
    participant Serde as serde_json
    participant Types as clp_rust_utils::sqs::event::S3 types
    rect rgb(230, 248, 255)
    SQS->>Serde: deliver JSON
    Serde->>Types: deserialize -> `S3`
    end
    alt valid S3 event
        Types->>Caller: populated `records` → `Record` → `Entity` (`Bucket`, `Object`)
    else invalid/unsupported schema
        Serde->>Caller: returns serde_json::Error
    end

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Inspect serde attribute mappings and field renames in components/clp-rust-utils/src/sqs/event/s3.rs.
Verify test payloads and assertions in components/clp-rust-utils/tests/sqs_test.rs.
Confirm pub use s3::*; in components/clp-rust-utils/src/sqs/event.rs exposes the intended API surface.

Pre-merge checks and finishing touches

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: adding Rust structs to support deserialization of AWS S3 event notifications via SQS. All file changes align with this core objective.

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

LinZhihao-723 · 2025-10-31T23:44:08Z

+#[rustfmt::skip]
+/// Example S3 messages copied from AWS documentation:
+/// <https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-content-structure.html#notification-content-structure-examples>
+const _EXAMPLE_DOC: () = ();


I don't have better ideas of how to make this doc legal to pass the formatter. Let me know if u know better options. @hoophalab

I have no idea, either. Let's just skip rustfmt for now

coderabbitai

Actionable comments posted: 4

📜 Review details

Configuration used: CodeRabbit UI

Review profile: ASSERTIVE

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 5c0a903 and 3dd53e4.

📒 Files selected for processing (4)

components/clp-rust-utils/src/sqs.rs (1 hunks)
components/clp-rust-utils/src/sqs/event.rs (1 hunks)
components/clp-rust-utils/src/sqs/event/s3.rs (1 hunks)
components/clp-rust-utils/tests/sqs_test.rs (1 hunks)

🧰 Additional context used

🧠 Learnings (1)

📓 Common learnings

Learnt from: junhaoliao
Repo: y-scope/clp PR: 0
File: :0-0
Timestamp: 2025-10-22T21:02:31.113Z
Learning: Repository y-scope/clp: Maintain deterministic CI/builds for Rust; add a check to verify Cargo.lock is in sync with Cargo.toml without updating dependencies (non-mutating verification in clp-rust-checks workflow).

Learnt from: LinZhihao-723
Repo: y-scope/clp PR: 549
File: components/core/tests/test-ir_encoding_methods.cpp:1180-1186
Timestamp: 2024-10-01T07:59:11.208Z
Learning: In the context of loop constructs, LinZhihao-723 prefers using `while (true)` loops and does not consider alternative loop constructs necessarily more readable.

Learnt from: LinZhihao-723
Repo: y-scope/clp PR: 549
File: components/core/tests/test-ir_encoding_methods.cpp:1180-1186
Timestamp: 2024-10-08T15:52:50.753Z
Learning: In the context of loop constructs, LinZhihao-723 prefers using `while (true)` loops and does not consider alternative loop constructs necessarily more readable.

🔇 Additional comments (2)

components/clp-rust-utils/src/sqs.rs (1)

2-2: LGTM!

The addition of the public event module is correct and follows Rust module conventions.

components/clp-rust-utils/tests/sqs_test.rs (1)

57-74: LGTM!

The test provides good coverage of S3 event deserialization, validating both successful deserialization with field verification and proper rejection of invalid event formats.

hoophalab

lgtm. one nitpick

Co-authored-by: hoophalab <200652805+hoophalab@users.noreply.github.com>

coderabbitai

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI

Review profile: ASSERTIVE

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between aa18414 and 3ddf6ff.

📒 Files selected for processing (1)

components/clp-rust-utils/tests/sqs_test.rs (1 hunks)

🧰 Additional context used

🧠 Learnings (3)

📓 Common learnings

Learnt from: junhaoliao
Repo: y-scope/clp PR: 0
File: :0-0
Timestamp: 2025-10-22T21:02:31.113Z
Learning: Repository y-scope/clp: Maintain deterministic CI/builds for Rust; add a check to verify Cargo.lock is in sync with Cargo.toml without updating dependencies (non-mutating verification in clp-rust-checks workflow).

Learnt from: LinZhihao-723
Repo: y-scope/clp PR: 549
File: components/core/tests/test-ir_encoding_methods.cpp:1180-1186
Timestamp: 2024-10-01T07:59:11.208Z
Learning: In the context of loop constructs, LinZhihao-723 prefers using `while (true)` loops and does not consider alternative loop constructs necessarily more readable.

Learnt from: LinZhihao-723
Repo: y-scope/clp PR: 549
File: components/core/tests/test-ir_encoding_methods.cpp:1180-1186
Timestamp: 2024-10-08T15:52:50.753Z
Learning: In the context of loop constructs, LinZhihao-723 prefers using `while (true)` loops and does not consider alternative loop constructs necessarily more readable.

📚 Learning: 2024-10-07T21:35:04.362Z

Learnt from: AVMatthews
Repo: y-scope/clp PR: 543
File: components/core/src/clp_s/JsonParser.cpp:735-794
Timestamp: 2024-10-07T21:35:04.362Z
Learning: In `components/core/src/clp_s/JsonParser.cpp`, within the `parse_from_ir` method, encountering errors from `kv_log_event_result.error()` aside from `std::errc::no_message_available` and `std::errc::result_out_of_range` is anticipated behavior and does not require additional error handling or logging.

Applied to files:

components/clp-rust-utils/tests/sqs_test.rs

📚 Learning: 2024-10-08T15:52:50.753Z

Learnt from: AVMatthews
Repo: y-scope/clp PR: 543
File: components/core/src/clp_s/JsonParser.cpp:756-765
Timestamp: 2024-10-08T15:52:50.753Z
Learning: In `components/core/src/clp_s/JsonParser.cpp`, within the `parse_from_ir()` function, reaching the end of log events in a given IR is not considered an error case. The errors `std::errc::no_message_available` and `std::errc::result_out_of_range` are expected signals to break the deserialization loop and proceed accordingly.

Applied to files:

components/clp-rust-utils/tests/sqs_test.rs

🔇 Additional comments (2)

components/clp-rust-utils/tests/sqs_test.rs (2)

1-2: LGTM!

The import correctly references the new S3 type from the SQS event module introduced in this PR.

8-46: LGTM!

The example S3 PUT event payload correctly follows AWS's documented notification content structure and provides comprehensive test data.

…3 event notifications sent via SQS. (y-scope#1537)

Implement S3 event.

3dd53e4

LinZhihao-723 requested a review from a team as a code owner October 31, 2025 23:43

LinZhihao-723 commented Oct 31, 2025

View reviewed changes

LinZhihao-723 requested a review from hoophalab October 31, 2025 23:44

coderabbitai Bot reviewed Oct 31, 2025

View reviewed changes

Comment thread components/clp-rust-utils/src/sqs/event.rs

Comment thread components/clp-rust-utils/src/sqs/event/s3.rs Outdated

Comment thread components/clp-rust-utils/src/sqs/event/s3.rs Outdated

Comment thread components/clp-rust-utils/tests/sqs_test.rs

Apply rabit's suggestion.

aa18414

hoophalab previously approved these changes Nov 3, 2025

View reviewed changes

Comment thread components/clp-rust-utils/tests/sqs_test.rs Outdated

Update components/clp-rust-utils/tests/sqs_test.rs

805b7ba

Co-authored-by: hoophalab <200652805+hoophalab@users.noreply.github.com>

LinZhihao-723 dismissed hoophalab’s stale review via 805b7ba November 3, 2025 18:12

Merge branch 'main' into sqs-event

3ddf6ff

LinZhihao-723 requested a review from hoophalab November 3, 2025 18:13

coderabbitai Bot reviewed Nov 3, 2025

View reviewed changes

Comment thread components/clp-rust-utils/tests/sqs_test.rs

hoophalab approved these changes Nov 3, 2025

View reviewed changes

LinZhihao-723 merged commit 7c95e23 into y-scope:main Nov 3, 2025
23 checks passed

junhaoliao pushed a commit to junhaoliao/clp that referenced this pull request May 17, 2026

feat(clp-rust-utils): Add structs to support deserialization of AWS S…

2006176

…3 event notifications sent via SQS. (y-scope#1537)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(clp-rust-utils): Add structs to support deserialization of AWS S3 event notifications sent via SQS.#1537

feat(clp-rust-utils): Add structs to support deserialization of AWS S3 event notifications sent via SQS.#1537
LinZhihao-723 merged 4 commits into
y-scope:mainfrom
LinZhihao-723:sqs-event

LinZhihao-723 commented Oct 31, 2025 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Oct 31, 2025 •

edited

Loading

Uh oh!

LinZhihao-723 Oct 31, 2025

Uh oh!

hoophalab Nov 3, 2025

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hoophalab left a comment

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

LinZhihao-723 commented Oct 31, 2025 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Validation performed

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram

Estimated code review effort

Pre-merge checks and finishing touches

Uh oh!

LinZhihao-723 Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

hoophalab Nov 3, 2025

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hoophalab left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

LinZhihao-723 commented Oct 31, 2025 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Oct 31, 2025 •

edited

Loading