Skip to content

Implement hex-to-base64 encoding migration for KeyPackage and Welcome#29

Merged
mubarakcoded merged 10 commits intomasterfrom
23-change-from-hex-to-base64-encoding-for-content-fields-on-keypackage-and-welcome-events
Dec 3, 2025
Merged

Implement hex-to-base64 encoding migration for KeyPackage and Welcome#29
mubarakcoded merged 10 commits intomasterfrom
23-change-from-hex-to-base64-encoding-for-content-fields-on-keypackage-and-welcome-events

Conversation

@mubarakcoded
Copy link
Contributor

@mubarakcoded mubarakcoded commented Nov 29, 2025

This implements a 3-phase migration strategy to reduce payload sizes by ~33%:

Phase 1 (Current):

  • Add dual-format reading support (both hex and base64)
  • Keep writing hex by default (use_base64_encoding: false)
  • Backward and forward compatible

Phase 2 (After 3 months):

  • Switch to base64 encoding via config flag
  • Still read both formats during transition

Phase 3 (After 6 months - Optional):

  • Remove hex support entirely

Changes:

  • Add MdkConfig struct with use_base64_encoding flag (default: false)
  • Implement smart format detection based on character set
    • Hex: only [0-9a-fA-F]
    • Base64: [A-Za-z0-9+/=]
  • Add conditional encoding in key package and welcome creation
  • Update builder pattern to support configuration

This ensures a smooth migration that handles the 3-month key package lifetime without breaking compatibility.

Summary by CodeRabbit

  • New Features

    • Builder-style MDK configuration with a toggle for base64 vs hex encoding; runtime-selectable encoding for outgoing payloads and encoding-aware decoding for incoming welcomes/key packages (encoding indicated via tags).
  • Bug Fixes

    • Clearer, encoding-aware decoding errors and persisted failure state when decoding fails.
  • Public API

    • Updated public returns and builder/config exposure to support encoding-tagged payloads and variable-length tags.
  • Tests

    • Expanded tests for cross-format decoding, invalid inputs, padding/edge cases, and size differences.

✏️ Tip: You can customize this high-level summary in your review settings.

@coderabbitai
Copy link

coderabbitai bot commented Nov 29, 2025

Walkthrough

Adds configurable content encoding (hex or base64) via MdkConfig/MdkBuilder and a ContentEncoding enum; introduces encode_content/decode_content utilities; makes KeyPackage and MLS Welcome encoding/runtime-aware (tags indicate base64 when used); updates parsing, decoding, and tests accordingly.

Changes

Cohort / File(s) Summary
Configuration & Builder
crates/mdk-core/src/lib.rs
Add MdkConfig { use_base64_encoding }, MdkBuilder<Storage> and builder APIs; expose MDK::builder(...); store config on MDK; add test helper to construct MDK with custom config.
Encoding utilities & helpers
crates/mdk-core/src/util.rs
Add pub enum ContentEncoding { Hex, Base64 }, as_tag_value, from_tag_value, from_tags; add encode_content(bytes, encoding) and decode_content(content, encoding, label) using BASE64 or hex; provide tag-based detection.
Key package handling & parsing
crates/mdk-core/src/key_packages.rs
create_key_package_for_event now returns (String, Vec<Tag>); select encoding at runtime from config; encode serialized key package with encode_content; add decode_key_package_content and parse_serialized_key_package(..., encoding); parse flow derives encoding from event tags; add logging and tests for hex/base64/invalid/cross-format cases.
Welcome & MLS rumor handling
crates/mdk-core/src/groups.rs, crates/mdk-core/src/welcomes.rs
Use runtime ContentEncoding from MDK::config when producing MLS Welcome rumor; encode welcome with encode_content and include an encoding tag only for base64; add decode_welcome_content and decode paths that respect tags and record failure state with reasons; update parsing to use decoded bytes.
Tests & logging
crates/mdk-core/src/{key_packages.rs,welcomes.rs}, tests/*`
Expand tests to cover base64 and hex encoding paths, invalid inputs, cross-format parsing, size comparisons and ambiguous inputs; add tracing/debug statements around encode/decode operations.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

  • Pay extra attention to:
    • Public API change: create_key_package_for_event return type now includes Vec<Tag>.
    • Correct propagation and use of MDK::config and builder semantics.
    • Consistent encoding tag semantics (tag present only for base64).
    • Correctness and error messages in ContentEncoding::from_tags and decode_content.
    • Tests handling ambiguous/invalid encodings and cross-format parsing.

Possibly related issues

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main objective of the PR—implementing a hex-to-base64 encoding migration for KeyPackage and Welcome events, which aligns with the substantial changes across multiple files.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch 23-change-from-hex-to-base64-encoding-for-content-fields-on-keypackage-and-welcome-events

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between aa182a3 and 30bc334.

📒 Files selected for processing (4)
  • crates/mdk-core/src/groups.rs (2 hunks)
  • crates/mdk-core/src/key_packages.rs (10 hunks)
  • crates/mdk-core/src/util.rs (2 hunks)
  • crates/mdk-core/src/welcomes.rs (5 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • crates/mdk-core/src/groups.rs
🧰 Additional context used
🧬 Code graph analysis (2)
crates/mdk-core/src/welcomes.rs (2)
crates/mdk-core/src/util.rs (3)
  • decode_content (142-156)
  • from_tags (92-104)
  • as_tag_value (62-67)
crates/mdk-core/src/lib.rs (1)
  • create_test_mdk (308-310)
crates/mdk-core/src/key_packages.rs (2)
crates/mdk-core/src/util.rs (4)
  • decode_content (142-156)
  • encode_content (117-122)
  • as_tag_value (62-67)
  • from_tags (92-104)
crates/mdk-core/src/lib.rs (2)
  • create_test_mdk_with_config (313-317)
  • create_test_mdk (308-310)
🔇 Additional comments (10)
crates/mdk-core/src/util.rs (2)

49-105: Well-designed encoding abstraction with sensible defaults.

The ContentEncoding enum cleanly encapsulates the encoding logic with Hex as the default for backward compatibility. The from_tags method correctly defaults to Hex when no encoding tag is present, aligning with the migration strategy.

Minor observation: The from_tag_value method uses case-insensitive matching (to_lowercase()), which is good for robustness, but as_tag_value always returns lowercase. This is consistent behavior.


117-156: LGTM!

The encode_content and decode_content utilities are well-implemented. The tuple return from decode_content with the format description is a nice touch for logging/debugging purposes. Error messages include the label parameter for clear context in error reporting.

crates/mdk-core/src/welcomes.rs (3)

271-302: Clean integration of tag-based encoding detection.

The decode_welcome_content method correctly delegates to decode_content and adds appropriate logging. The doc comment accurately describes the tag-based approach which eliminates ambiguity for strings like deadbeef.


319-376: LGTM!

The preview_welcome method correctly:

  1. Extracts encoding from event tags (defaulting to Hex for legacy events)
  2. Decodes content using the detected encoding
  3. Persists failure information in ProcessedWelcome for debugging
  4. Includes encoding context in error messages

The flow is clean and the error handling is comprehensive.


1203-1232: Excellent test for validating tag-based disambiguation.

The test_decode_welcome_ambiguous_hex_base64 test correctly validates that the same string (deadbeef) decodes to different bytes depending on the specified encoding. This is crucial for proving that the tag-based approach eliminates the ambiguity that would exist with heuristic detection.

crates/mdk-core/src/key_packages.rs (5)

48-104: Well-implemented encoding selection with backward-compatible tag handling.

The implementation correctly:

  1. Returns Vec<Tag> to accommodate the optional encoding tag
  2. Only adds the ["encoding", "base64"] tag when using base64 (hex is the implicit default)
  3. Uses the shared encode_content utility for consistency

This approach minimizes payload overhead for legacy hex encoding while enabling the new base64 format.


106-173: LGTM!

The decode_key_package_content and parse_serialized_key_package methods follow the same pattern as the welcome decoding, maintaining consistency across the codebase. The logging and error handling are appropriate.


217-232: LGTM!

The parse_key_package method correctly extracts the encoding from event tags before parsing, ensuring backward compatibility with legacy hex-encoded events that lack an encoding tag.


2247-2299: Good validation of the size reduction claim.

The test_key_package_size_comparison test validates the PR's claim of ~33% size reduction. The assertion range (25-40%) appropriately accounts for encoding overhead variance. The test also outputs actual sizes for visibility.


2412-2440: Consistent test coverage for ambiguous string handling.

This test mirrors test_decode_welcome_ambiguous_hex_base64 in welcomes.rs, ensuring consistent behavior across both key package and welcome decoding. The duplication is acceptable since these are independent test modules validating the same shared utility behavior.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions
Copy link

❌ Coverage: 86.59% → 86.48% (-0.11%)

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
crates/mdk-core/src/welcomes.rs (1)

290-337: Clarify error message and naming now that content may be base64 or hex

The dual‑format decode_welcome_content logic looks correct and backwards‑compatible, but:

  • Line 347 still formats "Error hex decoding welcome event: {:?}" even when the failure is from the base64 path, which can mislead debugging.
  • Variable name hex_content (line 344) is now inaccurate since it may hold bytes decoded from either hex or base64.

Consider updating both to be format‑agnostic, e.g.:

-        let hex_content = match self.decode_welcome_content(&welcome_event.content) {
+        let content_bytes = match self.decode_welcome_content(&welcome_event.content) {
             Ok(content) => content,
             Err(e) => {
-                let error_string = format!("Error hex decoding welcome event: {:?}", e);
+                let error_string = format!("Error decoding welcome event content: {:?}", e);

This keeps logs accurate while preserving behavior.

Also applies to: 344-364

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 5837e8d and 4ef368b.

📒 Files selected for processing (4)
  • crates/mdk-core/src/groups.rs (2 hunks)
  • crates/mdk-core/src/key_packages.rs (5 hunks)
  • crates/mdk-core/src/lib.rs (4 hunks)
  • crates/mdk-core/src/welcomes.rs (2 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
crates/mdk-core/src/key_packages.rs (2)
crates/mdk-core/src/lib.rs (1)
  • create_test_mdk_with_config (313-317)
crates/mdk-core/src/groups.rs (1)
  • relays (155-158)
🔇 Additional comments (6)
crates/mdk-core/src/groups.rs (1)

20-21: Runtime‑selectable welcome encoding is wired correctly

The welcome rumor encoding cleanly switches between hex and base64 based on self.config.use_base64_encoding, with clear debug logs on each branch, while preserving hex as the default. This aligns with the phased migration goals without changing call sites or types.

Also applies to: 1276-1290, 1293-1299

crates/mdk-core/src/key_packages.rs (2)

984-989: Good separation between “bad encoding” and “bad TLS” in parsing tests

The updated test_invalid_key_package_parsing now distinguishes:

  • Completely invalid encoding ("invalid!@#$%") → Error::KeyPackage.
  • Valid hex with invalid TLS payload ("deadbeef") → Error::Tls(..).

This aligns well with the new decode‑then‑deserialize split and should make debugging upstream issues easier.


2110-2275: Base64/hex encoding tests and size comparison give strong coverage

The new tests around MdkConfig::use_base64_encoding are well‑designed:

  • test_key_package_base64_encoding and test_key_package_hex_encoding_legacy verify that the config flag actually flips the wire format and that both formats round‑trip via parse_serialized_key_package.
  • test_key_package_cross_format_compatibility confirms both “hex MDK” and “base64 MDK” can parse either encoding, which is central to the phased migration story.
  • test_key_package_size_comparison validates the expected ~33% size reduction within a reasonable tolerance band, which is useful as a regression guard.

No issues from a correctness standpoint; these tests nicely lock in the intended behavior.

crates/mdk-core/src/lib.rs (3)

42-65: MdkConfig is minimal and appropriate for the current migration

The MdkConfig struct and its helpers (new, with_base64) are straightforward, default to the legacy hex behavior, and clearly document the base64 tradeoff. This is a clean way to carry encoding policy without over‑designing the config surface.


67-156: Builder wiring is correct; be aware of public‑struct field expansion

MdkBuilder correctly:

  • Defaults to MdkConfig::default() and exposes with_base64_encoding and with_config to customize behavior.
  • Constructs MDK with DEFAULT_CIPHERSUITE, SUPPORTED_EXTENSIONS, a RustCrypto provider, and the chosen config.
  • Is surfaced via MDK::builder, with MDK::new delegating to builder(storage).build() to preserve the hex‑by‑default behavior while encouraging the builder for custom configs.

One compatibility note: adding pub config: MdkConfig to the public MDK struct (lines 168–180) is a source‑breaking change for any external code constructing MDK via a struct literal. If the crate aims to preserve that usage, consider:

  • Making config non‑public (e.g., pub(crate) with accessor if needed), or
  • Accepting the semver break and bumping the public version accordingly.

Functionally the implementation looks good; this is mostly about external API expectations.

Also applies to: 168-180, 222-260


300-317: Test helper for custom config is a nice addition

create_test_mdk_with_config cleanly wraps the builder and is already used by the new key‑package tests to drive base64 vs hex behavior. This should make future config‑dependent tests much easier to write.

@github-actions
Copy link

❌ Coverage: 86.59% → 86.5% (-0.09%)

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (1)
crates/mdk-core/src/key_packages.rs (1)

100-153: LGTM!

The decode function correctly implements hex-first detection with fallthrough to base64 when hex decode fails. This addresses the robustness concern from the previous review about allowing base64 fallback.

🧹 Nitpick comments (4)
crates/mdk-core/src/welcomes.rs (4)

290-301: Doc comment is misplaced and merged with preview_welcome's doc.

Lines 290-300 describe decode_welcome_content but are incorrectly attached to the preview_welcome doc block above. The preview_welcome error documentation at line 290 abruptly transitions into describing decode_welcome_content.

 /// # Errors
 ///
-/// Returns an error if both hex and base64 decoding fail.
-///
-/// Decodes welcome content from either base64 or hex encoding.
-///
-/// Detects the format based on character set:
-/// - Hex uses only: 0-9, a-f, A-F
-/// - Base64 uses: A-Z, a-z, 0-9, +, /, =
-///
-/// If the string contains only hex characters, it attempts hex decoding first (legacy format).
-/// If hex decoding fails or if the string contains non-hex characters, it attempts base64 decoding.
-/// This provides maximum robustness against edge cases.
+/// Returns an error if the welcome message cannot be parsed or processed.
+
+    /// Decodes welcome content from either base64 or hex encoding.
+    ///
+    /// Detects the format based on character set:
+    /// - Hex uses only: 0-9, a-f, A-F
+    /// - Base64 uses: A-Z, a-z, 0-9, +, /, =
+    ///
+    /// If the string contains only hex characters, it attempts hex decoding first (legacy format).
+    /// If hex decoding fails or if the string contains non-hex characters, it attempts base64 decoding.
+    /// This provides maximum robustness against edge cases.
+    ///
+    /// # Errors
+    ///
+    /// Returns an error if both hex and base64 decoding fail.
     fn decode_welcome_content(&self, content: &str) -> Result<Vec<u8>, Error> {

344-347: Variable name and error message don't reflect dual-format decoding.

hex_content is misleading since the bytes may have been decoded from base64. Similarly, the error message at line 347 says "Error hex decoding" but decode_welcome_content tries both formats.

-        let hex_content = match self.decode_welcome_content(&welcome_event.content) {
+        let decoded_content = match self.decode_welcome_content(&welcome_event.content) {
             Ok(content) => content,
             Err(e) => {
-                let error_string = format!("Error hex decoding welcome event: {:?}", e);
+                let error_string = format!("Error decoding welcome event content: {:?}", e);

Also update the usage at line 366:

-        let welcome_preview = match self.parse_serialized_welcome(&hex_content) {
+        let welcome_preview = match self.parse_serialized_welcome(&decoded_content) {

307-336: Decode logic is sound; error message could be more informative.

The dual-format detection and fallthrough logic is correct. However, the error at line 333 only includes the base64 error. When hex-only content fails both decodes, including the hex error would help debugging.

     fn decode_welcome_content(&self, content: &str) -> Result<Vec<u8>, Error> {
         let is_hex_only = content.chars().all(|c| c.is_ascii_hexdigit());
+        let mut hex_error: Option<String> = None;

         if is_hex_only {
             if let Ok(bytes) = hex::decode(content) {
                 tracing::debug!(
                     target: "mdk_core::welcomes",
                     "Decoded welcome using hex (legacy format)"
                 );
                 return Ok(bytes);
             }
+            hex_error = Some("invalid hex".to_string());
             tracing::debug!(
                 target: "mdk_core::welcomes",
                 "Hex decode failed for hex-only string, attempting base64"
             );
         }

         match BASE64.decode(content) {
             Ok(bytes) => {
                 tracing::debug!(
                     target: "mdk_core::welcomes",
                     "Decoded welcome using base64 (new format)"
                 );
                 Ok(bytes)
             }
-            Err(e) => Err(Error::Welcome(format!(
-                "Failed to decode welcome as both hex and base64: {}",
-                e
-            ))),
+            Err(e) => {
+                let msg = match hex_error {
+                    Some(hex_err) => format!(
+                        "Failed to decode welcome: hex error: {}, base64 error: {}",
+                        hex_err, e
+                    ),
+                    None => format!("Failed to decode welcome as base64: {}", e),
+                };
+                Err(Error::Welcome(msg))
+            }
         }
     }

450-454: Tests correctly verify default hex behavior; consider adding base64 decode test.

The existing tests verify hex output which is correct for the default use_base64_encoding: false. However, there's no test validating that decode_welcome_content can successfully decode base64-encoded welcome content.

Consider adding a test similar to test_key_package_cross_format_compatibility in key_packages.rs to verify welcomes can be decoded from both formats.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 4ef368b and cb940aa.

📒 Files selected for processing (2)
  • crates/mdk-core/src/key_packages.rs (6 hunks)
  • crates/mdk-core/src/welcomes.rs (2 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
crates/mdk-core/src/key_packages.rs (2)
crates/mdk-core/src/lib.rs (1)
  • create_test_mdk_with_config (313-317)
crates/mdk-core/src/groups.rs (1)
  • relays (155-158)
🔇 Additional comments (8)
crates/mdk-core/src/welcomes.rs (1)

6-7: LGTM!

Standard base64 imports aligned with the key_packages.rs implementation.

crates/mdk-core/src/key_packages.rs (7)

4-5: LGTM!

Base64 imports consistent with welcomes.rs.


24-34: LGTM!

Documentation correctly describes the dual-format encoding controlled by MdkConfig::use_base64_encoding. This addresses the previous review feedback.


73-97: LGTM!

Clean conditional encoding with appropriate debug logging for traceability.


155-184: LGTM!

Clean separation of concerns - decoding is now delegated to the helper function while this method focuses on TLS deserialization and validation.


986-996: LGTM!

Test correctly validates that invalid encoding (neither hex nor base64) returns Error::KeyPackage. The test also keeps the existing hex-valid-but-invalid-MLS test case.


2112-2223: LGTM!

Excellent test coverage for the encoding feature:

  • Base64 and hex encoding tests verify correct format selection
  • Cross-format compatibility ensures both old and new clients can interoperate
  • This is critical for the 3-phase migration strategy

2225-2277: LGTM!

Good validation of the PR's primary objective. The 25-40% savings range appropriately accounts for base64 padding overhead while confirming the expected ~33% reduction.

@github-actions
Copy link

❌ Coverage: 86.59% → 86.55% (-0.04%)

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (2)
crates/mdk-core/src/welcomes.rs (1)

339-364: Error message in preview_welcome is now slightly misleading re: encoding

preview_welcome now calls decode_welcome_content, but the error string is still:

let error_string = format!("Error hex decoding welcome event: {:?}", e);

Given that the helper may decode via hex or base64, consider changing this to a neutral phrasing like "Error decoding welcome event" (and maybe drop “hex” from the log target message too) so logs remain accurate regardless of the encoding used.

crates/mdk-core/src/key_packages.rs (1)

2280-2333: Decode helper tests are good; consider aligning comments/name with actual behavior

The three tests for decode_key_package_content validate:

  • Non‑hex/base64 ("!!!") → combined hex+base64 error.
  • Hex‑only invalid ("abc") → hex attempt then base64 attempt, with error message confirming both were tried.
  • Valid hex ("00000000") → straightforward success.

Functionally this is solid and exercises both the base64‑direct path and the hex‑first path (including fallback attempt). Minor nit: test_decode_fallback_path and its comment still talk about “fallback path” where hex fails then base64 succeeds, but the body now only covers the simple “valid hex” success path. You might want to either rename the test or tweak the comment to reflect that it’s specifically checking the hex‑success branch.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between cb940aa and b56ea6a.

📒 Files selected for processing (2)
  • crates/mdk-core/src/key_packages.rs (6 hunks)
  • crates/mdk-core/src/welcomes.rs (3 hunks)
🧰 Additional context used
🧬 Code graph analysis (1)
crates/mdk-core/src/welcomes.rs (3)
crates/mdk-core/src/lib.rs (1)
  • create_test_mdk (308-310)
crates/mdk-core/src/encrypted_media/manager.rs (1)
  • create_test_mdk (351-353)
crates/mdk-core/src/encrypted_media/crypto.rs (1)
  • create_test_mdk (204-206)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: Test Coverage
🔇 Additional comments (6)
crates/mdk-core/src/welcomes.rs (2)

290-337: Dual-format welcome decoding helper is sound and aligns with key package behavior

The decode_welcome_content helper (including its docs) correctly:

  • Detects “hex‑only” via is_ascii_hexdigit.
  • Prefers hex decoding for legacy welcomes and falls back to base64 on hex failure.
  • Uses base64 for any content with non‑hex chars.
  • Maps the final failure into a single, clear Error::Welcome("Failed to decode welcome as both hex and base64: ...").

This matches the migration plan and the key‑package decoding strategy, giving good backward compatibility with minimal ambiguity. I don’t see functional issues here.


1105-1156: Welcome decode tests cover invalid and success paths appropriately

The three tests for decode_welcome_content exercise:

  • Non‑hex, non‑base64 input ("!!!") → combined error.
  • Hex‑only but structurally invalid ("abc") → hex attempt then base64 attempt, with error mentioning both formats.
  • Valid hex ("00000000") → successful decode.

This gives good coverage of the main edge cases for the helper. No changes needed.

crates/mdk-core/src/key_packages.rs (4)

24-28: Config-driven key package encoding looks correct and well-documented

The updated doc comment and encoding branch in create_key_package_for_event correctly:

  • Describe the content as “encoded” (hex or base64) and tie the choice to MdkConfig::use_base64_encoding.
  • Use base64 when the flag is true and hex otherwise, with tracing distinguishing “new” vs “legacy” formats.

This matches the migration plan and keeps the external API stable (still returns a String + tags tuple). Implementation and docs are consistent; no issues here.

Also applies to: 71-87, 97-97


100-153: Key package decode helper and parser provide robust dual-format handling

The new decode_key_package_content and updated parse_serialized_key_package:

  • Use the same hex-only detection heuristic as welcomes and try hex first for legacy strings, then base64 on hex failure.
  • For non‑hex strings, go straight to base64.
  • Map ultimate decode failures into Error::KeyPackage("Failed to decode key package as both hex and base64: ...").
  • Keep TLS deserialization failures surfaced as Error::Tls(..) from the subsequent tls_deserialize.

This cleanly separates “cannot decode at all” vs “decoded but not a valid TLS KeyPackage”, which is exactly what the tests expect. There is a very low‑probability corner case where a base64 string composed only of hex characters will be misinterpreted as hex (and then likely fail TLS), but given how unlikely that is and the desire to prefer legacy hex, this tradeoff seems acceptable.

Also applies to: 155-176


986-995: Invalid-encoding test now correctly targets the decode helper path

The first half of test_invalid_key_package_parsing now verifies that a content string that is neither valid base64 nor valid hex ("invalid!@#$%") yields an Error::KeyPackage, while a valid-hex but non‑KeyPackage string ("deadbeef") still yields Error::Tls(..).

This neatly validates the new error taxonomy (decode vs TLS failure). Looks good.


2112-2277: Base64/hex mode, cross-format compatibility, and size tests are thorough

The new tests:

  • test_key_package_base64_encoding / test_key_package_hex_encoding_legacy assert that:
    • The config flag switches the on‑wire encoding.
    • Each mode can parse its own output.
  • test_key_package_cross_format_compatibility ensures both hex- and base64-configured MDKs can parse both formats, which is key for the migration window.
  • test_key_package_size_comparison measures hex vs base64 lengths and asserts a 25–40% size reduction, which is a reasonable band and unlikely to be brittle across minor serialization changes.

Collectively these give strong confidence that the config flag works as intended and that mixed ecosystems will interoperate during the rollout.

@github-actions
Copy link

⚠️ Coverage: 86.59% → 86.59% (0.00%)

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
crates/mdk-core/src/welcomes.rs (2)

272-301: Doc comments are now attached to the wrong function

The block starting at Line 272 (“Previews a welcome message…”) currently attaches to decode_welcome_content, since it’s the next item after the docs. As a result:

  • decode_welcome_content is documented as if it were the preview API.
  • preview_welcome (Line 339) has no doc comment anymore.

Consider splitting and reordering the docs so each function has an accurate header, e.g.:

  • Keep the “Previews a welcome message…” block immediately above fn preview_welcome.
  • Add a focused doc block for decode_welcome_content describing the dual-format decoding and hex-first heuristic.

This keeps the public documentation consistent with the actual behavior.


344-363: Update error message and naming now that decoding is hex/base64, not hex-only

preview_welcome now uses decode_welcome_content, which may decode either hex or base64, but:

  • The variable is still called hex_content, even when base64 is used.
  • The error string says Error hex decoding welcome event, which is misleading for base64 failures and for the dual-format behavior.

Consider:

-        let hex_content = match self.decode_welcome_content(&welcome_event.content) {
+        let decoded_content = match self.decode_welcome_content(&welcome_event.content) {
@@
-                let error_string = format!("Error hex decoding welcome event: {:?}", e);
+                let error_string =
+                    format!("Error decoding welcome event content (hex/base64): {:?}", e);
@@
-        let welcome_preview = match self.parse_serialized_welcome(&hex_content) {
+        let welcome_preview = match self.parse_serialized_welcome(&decoded_content) {

This keeps logs and failure_reason accurate as the encoding migrates.

🧹 Nitpick comments (3)
crates/mdk-core/src/welcomes.rs (1)

1105-1188: Align test naming/comments with actual code paths in decode_welcome_content

The new tests give good coverage of invalid inputs, hex-only odd-length inputs, valid hex, and base64 with padding. One small mismatch:

  • test_decode_welcome_fallback_path uses "00000000", which decodes successfully as hex and never exercises the hex-fail→base64 “fallback” path described in the comment.

You can either:

  • Rename/comment it as a straightforward “valid hex decodes” test, or
  • Change the input to a hex-only-but-invalid string that base64 can decode, if you specifically want to exercise the fallback behavior.

Also, the suite duplicates the structure of the key package decode tests; if you find yourself tweaking the heuristic later, consider extracting a small shared helper for the dual-format decode tests to keep them in sync.

crates/mdk-core/src/key_packages.rs (2)

2112-2277: Config-mode, cross-format, and size comparison tests provide strong migration coverage

The new tests around base64/hex modes look good:

  • test_key_package_base64_encoding verifies that with use_base64_encoding: true, the content is valid base64 and round-trips through parse_serialized_key_package.
  • test_key_package_hex_encoding_legacy confirms the legacy hex path still works when the flag is false.
  • test_key_package_cross_format_compatibility shows that both hex- and base64-configured MDK instances can parse both encodings, which is exactly what you want during migration.
  • test_key_package_size_comparison demonstrates the expected ~25–40% size savings from base64 vs hex at the string level.

One tiny note: the “not valid hex” assertion in test_key_package_base64_encoding depends on the particular serialized key package instance; in practice that’s fine (the probability of a random base64 string of this length being valid hex is astronomically low), but the comment already hints at the nuance. No change strictly required.


2280-2365: Decode tests mirror welcomes; adjust fallback test naming and consider deduping

The decode tests here parallel those in welcomes.rs and give good coverage of:

  • Non-hex/non-base64 input ("!!!"),
  • Hex-only but invalid strings ("abc"),
  • Valid hex ("00000000"),
  • Base64 with non-hex chars and padding.

As with the welcomes tests, test_decode_fallback_path uses "00000000", which succeeds via the hex path and never exercises the “hex fails then base64” fallback described in the comment. Either:

  • Rename/comment it as a “valid hex decodes” test, or
  • Adjust the input if you genuinely want to hit the fallback branch.

Given the duplication between key package and welcome decoding tests, you might also consider a small shared helper or parametrized test pattern to keep these suites in lockstep when the heuristic changes.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between b56ea6a and 4447e44.

📒 Files selected for processing (2)
  • crates/mdk-core/src/key_packages.rs (6 hunks)
  • crates/mdk-core/src/welcomes.rs (3 hunks)
🧰 Additional context used
🧬 Code graph analysis (2)
crates/mdk-core/src/key_packages.rs (3)
crates/mdk-core/src/lib.rs (2)
  • create_test_mdk_with_config (313-317)
  • create_test_mdk (308-310)
crates/mdk-core/src/encrypted_media/manager.rs (1)
  • create_test_mdk (351-353)
crates/mdk-core/src/encrypted_media/crypto.rs (1)
  • create_test_mdk (204-206)
crates/mdk-core/src/welcomes.rs (3)
crates/mdk-core/src/lib.rs (1)
  • create_test_mdk (308-310)
crates/mdk-core/src/encrypted_media/manager.rs (1)
  • create_test_mdk (351-353)
crates/mdk-core/src/encrypted_media/crypto.rs (1)
  • create_test_mdk (204-206)
🔇 Additional comments (3)
crates/mdk-core/src/welcomes.rs (1)

6-7: Base64 imports line up with new decoding helper

The added nostr::base64 imports match the usage in decode_welcome_content and keep all base64 operations localized here. No issues.

crates/mdk-core/src/key_packages.rs (2)

4-5: Config-driven encoding for key packages looks correct and well-documented

Using self.config.use_base64_encoding to switch between:

  • hex (legacy, default) and
  • base64 (new, smaller) via BASE64.encode

matches the PR’s migration plan, and the updated doc comment now correctly describes that the content may be hex or base64. The added tracing around the chosen encoding format should also make rollout debugging easier.

Also applies to: 24-35, 73-87, 97-97


986-996: Invalid parsing test cleanly distinguishes encoding vs TLS failures

The updated test_invalid_key_package_parsing now:

  • Uses a clearly invalid string ("invalid!@#$%") to assert that encoding errors map to Error::KeyPackage.
  • Uses "deadbeef" to assert that valid-hex-but-invalid-TLS input surfaces as Error::Tls(..).

This makes the boundary between “decode failed” and “TLS parse failed” explicit and future-proof.

@github-actions
Copy link

✅ Coverage: 86.59% → 86.62% (+0.03%)

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (4)
crates/mdk-core/src/util.rs (1)

50-62: Dual-format decoding logic is correct; consider enriching hex error detail

is_hex_only plus decode_dual_format give the desired behavior: hex-only inputs try hex first and fall back to base64, and non-hex inputs go straight to base64 with accurate messages about which formats were attempted. One optional improvement would be to capture and include the hex decode error in the final message for the hex_only case to aid debugging without changing the control flow.

Also applies to: 64-110

crates/mdk-core/src/welcomes.rs (2)

271-299: Doc block now mismatches decode_welcome_content and drops preview_welcome docs

The /// block starting at Line 271 currently describes previewing a welcome (staged welcome, group data, arguments, etc.) but is attached to decode_welcome_content (the next item) instead of preview_welcome. As a result:

  • preview_welcome has no doc comment.
  • decode_welcome_content has a description that doesn't match its signature or behavior.

It would be clearer to restore the preview-specific docs to sit directly above fn preview_welcome and give decode_welcome_content a focused, shorter doc explaining the dual-format decoding and error semantics.


311-336: Update error message/variable name to reflect dual-format decoding

preview_welcome now decodes via decode_welcome_content, but:

  • The variable is still named hex_content.
  • The error string uses "Error hex decoding welcome event".

This is slightly misleading now that base64 is also supported. Consider renaming the variable to something like decoded_content and changing the message to "Error decoding welcome event" (or similar) so logs and failure_reason remain accurate regardless of encoding.

crates/mdk-core/src/key_packages.rs (1)

2086-2197: Encoding-mode and cross-format tests look solid; one assertion is theoretically flaky

The new tests:

  • Verify base64 encoding when use_base64_encoding = true and hex encoding when false.
  • Confirm both hex- and base64-configured MDK instances can parse both encodings.

One minor point: test_key_package_base64_encoding asserts that hex::decode(&key_package_str).is_err(). While in practice it's astronomically unlikely that a base64 string for a real key package would consist only of hex characters, this does introduce a tiny theoretical flakiness. If you want to be completely deterministic, you could drop that assertion and rely on the positive checks (valid base64 + successful round-trip parsing) to validate behavior.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 4447e44 and a1c78a4.

📒 Files selected for processing (3)
  • crates/mdk-core/src/key_packages.rs (6 hunks)
  • crates/mdk-core/src/util.rs (2 hunks)
  • crates/mdk-core/src/welcomes.rs (3 hunks)
🧰 Additional context used
🧬 Code graph analysis (2)
crates/mdk-core/src/welcomes.rs (2)
crates/mdk-core/src/util.rs (1)
  • decode_dual_format (82-110)
crates/mdk-core/src/lib.rs (1)
  • create_test_mdk (308-310)
crates/mdk-core/src/key_packages.rs (2)
crates/mdk-core/src/util.rs (1)
  • decode_dual_format (82-110)
crates/mdk-core/src/lib.rs (2)
  • create_test_mdk_with_config (313-317)
  • create_test_mdk (308-310)
🔇 Additional comments (10)
crates/mdk-core/src/util.rs (1)

2-3: Base64 engine import/alias is appropriate for shared decoding

Using nostr::base64::engine::general_purpose::STANDARD as BASE64 centralizes the base64 engine choice and keeps callers consistent. No issues here.

crates/mdk-core/src/welcomes.rs (3)

13-14: Import of decode_dual_format cleanly centralizes welcome decoding

Pulling in decode_dual_format from util keeps welcome decoding aligned with key package behavior and avoids duplication. No issues.


300-309: decode_welcome_content correctly wraps decode_dual_format with logging

The helper cleanly delegates to decode_dual_format("welcome"), maps the String into Error::Welcome, and logs which format was used. The behavior matches the tests and the migration plan.


1077-1160: New tests give good coverage of dual-format welcome decoding

The added tests exercise:

  • Non-hex/non-base64 inputs (base64-only attempt).
  • Hex-only invalid inputs (hex attempt then base64 fallback with combined error wording).
  • Valid hex path.
  • Valid base64 (including padding) where hex is skipped.

This provides solid coverage for the new helper and should catch regressions in the detection/fallback behavior.

crates/mdk-core/src/key_packages.rs (6)

4-6: Imports correctly align key package handling with shared dual-format utilities

Bringing in the base64 engine alias and decode_dual_format (alongside NostrTagFormat) keeps key package encoding/decoding consistent with welcomes and the shared util helper. No concerns here.

Also applies to: 16-17


24-28: Config-driven encoding in create_key_package_for_event matches docs and migration plan

The updated docs clearly describe hex vs base64 based on MdkConfig::use_base64_encoding, and the implementation honors that with a single encoded_content branch plus debug logging for each path. This aligns with the 3‑phase migration strategy and keeps the public behavior well-documented.

Also applies to: 34-35, 71-87, 97-98


100-127: Dual-format key package decoding is correct and error taxonomy is clean

decode_key_package_content correctly:

  • Delegates to decode_dual_format("key package") to handle hex-first, then base64 fallback.
  • Logs which format was used.
  • Maps any failure into Error::KeyPackage, while parse_serialized_key_package keeps TLS deserialization failures as Error::Tls.

This separation of concerns and error variants is clear and matches the tests’ expectations.

Also applies to: 129-152


960-970: Invalid-encoding parsing test correctly targets Error::KeyPackage path

The new check for parse_serialized_key_package("invalid!@#$%") ensuring it yields Err(Error::KeyPackage(_)) nicely verifies that non-hex/non-base64 inputs fail in the decode layer and never reach TLS parsing. This is the intended behavior.


2200-2251: Size-comparison test correctly demonstrates expected base64 savings

The size comparison between hex- and base64-encoded key packages, with a 25–40% savings window and a printed percentage, is a good sanity check that the migration yields the expected payload reduction without overfitting to an exact ratio.


2253-2339: Decode helper tests thoroughly cover dual-format edge cases

The new tests for decode_key_package_content mirror the welcome tests and exercise:

  • Non-hex/non-base64 input.
  • Hex-only invalid strings with both hex and base64 attempts.
  • Valid hex-only path.
  • Base64 inputs (with and without padding) that bypass hex.

These give strong confidence that the helper behaves as intended across edge cases and that error messages match expectations.

@github-actions
Copy link

✅ Coverage: 86.59% → 86.62% (+0.03%)

… events

This implements a 3-phase migration strategy to reduce payload sizes by ~33%:

Phase 1 (Current):
- Add dual-format reading support (both hex and base64)
- Keep writing hex by default (use_base64_encoding: false)
- Backward and forward compatible

Phase 2 (After 3 months):
- Switch to base64 encoding via config flag
- Still read both formats during transition

Phase 3 (After 6 months - Optional):
- Remove hex support entirely

Changes:
- Add MdkConfig struct with use_base64_encoding flag (default: false)
- Implement smart format detection based on character set
  - Hex: only [0-9a-fA-F]
  - Base64: [A-Za-z0-9+/=]
- Add conditional encoding in key package and welcome creation
- Update builder pattern to support configuration

This ensures a smooth migration that handles the 3-month key package
lifetime without breaking compatibility.
- Update documentation to mention MdkConfig::use_base64_encoding flag
- Add fallback logic: if hex decode fails, try base64 before giving up
- Fix clippy warnings about using map over inspect
- Improve error messages to indicate both formats were tried

This makes the decoder more robust against edge cases while maintaining
backward compatibility.
Adds 6 new tests to improve code coverage:
- test_decode_invalid_hex_string: Tests non-hex invalid input
- test_decode_hex_only_invalid: Tests hex-only but invalid input with fallback
- test_decode_fallback_path: Tests valid hex decoding

These tests cover the new fallback logic where hex decode failure
triggers base64 attempt, addressing the coverage regression.
Adds 4 more tests to increase code coverage:
- test_decode_base64_with_special_chars: Tests base64 with non-hex chars
- test_decode_base64_with_padding: Tests multiple base64 strings with padding

These tests provide comprehensive coverage of the base64 decode path,
ensuring all code branches are exercised.

Changes:
- Removed failing tests that attempted impossible scenarios
- Added practical base64 test cases with various padding scenarios
- Extract shared decode_dual_format helper to util.rs
- Remove duplicated code from key_packages.rs and welcomes.rs
- Fix error messages to accurately reflect which formats were attempted
- Update tests to match new error message format
- Move doc comments to correct functions (decode_welcome_content vs preview_welcome)
- Rename hex_content to decoded_content for accuracy
- Update error message from 'hex decoding' to 'hex/base64'
- Rename test_decode_*_fallback_path to test_decode_*_valid_hex for accuracy
@mubarakcoded mubarakcoded force-pushed the 23-change-from-hex-to-base64-encoding-for-content-fields-on-keypackage-and-welcome-events branch from 2cc6b91 to d580910 Compare November 30, 2025 00:12
@github-actions
Copy link

✅ Coverage: 86.59% → 86.62% (+0.03%)

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
crates/mdk-core/src/key_packages.rs (1)

2274-2293: Test may silently pass if base64 decoding succeeds.

Same issue as in welcomes.rs - the if let Err(err) pattern means if "abc" decodes successfully as base64, the test passes without any assertions.

Apply the same fix as suggested for test_decode_welcome_hex_only_invalid:

-        if let Err(err) = result {
-            let err_msg = err.to_string();
-            assert!(
-                err_msg.contains("attempted hex and base64"),
-                "Error should indicate both formats were tried for hex-only string, got: {}",
-                err_msg
-            );
-        }
+        let err = result.expect_err("Odd-length hex-only string should fail both hex and base64 decoding");
+        let err_msg = err.to_string();
+        assert!(
+            err_msg.contains("attempted hex and base64"),
+            "Error should indicate both formats were tried for hex-only string, got: {}",
+            err_msg
+        );
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between a1c78a4 and d580910.

📒 Files selected for processing (5)
  • crates/mdk-core/src/groups.rs (2 hunks)
  • crates/mdk-core/src/key_packages.rs (6 hunks)
  • crates/mdk-core/src/lib.rs (4 hunks)
  • crates/mdk-core/src/util.rs (2 hunks)
  • crates/mdk-core/src/welcomes.rs (5 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
  • crates/mdk-core/src/groups.rs
  • crates/mdk-core/src/util.rs
🧰 Additional context used
🧬 Code graph analysis (2)
crates/mdk-core/src/welcomes.rs (4)
crates/mdk-core/src/util.rs (1)
  • decode_dual_format (82-110)
crates/mdk-core/src/lib.rs (1)
  • create_test_mdk (308-310)
crates/mdk-core/src/encrypted_media/manager.rs (1)
  • create_test_mdk (351-353)
crates/mdk-core/src/encrypted_media/crypto.rs (1)
  • create_test_mdk (204-206)
crates/mdk-core/src/key_packages.rs (2)
crates/mdk-core/src/util.rs (1)
  • decode_dual_format (82-110)
crates/mdk-core/src/lib.rs (2)
  • create_test_mdk_with_config (313-317)
  • create_test_mdk (308-310)
🔇 Additional comments (16)
crates/mdk-core/src/welcomes.rs (4)

13-13: LGTM!

Import of shared decode_dual_format utility aligns with the DRY principle, avoiding duplication of the detection and decoding logic across modules.


271-297: LGTM!

Clean implementation that properly delegates to the shared decode_dual_format utility while maintaining module-specific logging and error types.


314-342: LGTM!

The integration of dual-format decoding into preview_welcome is well-executed. Error messages appropriately mention both hex and base64, and the failure path correctly records the decoding error in the ProcessedWelcome state.


1123-1165: Good test coverage for dual-format decoding.

Tests cover the key scenarios: valid hex decoding, base64 with non-hex characters (triggering direct base64 path), and base64 with various padding patterns.

crates/mdk-core/src/lib.rs (4)

42-65: Clean configuration struct with sensible defaults.

MdkConfig is well-documented with clear default behavior (hex encoding for backward compatibility). The with_base64() constructor provides a convenient way to enable the new format.


67-156: Well-implemented builder pattern.

The MdkBuilder provides a clean fluent API with good documentation. The examples in doc comments demonstrate both the default and configured usage paths clearly.


178-179: Backward-compatible API evolution.

Adding the config field and builder() method while keeping MDK::new() functional ensures existing code continues to work. The delegation from new() to builder().build() maintains a single construction path.

Also applies to: 222-260


311-317: LGTM!

The new test helper enables testing configuration-dependent behavior while maintaining the existing create_test_mdk() for tests that don't need custom configuration.

crates/mdk-core/src/key_packages.rs (8)

4-5: LGTM!

Imports correctly bring in the base64 encoding/decoding capabilities and the shared decode_dual_format utility.

Also applies to: 16-16


22-48: Doc comments now accurately reflect dual-format support.

The updated documentation correctly explains that encoding format depends on MdkConfig::use_base64_encoding, addressing the previous review feedback.


73-97: LGTM!

The encoding logic correctly branches based on configuration, with helpful debug logging to track which encoding path is used.


100-127: LGTM!

Clean implementation that properly uses the shared decode_dual_format utility, addressing the previous review suggestion to reduce duplication between key package and welcome decoding.


129-158: LGTM!

The updated method cleanly separates decoding concerns (now handled by decode_key_package_content) from TLS deserialization and validation, improving code organization.


2086-2197: Excellent test coverage for encoding configuration and cross-format compatibility.

These tests thoroughly validate:

  • Base64 encoding when configured
  • Hex encoding (legacy) as default
  • Cross-format parsing (both MDK instances can read both formats)

This provides strong confidence in the migration path.


2199-2251: Good size comparison test validating the ~33% savings claim.

The test explicitly calculates and verifies the size reduction, with reasonable tolerance (25-40%) accounting for encoding overhead variations.


2295-2336: LGTM!

The remaining decode tests cover valid hex, base64 with special characters, and base64 with various padding patterns comprehensively.

@github-actions
Copy link

✅ Coverage: 86.59% → 86.64% (+0.05%)

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (2)
crates/mdk-core/src/welcomes.rs (2)

271-297: Dual‑format decoding helper is correct; consider making it stateless/free for reuse

decode_welcome_content cleanly delegates to decode_dual_format, preserves the label, and logs the chosen format; behavior matches the migration design (hex‑first for hex‑only, else base64). Since the method doesn’t use self, you could optionally make it an associated or free helper (e.g., fn decode_welcome_content(content: &str)) to signal that it’s pure and make it easier to reuse from other modules without needing an MDK instance.


319-364: Error handling correctly persists failure state, but nested error formatting and retry semantics deserve a second look

The decode/parse error branches correctly:

  • Mark ProcessedWelcome as Failed with a failure_reason.
  • Persist that state.
  • Log via tracing::error!.
  • Return an Error::Welcome with the same message.

Two minor points to verify/improve:

  1. Nested error message formatting
    You format the inner Error with {:?} and then wrap it again in Error::Welcome:

    let error_string = format!("Error decoding welcome event content (hex/base64): {:?}", e);
    // ...
    return Err(Error::Welcome(error_string));

    Because e is already an Error, this can yield messages like
    Welcome("Error decoding ...: Welcome(\"Failed to decode welcome ...\")").
    Using {e} (Display) or propagating the inner message directly would make logs and stored failure_reasons cleaner and more user‑facing:

    let error_string = format!("Error decoding welcome event content (hex/base64): {}", e);

    The same applies to the parse branch ("Error previewing welcome: {:?}").

  2. Non‑retryable state for decode/parse failures
    Because preview_welcome writes a ProcessedWelcome with state Failed and is_welcome_processed only checks for existence, any subsequent process_welcome calls for the same wrapper_event_id will treat the welcome as already processed and will not retry decoding/parsing. That’s probably what you want for permanently malformed content, but it does change behavior from “pure preview” to “preview with side effects” and makes decode/parse failures non‑retryable.

    If that’s intentional, it might be worth a short comment near is_welcome_processed or the failure branches; if not, you may want to distinguish “failed” vs “processed” in the is_welcome_processed check (e.g., only treating Processed as terminal).

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between d580910 and f5f162e.

📒 Files selected for processing (2)
  • crates/mdk-core/src/key_packages.rs (6 hunks)
  • crates/mdk-core/src/welcomes.rs (5 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • crates/mdk-core/src/key_packages.rs
🧰 Additional context used
🧬 Code graph analysis (1)
crates/mdk-core/src/welcomes.rs (2)
crates/mdk-core/src/util.rs (1)
  • decode_dual_format (82-110)
crates/mdk-core/src/lib.rs (1)
  • create_test_mdk (308-310)
🔇 Additional comments (1)
crates/mdk-core/src/welcomes.rs (1)

1081-1163: New decode_welcome_ tests exercise key hex/base64 edge cases well*

The new tests cover:

  • Non‑hex/non‑base64 input ("!!!") and assert we report a base64‑decode failure.
  • Hex‑only but hex‑invalid input ("abc") and assert the combined “attempted hex and base64” error.
  • Happy paths for valid hex and base64 (including padding variants).

This gives good confidence that decode_dual_format’s detection order and error messages are wired correctly through decode_welcome_content. Also, the hex‑only‑invalid test now unconditionally asserts an error before inspecting the message, addressing the earlier concern about potentially silent passes.

Copy link
Contributor

@dannym-arx dannym-arx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great to me!

My only comment is whether it's even possible to correctly detect if something is hex or base64 with 100% accuracy

c0ffeebabe is valid hex, but also valid base64 of course.

perhaps a version tag makes more sense (no tag = hex, v1 = base64)? but I'll leave this as an open question

Implements version-tagged dual-format encoding to support migration from hex
to base64 encoding. This eliminates ambiguity where strings like "deadbeef"
are valid in both formats but decode to completely different bytes.
@github-actions
Copy link

✅ Coverage: 86.59% → 86.68% (+0.09%)

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (2)
crates/mdk-core/src/groups.rs (1)

28-29: Welcome encoding uses config correctly; consider hoisting encoding out of the loop

The switch to encode_content(&serialized_welcome, self.config.use_base64_encoding) plus the debug log is correct and aligns with the migration plan.

You could optionally compute encoded_welcome once before the for event in key_package_events loop (and log once) since serialized_welcome and the chosen format don’t vary per event, avoiding redundant encoding and log noise.

Also applies to: 1275-1294

crates/mdk-core/src/key_packages.rs (1)

951-965: Key package decode tests line up well with the new dual-format utilities

The updated tests for parsing and decoding key package content cover the important paths:

  • test_invalid_key_package_parsing now correctly expects Error::KeyPackage for non-hex/non-v1 strings and Error::Tls for hex that deserializes badly.
  • The new base64 tests (test_key_package_base64_encoding, cross-format compatibility, size comparison) verify:
    • v1: prefixing and valid base64 payloads.
    • That both hex- and base64-configured MDK instances can parse both formats.
    • Expected size savings for base64 vs hex.
  • The decode_key_package_content tests mirror the welcome decode tests for invalid hex, valid hex, v1 base64 (valid and invalid), and ambiguous strings like "deadbeef".

All of this gives strong confidence that the migration will behave as intended for key packages, and that decode semantics match decode_dual_format. Any remaining duplication between key-package and welcome decode tests is minor and can be refactored later if it becomes noisy.

Also applies to: 2081-2396

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between f5f162e and a418fb3.

📒 Files selected for processing (4)
  • crates/mdk-core/src/groups.rs (2 hunks)
  • crates/mdk-core/src/key_packages.rs (6 hunks)
  • crates/mdk-core/src/util.rs (2 hunks)
  • crates/mdk-core/src/welcomes.rs (5 hunks)
🧰 Additional context used
🧬 Code graph analysis (3)
crates/mdk-core/src/welcomes.rs (2)
crates/mdk-core/src/util.rs (1)
  • decode_dual_format (86-103)
crates/mdk-core/src/lib.rs (1)
  • create_test_mdk (308-310)
crates/mdk-core/src/groups.rs (1)
crates/mdk-core/src/util.rs (1)
  • encode_content (60-66)
crates/mdk-core/src/key_packages.rs (2)
crates/mdk-core/src/util.rs (2)
  • decode_dual_format (86-103)
  • encode_content (60-66)
crates/mdk-core/src/lib.rs (2)
  • create_test_mdk_with_config (313-317)
  • create_test_mdk (308-310)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: Test Coverage
🔇 Additional comments (2)
crates/mdk-core/src/util.rs (1)

50-103: Dual-format helpers look correct and symmetric

encode_content and decode_dual_format cleanly implement the versioned "v1:" base64 vs legacy hex split, with clear error messages and a format descriptor for logging. No functional issues spotted here.

crates/mdk-core/src/welcomes.rs (1)

319-365: Error handling in preview_welcome is now robust and stateful

The new preview_welcome flow correctly:

  • Uses decode_welcome_content for dual-format decoding.
  • On decode or parse failure, creates a ProcessedWelcome with Failed state and failure_reason, persists it, logs the error, and returns Error::Welcome.

This makes processing idempotent and gives callers a clear failure trail without changing the happy path.

Update documentation in key_packages.rs and welcomes.rs to accurately
describe the version-tagged dual-format encoding implementation.
@github-actions
Copy link

✅ Coverage: 86.59% → 86.68% (+0.09%)

@github-actions
Copy link

github-actions bot commented Dec 1, 2025

✅ Coverage: 86.59% → 86.65% (+0.06%)

Replace the v1: content prefix approach with Nostr-native tag-based
encoding for KeyPackage and Welcome events.

Changes:
- Add ["encoding", "base64"] tag when using base64 encoding
- Default to hex when no encoding tag present (backward compatible)
- Update ContentEncoding enum with from_tags() method
- Update all decode methods to use explicit encoding parameter
- Update tests for tag-based approach

The tag-based approach:
- Is more Nostr-native (uses standard tag conventions)
- Is backward compatible (unknown tags are ignored by old clients)
- Eliminates ambiguity without modifying content
@mubarakcoded mubarakcoded force-pushed the 23-change-from-hex-to-base64-encoding-for-content-fields-on-keypackage-and-welcome-events branch from aa182a3 to 30bc334 Compare December 1, 2025 17:27
@github-actions
Copy link

github-actions bot commented Dec 1, 2025

✅ Coverage: 86.59% → 86.65% (+0.06%)

Copy link
Member

@erskingardner erskingardner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One small thought but definitely don't let that stop you.

/// This function generates an encoded key package that is used as the content field of a kind:443 Nostr event.
/// The encoding format is determined by `MdkConfig::use_base64_encoding`:
/// - When `false` (default): uses hex encoding (legacy format, no encoding tag)
/// - When `true`: uses base64 encoding with `["encoding", "base64"]` tag (~33% smaller)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry if you guys have already hashed this out but do we really need the tag? I know detecting base64 vs hex isn't perfect but it seems like we should be able to do it reliably enough.

I don't care that much - we can always remove the tag later once we've cycled out all the hex key packages.

Copy link
Contributor

@dannym-arx dannym-arx Dec 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it depends on the length of the message. If the message is long enough, we can be pretty much 100% sure whether it's hex or base64 (certainty → 100% as length → ∞). But for shorter messages, we have to rely on tricks (like if there's an odd number of characters, it's definitely base64).

for every char there's a 34.4% chance that it's hex and not base64, at 32 chars that's a very low chance, but not zero (on the range of 1e-15).

personally i feel like the tag approach is the best solution if we want to be sure to not break anything, but at the same time, the chances are low enough where it doesn't affect it that much

Copy link
Contributor

@dannym-arx dannym-arx Dec 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$$P_{\text{ambiguous}}(\text{msglen}) = \left(\frac{22}{64}\right)^{\text{msglen}} \approx 0.344^\text{msglen}$$

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

k, cool. let's keep the tag.

@mubarakcoded mubarakcoded merged commit 94792c1 into master Dec 3, 2025
11 checks passed
Copy link
Contributor

@jgmontoya jgmontoya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments, let's try to perhaps address them on a future PR

Comment on lines +1298 to +1301
// Add encoding tag only for base64 (hex is the default, no tag needed)
if encoding == ContentEncoding::Base64 {
tags.push(Tag::custom(TagKind::Custom("encoding".into()), ["base64"]));
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should always push the encoding tag, even when using hex, just for standardness reasons

/// This function generates a hex-encoded key package that is used as the content field of a kind:443 Nostr event.
/// This function generates an encoded key package that is used as the content field of a kind:443 Nostr event.
/// The encoding format is determined by `MdkConfig::use_base64_encoding`:
/// - When `false` (default): uses hex encoding (legacy format, no encoding tag)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same idea here. The spec is now saying we should include the encoding tag, so we should.

public_key: &PublicKey,
relays: I,
) -> Result<(String, [Tag; 6]), Error>
) -> Result<(String, Vec<Tag>), Error>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to always include the tag and keep this length fixed

Comment on lines +98 to +101
// Add encoding tag only for base64 (hex is the default, no tag needed)
if encoding == ContentEncoding::Base64 {
tags.push(Tag::custom(TagKind::Custom("encoding".into()), ["base64"]));
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here

Comment on lines +123 to +137
fn decode_key_package_content(
&self,
content: &str,
encoding: ContentEncoding,
) -> Result<Vec<u8>, Error> {
let (bytes, format) =
decode_content(content, encoding, "key package").map_err(Error::KeyPackage)?;

tracing::debug!(
target: "mdk_core::key_packages",
"Decoded key package using {}", format
);

Ok(bytes)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method seems a bit unnecessary to me, why not call decode_content directly at the calling point?

"Should NOT have encoding tag when using hex (default)"
);

// Verify we can parse it back with hex encoding
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's try to remove all the comments that don't really add anything or don't clarify anything that isn't already evident by looking at the code

Comment on lines +2247 to +2299
#[test]
fn test_key_package_size_comparison() {
let test_pubkey =
PublicKey::from_hex("884704bd421671e01c13f854d2ce23ce2a5bfe9562f4f297ad2bc921ba30c3a6")
.unwrap();
let relays = vec![RelayUrl::parse("wss://relay.example.com").unwrap()];

// Create with hex
let hex_config = crate::MdkConfig {
use_base64_encoding: false,
};
let hex_mdk = crate::tests::create_test_mdk_with_config(hex_config);

let (hex_key_package, _) = hex_mdk
.create_key_package_for_event(&test_pubkey, relays.clone())
.expect("Failed to create hex key package");

// Create with base64
let base64_config = crate::MdkConfig {
use_base64_encoding: true,
};
let base64_mdk = crate::tests::create_test_mdk_with_config(base64_config);

let (base64_key_package, _) = base64_mdk
.create_key_package_for_event(&test_pubkey, relays)
.expect("Failed to create base64 key package");

let hex_size = hex_key_package.len();
let base64_size = base64_key_package.len();

// Base64 should be smaller than hex
assert!(
base64_size < hex_size,
"Base64 ({} bytes) should be smaller than hex ({} bytes)",
base64_size,
hex_size
);

// Calculate the savings
let savings_percent = ((hex_size - base64_size) as f64 / hex_size as f64) * 100.0;
println!(
"Size comparison: hex={} bytes, base64={} bytes, savings={:.1}%",
hex_size, base64_size, savings_percent
);

// Base64 should be approximately 33% smaller (hex is 2x, base64 is 1.33x)
// Allow some variance due to encoding overhead
assert!(
savings_percent > 25.0 && savings_percent < 40.0,
"Expected savings between 25-40%, got {:.1}%",
savings_percent
);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this test is really useful. This is basically testing a property of the encodings themselves and not our code. I'd just remove it entirely

Comment on lines +2340 to +2349
#[test]
fn test_decode_valid_hex() {
let mdk = create_test_mdk();

// Test that valid hex strings decode successfully
let valid_hex = "00000000";
let result = mdk.decode_key_package_content(valid_hex, ContentEncoding::Hex);
assert!(result.is_ok(), "Valid hex should decode successfully");
assert_eq!(result.unwrap(), vec![0, 0, 0, 0]);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test seems a bit redundant given we have the test_key_package_hex_encoding_legacy test

Comment on lines +2382 to +2393
#[test]
fn test_decode_base64() {
let mdk = create_test_mdk();

// Test base64 format
let base64_str = "SGVsbG8="; // "Hello" in base64
let result = mdk.decode_key_package_content(base64_str, ContentEncoding::Base64);

assert!(result.is_ok(), "Should decode valid base64");
let decoded = result.unwrap();
assert_eq!(decoded, b"Hello");
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, it seems like this was already tested no?

Comment on lines +288 to +295
fn decode_welcome_content(
&self,
content: &str,
encoding: ContentEncoding,
) -> Result<Vec<u8>, Error> {
let (bytes, format) =
decode_content(content, encoding, "welcome").map_err(Error::Welcome)?;

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we could just call the decode_content directly instead of having this additional method which doesn't really seem to be doing a lot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Change from hex to base64 encoding for content fields on KeyPackage and Welcome events

4 participants