Skip to content

Omaha crate improvements#88

Merged
dongsupark merged 31 commits intoflatcar:mainfrom
james-parky:dev-omaha-improvements
Sep 30, 2025
Merged

Omaha crate improvements#88
dongsupark merged 31 commits intoflatcar:mainfrom
james-parky:dev-omaha-improvements

Conversation

@james-parky
Copy link
Copy Markdown
Contributor

@james-parky james-parky commented Sep 12, 2025

General Improvements to the omaha Crate

The following is a slightly ammended collection of the commit bodies from the constituent commits of this PR. They serve not only as a description of all changes, but also (if read in written order) may serve to explain the justification behind the changes.

Hashing Algorithms

  • Rename HashAlgo trait to Hasher since that is what structs implementing it do.
  • Rework of core Hash struct and Hasher trait. Rather than provide Hasher (prev. HashAlgo) impl structs to users that can then use them in conjunction with the sha1/sha2 etc. hasher that they are meant to represent, allow the structs implementing Hasher (Sha1 and Sha2) to perform hashing logic themselves. Therefore add three new trait methods: new(), update(), and finalize(). Other methods can be added later if required, but for now this is sufficient.
  • Remove Hasher::from_boxed(). There is no reason for it to exist if you encapsulate the hashing logic (update and finalize) into the trait itself.
  • Add associated trait method Hasher::try_from_hex_string() for use in the pre-existing sha256_hex mod (for use in hard_xml reading). Note I rename this module to sha256_from_str since that it literally all it is for, and the word "hex" isn't used consistently across the module.
  • Add new sha1_from_str module that mimics the sha256 one. Move this and the other one to hash_types.rs to be defined together with the types they use.
  • Rather than having a Hash struct that has some methods, just define two type aliases Sha1Digest and Sha256Digest when the output of these hashing algorithms is needed (where you could otherwise use H::Output where H is a struct implementing Hasher.
  • Declare that Hasher::Output must implement PartialEq so that we can compare hash outputs (and therefor compare the above mentioned ShaXXXDigest types).
  • Simplify download.rs:hash_on_disk() with the fact that the trait implementations encapsulate their own hashing logic.
  • Temporarily switch to printing all instances of a Hasher::Output to a Debug print.
  • Introduce associated const FINGERPRINT_SIZE to Hasher trait and comment about that there is a better way to potentially represent the output type of these structs in the future if a particular RFC stabilises.
  • Doc strings for contained structs, traits and functions.
  • Add various unit tests.

Requests and Responses

  • Remove #[rustfmt::skip]s in response.rs. rustfmt almost always produces better formatting than doing it manually.
  • Introduce preliminary custom Error and Result types -- planning on removing reliance on anyhow crate just for contexts. We can create custom error strings using a crate-specific Error enum. This also removes the need for the ct-codecs dependency.
  • Remove now unused dependencies anyhow and digest.
  • Remove Uuid type and provide unit tests for xml read/write. This was documenting as existing to provide serialisation of the underlying inner value of type uuid::Uuid with surrounding braces. Whenever uuids are used throughout the codebase I image they are used just as uuid::Uuids, therefore we don't need a type just for (de)seriailsation purposes -- just provide a with = attribute for XmlRead and XmlWrite impls.
  • Also provide here preliminary unit tests for reading/writing some omaha structs using the base uuid::Uuid type to prove this is sufficient.
  • Introduce util function for creating XmlError values.
  • Reduce code by making better use of derive(XmlRead). There were some structs that had custom implementation of XmlRead when it could have been derived, so derive instead. Since we cannot derive or manually implement XmlRead for url::Url, we must create a newtype Urls(Vec<Url>) and implement XmlRead for that (based off the original code).
  • Write unit tests for XmlRead-ing new Urls struct.
  • Write some doc strings and remove now redundant comment.
  • Write unit tests for XmlRead-ing Package struct with a generic helper function. This also requires going through and deriving PartialEq on the response field structs. This is done behind a cfg_attr(test, ) though since there is currently no business-logic requirement for it to be derived fully.
  • Re-order struct defs in a top-down way to make generated docs read better when they are written.

Other

  • Remove type.rs and therefore the FileSize struct. It doesn't appear to have a need to exist; using a FileSize in places that could just be a usize doesn't provide any extra type-safety, and doesn't really provide any extra semantic meaning to a dev. Wherever it was used it was just being converted back and forth from a usize anyway. Furthermore, there is no special (de)serialisation logic that is required for it -- it is also just a usize in the XML.

Notes

  • Further additions/changes e.g. docstrings and more structs in the request.rs and response.rs files will require conversation with someone more familiar with ue, but happy to do the work.

@james-parky james-parky requested a review from a team as a code owner September 12, 2025 20:20
@james-parky james-parky force-pushed the dev-omaha-improvements branch from 6e9da5d to fd49c6b Compare September 12, 2025 20:24
@james-parky james-parky changed the title WIP: Omaha crate improvements Omaha crate improvements Sep 12, 2025
Copy link
Copy Markdown
Member

@dongsupark dongsupark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First of all, thank you for huge effort you put into this PR.

I like the approach in general. I am still in middle of reading, but I wrote several comments below. Will read the rest tomorrow.

The thing is, if a PR has over 10 commits that have own different purposes, it becomes hard for reviewers to look into each commit, and follow what you want to do.

In general, I would recommend creating multiple PRs that have own purposes. For example one PR for trivial changes(renaming variables/functions, reordering, cleaning up), another PR for dealing with improving xml parts, and another PR for rewriting hasher structure like the beginning of this PR. Then you could get the easiest part merged immediately, and after that rebase harder PRs on top of that, and go through review cycles etc. That would be really helpful for reviewers.

@james-parky
Copy link
Copy Markdown
Contributor Author

The thing is, if a PR has over 10 commits that have own different purposes, it becomes hard for reviewers to look into each commit, and follow what you want to do.

Yes fair enough. I know that this PR seems almost random and I suppose it kind of is. I was just changing things as I thought of them, without small initial goals for refactoring -- so things ended up order of a more logical order. Will keep to smaller PRs in the future though.

Since that is what structs implementing it do. This will make more sense
with further commits.

Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
- Rather than provide Hashers (prev. HashAlgo) impl structs to users that
  can then use them in conjunction with the sha1/sha2 etc. hasher that they
  are meant to represent, allow the structs implementing Hasher (Sha1 and
  Sha2) to perform hashing logic themselves. Therefore add three new trait
  methods: new(), update(), and finalize().

- Remove Hasher::from_boxed(). There is no reason for it to exist if you
  encapsulate the hashing logic (update and finalize) into the trait
  itself.

- Add associated trait method Hasher::try_from_hex_string() for use in the
  pre-existing sha256_hex mod (for use in hard_xml Reading). Note I rename
  this module to sha256_from_str since that it literally all it is for,
  and the word "hex" isn't used consistently across the module.

- Add new sha1_from_str module that mimics the sha256 one.

- Rather than having a Hash struct that has some methods, just define two
  type aliases Sha1Digest and Sha256Digest when the output of these
  hashing algorithms is needed (where you could otherwise use H::Output
  where H is a struct implementing Hasher.

- Declare that Hasher::Output must implement PartialEq so that we can
  compare hash outputs (and therefor compare the above mentioned
  ShaXXXDigest types).

- Simplify download.rs:hash_on_disk() with the fact that the trait
  implementations encapsulate their own hashing logic.

- Temporarily switch to printing all instances of a Hasher::Output to a
  Debug print.

Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
And comment about that there is a better way to potentially do this in the
future if a particular RFC stabilises.

Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
rustfmt almost always produces better formatting than doing it manually.

Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
Planning on removing reliance on anyhow crate just for contexts. We can
create custom error strings using a crate-specific Error enum.

Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
anyhow and digest

Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
This was documenting as existing to provide serialisation of the underlying
inner value of type uuid::Uuid with surrounding braces. Whenever uuids are
used throughout the codebase I image they are used just as uuid::Uuids,
therefore we don't need a type just for (de)seriailsation purposes -- just
provide a `with = ` attribute for XmlRead and XmlWrite.

Also provide here preliminary unit tests for reading/writing some omaha
structs using the base uuid::Uuid type to prove this is sufficient.

Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
To be logically close to the types they are using.

Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
There were some structs that had custom implementation of XmlRead when it
could have been derived, so derive instead. Since we cannot derive or
manually implement XmlRead for url::Url, we must create a newtype
Urls(Vec<Url>) and implement XmlRead for that (using the original code).

Note that I am fairly sure that this doesn't change behaviour but I am yet
to write unit tests.

Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
This also requires going through and deriving PartialEq on the response
field structs. This is done behind a cfg_attr(test, ) though since there is
currently no business-logic requirement for it to be derived fully.

Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
In a top-down way to make generated docs read better when they are written.

Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
It doesn't appear to have a need to exist; using a FileSize in places that
could just be a usize doesn't provide any extra type-safety, and doesn't
really provide any extra semantic meaning to a dev. Wherever it was used it
was just being converted back and forth from a usize anyway. Furthermore,
there is no special (de)serialisation logic that is required for it -- it
is also just a usize in the XML.

Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
james-parky added a commit to james-parky/ue-rs that referenced this pull request Sep 18, 2025
PR flatcar#87 created the download crate which caused merge conflicts with flatcar#88.
@james-parky james-parky force-pushed the dev-omaha-improvements branch from fd49c6b to 7d2e60e Compare September 18, 2025 20:18
PR flatcar#87 created the download crate which caused merge conflicts with flatcar#88.

Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
@james-parky james-parky force-pushed the dev-omaha-improvements branch from 7d2e60e to 2f69107 Compare September 18, 2025 20:19
@james-parky
Copy link
Copy Markdown
Contributor Author

Just rebased this onto new main branch after #87 was merged.

Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
Copy link
Copy Markdown
Member

@dongsupark dongsupark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our CI fails at clippy warnings by default. Can you please address them?

error: using `clone` on type `Option<[u8; 32]>` which implements the `Copy` trait
   --> src/download/package.rs:104:13
    |
104 |             self.hash_sha256.clone(),
    |             ^^^^^^^^^^^^^^^^^^^^^^^^ help: try removing the `clone` call: `self.hash_sha256`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#clone_on_copy
    = note: `-D clippy::clone-on-copy` implied by `-D warnings`
    = help: to override `-D warnings` add `#[allow(clippy::clone_on_copy)]`
error: using `clone` on type `Option<[u8; 20]>` which implements the `Copy` trait
   --> src/download/package.rs:105:13
    |
105 |             self.hash_sha1.clone(),
    |             ^^^^^^^^^^^^^^^^^^^^^^ help: try removing the `clone` call: `self.hash_sha1`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#clone_on_copy
error: using `clone` on type `[u8; 32]` which implements the `Copy` trait
   --> src/download/package.rs:122:70
    |
122 |         debug!("    sha256 match?      {}", self.hash_sha256 == Some(calculated_sha256.clone()));
    |                                                                      ^^^^^^^^^^^^^^^^^^^^^^^^^ help: try removing the `clone` call: `calculated_sha256`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#clone_on_copy
error: using `clone` on type `[u8; 20]` which implements the `Copy` trait
   --> src/download/package.rs:125:66
    |
125 |         debug!("    sha1 match?      {}", self.hash_sha1 == Some(calculated_sha1.clone()));
    |                                                                  ^^^^^^^^^^^^^^^^^^^^^^^ help: try removing the `clone` call: `calculated_sha1`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#clone_on_copy
error: using `clone` on type `[u8; 32]` which implements the `Copy` trait
   --> src/download/package.rs:127:67
    |
127 | ... self.hash_sha256 != Some(calculated_sha256.clone()) || self.hash_sha1.is_some() && self.hash_sha1 != Some(calculated_sha1.clone()) {
    |                              ^^^^^^^^^^^^^^^^^^^^^^^^^ help: try removing the `clone` call: `calculated_sha256`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#clone_on_copy
error: using `clone` on type `[u8; 20]` which implements the `Copy` trait
   --> src/download/package.rs:127:148
    |
127 | ...) && self.hash_sha1 != Some(calculated_sha1.clone()) {
    |                                ^^^^^^^^^^^^^^^^^^^^^^^ help: try removing the `clone` call: `calculated_sha1`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#clone_on_copy
error: using `clone` on type `[u8; 32]` which implements the `Copy` trait
   --> src/download/package.rs:155:34
    |
155 |         let hdhashvec: Vec<u8> = hdhash.clone().into();
    |                                  ^^^^^^^^^^^^^^ help: try removing the `clone` call: `hdhash`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#clone_on_copy
error: using `clone` on type `[u8; 32]` which implements the `Copy` trait
   --> src/download/mod.rs:117:65
    |
117 |     debug!("    sha256 match?      {}", expected_sha256 == Some(calculated_sha256.clone()));
    |                                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^ help: try removing the `clone` call: `calculated_sha256`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#clone_on_copy
error: using `clone` on type `[u8; 20]` which implements the `Copy` trait
   --> src/download/mod.rs:120:61
    |
120 |     debug!("    sha1 match?      {}", expected_sha1 == Some(calculated_sha1.clone()));
    |                                                             ^^^^^^^^^^^^^^^^^^^^^^^ help: try removing the `clone` call: `calculated_sha1`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#clone_on_copy
error: using `clone` on type `[u8; 32]` which implements the `Copy` trait
   --> src/download/mod.rs:122:61
    |
122 |     if expected_sha256.is_some() && expected_sha256 != Some(calculated_sha256.clone()) {
    |                                                             ^^^^^^^^^^^^^^^^^^^^^^^^^ help: try removing the `clone` call: `calculated_sha256`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#clone_on_copy
error: using `clone` on type `[u8; 20]` which implements the `Copy` trait
   --> src/download/mod.rs:125:57
    |
125 |     if expected_sha1.is_some() && expected_sha1 != Some(calculated_sha1.clone()) {
    |                                                         ^^^^^^^^^^^^^^^^^^^^^^^ help: try removing the `clone` call: `calculated_sha1`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#clone_on_copy
error: using `clone` on type `Option<[u8; 32]>` which implements the `Copy` trait
   --> src/download/mod.rs:142:60
    |
142 |         || do_download_and_hash(client, url.clone(), path, expected_sha256.clone(), expected_sha1.clone()),
    |                                                            ^^^^^^^^^^^^^^^^^^^^^^^ help: try removing the `clone` call: `expected_sha256`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#clone_on_copy
error: using `clone` on type `Option<[u8; 20]>` which implements the `Copy` trait
   --> src/download/mod.rs:142:85
    |
142 |         || do_download_and_hash(client, url.clone(), path, expected_sha256.clone(), expected_sha1.clone()),
    |                                                                                     ^^^^^^^^^^^^^^^^^^^^^ help: try removing the `clone` call: `expected_sha1`
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#clone_on_copy

dongsupark added a commit to flatcar/scripts that referenced this pull request Sep 26, 2025
dongsupark added a commit to flatcar/scripts that referenced this pull request Sep 26, 2025
dongsupark added a commit to flatcar/scripts that referenced this pull request Sep 26, 2025
dongsupark added a commit to flatcar/scripts that referenced this pull request Sep 26, 2025
dongsupark added a commit to flatcar/scripts that referenced this pull request Sep 27, 2025
dongsupark added a commit to flatcar/scripts that referenced this pull request Sep 27, 2025
Copy link
Copy Markdown
Member

@dongsupark dongsupark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Thanks.
CI of flatcar/scripts#3310 passed, which means no regression with this PR.
Once you address clippy issues, I will merge it.

@james-parky
Copy link
Copy Markdown
Contributor Author

Once you address clippy issues, I will merge it.

Yes I will do that later today, thanks

Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
@james-parky
Copy link
Copy Markdown
Contributor Author

@dongsupark fixed clippy lint warnings

@dongsupark dongsupark merged commit 5d4db08 into flatcar:main Sep 30, 2025
1 check passed
dongsupark added a commit to flatcar/scripts that referenced this pull request Sep 30, 2025
cnd4 pushed a commit to cnd4/flatcar-scripts that referenced this pull request Oct 3, 2025
dongsupark pushed a commit that referenced this pull request Dec 10, 2025
PR #87 created the download crate which caused merge conflicts with #88.

Signed-off-by: james-parky <72101291+james-parky@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants