Skip to content

Remove a bunch of try_into().expect()#4200

Merged
apoelstra merged 2 commits intorust-bitcoin:masterfrom
Kixunil:remove_try_into_expect
Mar 21, 2025
Merged

Remove a bunch of try_into().expect()#4200
apoelstra merged 2 commits intorust-bitcoin:masterfrom
Kixunil:remove_try_into_expect

Conversation

@Kixunil
Copy link
Copy Markdown
Collaborator

@Kixunil Kixunil commented Mar 5, 2025

Previously we've used try_into().expect() because const generics wer
unavailable. Then they become available but we didn't realize we could
already convert a bunch of code to not use panicking conversions. But we
can (and could for a while).

This adds an extension trait for arrays to provide basic non-panicking
operations retutning arrays, so they can be composed with other
functions accepting arrays without any conversions. It also refactors a
bunch of code to use the non-panicking constructs but it's certainly not
all of it. That could be done later. This just aims at removing the
ugliest offenders and demonstrate the usefulness of this approach.

@Kixunil Kixunil added the Correctness Improves the chance that the code will be correct - either our or consumer (via well-designed API) label Mar 5, 2025
@github-actions github-actions bot added C-bitcoin PRs modifying the bitcoin crate C-hashes PRs modifying the hashes crate C-internals PRs modifying the internals crate C-primitives C-base58 PRs modifying the base58 crate labels Mar 5, 2025
@coveralls
Copy link
Copy Markdown

coveralls commented Mar 5, 2025

Pull Request Test Coverage Report for Build 13977619809

Details

  • 161 of 189 (85.19%) changed or added relevant lines in 13 files are covered.
  • 4 unchanged lines in 2 files lost coverage.
  • Overall coverage increased (+0.06%) to 83.173%

Changes Missing Coverage Covered Lines Changed/Added Lines %
bitcoin/src/crypto/key.rs 10 11 90.91%
bitcoin/src/bip32.rs 90 92 97.83%
bitcoin/src/crypto/taproot.rs 5 10 50.0%
internals/src/array.rs 20 26 76.92%
internals/src/slice.rs 4 18 22.22%
Files with Coverage Reduction New Missed Lines %
bitcoin/src/taproot/mod.rs 1 77.27%
bitcoin/src/internal_macros.rs 3 61.24%
Totals Coverage Status
Change from base Build 13977478725: 0.06%
Covered Lines: 22030
Relevant Lines: 26487

💛 - Coveralls

@apoelstra
Copy link
Copy Markdown
Member

oooo. Concept ACK.

@Kixunil
Copy link
Copy Markdown
Collaborator Author

Kixunil commented Mar 5, 2025

Forgot to say the warning is intentional for now - I think I'll fix that bug first before merging this. (Or anyone else can tonight, I don't think I'll do it today.)

@tcharding
Copy link
Copy Markdown
Member

Nice. Here are two more tests I wrote while understanding the extension trait.

    #[test]
    fn five_to_two() {
        let slice = [1, 2, 3, 4, 5];
        let (left, right) = slice.as_chunks::<2>();
        assert_eq!(left, &[[1, 2], [3, 4]]);
        assert_eq!(right, &[5]);

        let (first, rest) = slice.split_first_chunk::<2>().unwrap();
        assert_eq!(first, &[1, 2]);
        assert_eq!(rest, &[3, 4, 5]);

        let (rest, last) = slice.split_last_chunk::<2>().unwrap();
        assert_eq!(rest, &[1, 2, 3]);
        assert_eq!(last, &[4, 5]);
    }

    #[test]
    fn get_array() {
        let slice = [1, 2, 3, 4, 5];

        let chunk = slice.get_array::<2>(0).unwrap();
        assert_eq!(chunk, &[1, 2]);

        // offset is an index of `T` not an index of chunks.
        let chunk = slice.get_array::<2>(1).unwrap();
        assert_eq!(chunk, &[2, 3]);

        let chunk = slice.get_array::<2>(3).unwrap();
        assert_eq!(chunk, &[4, 5]);

        assert!(slice.get_array::<2>(4).is_none());
    }

At first I thought offset was a chunk index not an index of Ts.

@jirijakes
Copy link
Copy Markdown
Contributor

Nice!

Pity rust-analyzer does not report error:

image

@Kixunil
Copy link
Copy Markdown
Collaborator Author

Kixunil commented Mar 6, 2025

@jirijakes Yeah, I've seen someone mentioned that postmonomorphization errors only show up with cargo build, not cargo check and presumable RA uses check (very reasonable) and postmono error is the only feasible option right now (that is still compile time).

@jirijakes
Copy link
Copy Markdown
Contributor

@Kixunil, you're right. There is a related issue: rust-lang/rust#99682.

So, cargo check will not report these Hacky errors, only cargo build will. Rust-analyzer can be configured to use build if needed but I don't think it makes much sense.

@Kixunil Kixunil force-pushed the remove_try_into_expect branch from e7c3070 to 1b22f7d Compare March 15, 2025 14:40
@github-actions github-actions bot removed the C-hashes PRs modifying the hashes crate label Mar 15, 2025
@Kixunil Kixunil force-pushed the remove_try_into_expect branch 5 times, most recently from e3ed73d to 6bd2749 Compare March 15, 2025 16:25
@Kixunil Kixunil marked this pull request as ready for review March 15, 2025 16:53
tcharding
tcharding previously approved these changes Mar 17, 2025
Copy link
Copy Markdown
Member

@tcharding tcharding left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 6bd2749

@Kixunil
Copy link
Copy Markdown
Collaborator Author

Kixunil commented Mar 17, 2025

@jirijakes FTR I've seen a proposal to make check also evaluate these consts which would be much cleaner solution but something that requires work on the compiler side.

@Kixunil
Copy link
Copy Markdown
Collaborator Author

Kixunil commented Mar 17, 2025

@apoelstra do you want me to redo my PR or do it in a followup to save Tobin's ACK?

@apoelstra
Copy link
Copy Markdown
Member

@Kixunil can you force-push the PR? I think it'll be OK -- I should be able to ACK it and Tobin should be online before I sign off so I can start merging it.

@Kixunil
Copy link
Copy Markdown
Collaborator Author

Kixunil commented Mar 17, 2025

I need to run now (some unexpected stuff came up) but I think I'll be able to do it by the end of the day.


/// Tries to access a sub-array of length `ARRAY_LEN` at the specified `offset`.
///
/// Returns `None` in case of out-of-bounds access.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not return an Error type with useful information instead of just None? Although that can be done later not needed with this PR.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because these are low-level primitives which don't have much information about the error, so ok_or is likely better but also this one is currently only used to implement APIs consistent with std which return Option so any such information would be a waste.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. So besides mirroring std, since there is only one specific error then it's not informative to return why it error-ed it sounds like. Thanks, just looking to understand better the rational for when to return None and when to return an Error.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One is in iterator - trying to go past the end means end of iteration. The other is get method which is similar enough to get on slices to warrant matching API. std has Option there not Result. While returning indices in error types is somewhat convenient in erroneous cases using get is usually an operation that's handled by the caller.

@Kixunil Kixunil force-pushed the remove_try_into_expect branch 2 times, most recently from 6dc30d2 to 474977f Compare March 17, 2025 18:09
(left, right)
}

fn get_array<const ARRAY_LEN: usize>(&self, offset: usize) -> Option<&[Self::Item; ARRAY_LEN]> {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One is in iterator - trying to go past the end means end of iteration. The other is get method which is similar enough to get on slices to warrant matching API. std has Option there not Result. While returning indices in error types is somewhat convenient in erroneous cases using get is usually an operation that's handled by the caller.

I see, you're talking about when methods impl the trait. I guess doc comments here aren't need since it's implied by the trait.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was talking about where these methods are used.

apoelstra
apoelstra previously approved these changes Mar 18, 2025
Copy link
Copy Markdown
Member

@apoelstra apoelstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 474977f; successfully ran local tests

tcharding
tcharding previously approved these changes Mar 18, 2025
Copy link
Copy Markdown
Member

@tcharding tcharding left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 474977f

@apoelstra
Copy link
Copy Markdown
Member

Sorry, this needs another rebase. Process failure on my part -- I closed this tab because I was waiting for Tobin's ACK, and then I merged #4242 because it didn't conflict with any open tabs.

@Kixunil Kixunil dismissed stale reviews from tcharding and apoelstra via c1de142 March 18, 2025 14:44
@Kixunil Kixunil force-pushed the remove_try_into_expect branch from 474977f to c1de142 Compare March 18, 2025 14:44
@Kixunil
Copy link
Copy Markdown
Collaborator Author

Kixunil commented Mar 18, 2025

Crap, these ridiculous conflicts are driving me crazy. I was thinking for a while of having a version control system that would store the entire source as a tree (AST + files + directories), not text and make formatting completely machine-generated. It'd make this kind of reformatting impossible (but still settable client-side; obviously, rustfmt would need some more tweaks) and many changes would become trivially mergable - e.g. use items added to the same line.

Out of curiosity, does the idea sound appealing to you? Of course I'm not going to implement this any time soon, at least not in usable form, I'm just wondering if I'm the only one bothered by it. :)

@apoelstra
Copy link
Copy Markdown
Member

Out of curiosity, does the idea sound appealing to you?

Yes, though I suspect you can get 95% of the way there with custom git merge strategies. And the last 5% probably is not worth the switching cost of learning a new VCS.

In Bitcoin Core there is a notion of a "scripted diff" where if your diff is expressible in shell (e.g. cargo +nightly fmt and somehow we decide on compiler pins) then you can put that shell script into your commit message and CI will verify that the commit is exactly the result of the script. This dramatically simplifies review for stuff that is basically a sed -i across the codebase, but it could also be used as a git merge strategy, where if git sees a conflict between a scripted diff and a normal one, it attempts to apply the normal one and then re-run the script.

Relatedly, it would also be nice to have a git diff driver that collapsed out scripted stuff so you don't have to wade through a bajillion renames to find the one real change..

@Kixunil
Copy link
Copy Markdown
Collaborator Author

Kixunil commented Mar 18, 2025

I suspect you can get 95% of the way there with custom git merge strategies.

Some amount yes but I did a bit of digging into it and I haven't found any documentation on how to implement them (just merge drivers) and the one merge driver I made for merging use is pretty cool but for some reason git refuses to use it with cherry-picks. Also pijul as some cool ideas that I think would work even better with trees.

If we could just use non-nightly rustfmt it might be also worth applying it as a filter. Scripted diffs are nice but I'd love to see some tool that can do stuff like "rename this Rust variable".

Anyway, this all is just entirely theoretical.

tcharding
tcharding previously approved these changes Mar 19, 2025
Copy link
Copy Markdown
Member

@tcharding tcharding left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK c1de142

apoelstra
apoelstra previously approved these changes Mar 19, 2025
Copy link
Copy Markdown
Member

@apoelstra apoelstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK c1de142; successfully ran local tests

@apoelstra
Copy link
Copy Markdown
Member

Needs rebase after #4248.

Kixunil added 2 commits March 20, 2025 20:19
Previously we've used `try_into().expect()` because const generics were
unavailable. Then they became available but we didn't realize we could
already convert a bunch of code to not use panicking conversions. But we
can (and could for a while).

This adds an extension trait for arrays to provide basic non-panicking
operations returning arrays, so they can be composed with other
functions accepting arrays without any conversions. It also refactors a
bunch of code to use the non-panicking constructs but it's certainly not
all of it. That could be done later. This just aims at removing the
ugliest offenders and demonstrate the usefulness of this approach.

Aside from this, to avoid a bunch of duplicated work, this refactors
BIP32 key parsing to use a common method where xpub and xpriv are
encoded the same. Not doing this already led to a mistake where xpriv
implemented some additional checks that were missing in xpub. Thus this
change also indirectly fixes that bug.
These are defined in the BIP as invalid. The previous commit fixed a bug
where invalid key was parsed as valid and this bug can be caught by
these vectors. Therefore, if this commit is ordered before the last one
the test will fail.
@Kixunil Kixunil dismissed stale reviews from apoelstra and tcharding via 437562e March 20, 2025 19:19
@Kixunil Kixunil force-pushed the remove_try_into_expect branch from c1de142 to 437562e Compare March 20, 2025 19:19
@Kixunil
Copy link
Copy Markdown
Collaborator Author

Kixunil commented Mar 20, 2025

Damn, this is exactly the kind of merge conflict that should be resolvable in a smarter version control... Anyway, rebased.

@apoelstra
Copy link
Copy Markdown
Member

Yeah. Super frustrating that we gotta manually re-apply renames like this.

Queued for re-ACK and I'll also queue for merge since the range-diff from what Tobin ACKed is nearly trivial. (Though he may show up and sneak a new ACK in before my scripts are done running :)).

Copy link
Copy Markdown
Member

@tcharding tcharding left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 437562e

Copy link
Copy Markdown
Member

@apoelstra apoelstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 437562e; successfully ran local tests

@apoelstra apoelstra merged commit 6620a29 into rust-bitcoin:master Mar 21, 2025
24 checks passed
@Kixunil Kixunil deleted the remove_try_into_expect branch March 21, 2025 13:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

C-base58 PRs modifying the base58 crate C-bitcoin PRs modifying the bitcoin crate C-internals PRs modifying the internals crate C-primitives Correctness Improves the chance that the code will be correct - either our or consumer (via well-designed API)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants