Skip to content

Run CI for primitives#3195

Closed
tcharding wants to merge 3 commits intorust-bitcoin:masterfrom
tcharding:08-21-ci-primitives
Closed

Run CI for primitives#3195
tcharding wants to merge 3 commits intorust-bitcoin:masterfrom
tcharding:08-21-ci-primitives

Conversation

@tcharding
Copy link
Copy Markdown
Member

Epic fail here, primitives is not being checked in CI. This is the second big CI bug we have had in as many weeks.

Turns out we have a bunch serde code that assumes there is an allocator. Patch 2 enables alloc from serde - this is in line with feature gating wildly on alloc when moving code from bitcoin during crates smashing.

This PR does not claim to be the best solution but is a small step forward.

We forgot to add the `ordered` feature to `test_vars.sh`.
Currently the `serde` feature requires an allocator, this is a hang over
from code being written for the `bitcoin` crate.

This was not found because we are not testing `primitives` correctly in
CI (there is the `fuzz` bug and also `primitives` is not even in the
CRATES list).
This should have been done when we introduced the `primitives`
crate - epic fail.

Intentionally put it before `fuzz` because of the known bug in the CI
script where nothing after `fuzz` gets tested.
Copy link
Copy Markdown
Collaborator

@Kixunil Kixunil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 36a7476 but I want Tobin to look at my comments and consider them before merging.

std = ["alloc", "internals/std", "io/std", "units/std"]
alloc = ["internals/alloc", "io/alloc", "units/alloc"]
serde = ["dep:serde", "internals/serde", "units/serde"]
serde = ["dep:serde", "internals/serde", "units/serde", "alloc"]
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should try to avoid this before 1.0 because then it'll become kinda breaking. (Strictly, people should enable every feature they use directly but it's likely they'll miss it.) We should just bound those serde imlps on all(feature = "serde", feature = alloc). Preferably in this PR but I can tolerate a separate PR if it's before release.


# Crates in this workspace to test (note "fuzz" is only built not tested).
CRATES=("addresses" "base58" "bitcoin" "fuzz" "hashes" "internals" "io" "units")
CRATES=("addresses" "base58" "bitcoin" "primitives" "fuzz" "hashes" "internals" "io" "units")
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The real process mega fail is that we're no longer using automated way to get the list of the crates. I know you said this PR is not perfect but I'd really, really like to see it fixed right here rather than kicking it down the road. You should be able to just copy the command from old revisions. And regarding fuzz order, I suggest to just patch-out the crate from the list and then explicitly add it to the end (why isn't it at the end already?)

@tcharding
Copy link
Copy Markdown
Member Author

tcharding commented Aug 21, 2024

@Kixunil, both your points above are valid. I ask for merge of this as is so we can get #3194 in. I will start working however on fixes to both your concerns. FTR #3194 enables hex in the serde feature as well so is even worse. I'll fix it all in one go. The CRATES thing I'll do now.

@tcharding tcharding marked this pull request as draft August 21, 2024 06:49
@tcharding tcharding marked this pull request as ready for review August 21, 2024 06:50
@apoelstra
Copy link
Copy Markdown
Member

FWIW I have been testing primitives locally. Presumably the issues found here are ones that would have been caught had we merged #2861 (now #3200) so that I could also start running doctests.

I'm not going to look at this until #3200 is in.

@tcharding
Copy link
Copy Markdown
Member Author

I'm confused by the ordering requirements of this PR and #3200 to get past your local CI, I don't see any?

@apoelstra
Copy link
Copy Markdown
Member

They both need to be merged to pass my local CI. It doesn't matter what order.

@apoelstra
Copy link
Copy Markdown
Member

But I am annoyed that #2861 sat for ages, leaving my local CI in a degraded state, and then you had to independently discover that the Github CI was broken. This would have been immediately noticed when doing final tests on #2861.

@tcharding
Copy link
Copy Markdown
Member Author

But I am annoyed that #2861 sat for ages

Annoyed at me, at yourself, or at our process?

@tcharding tcharding mentioned this pull request Aug 22, 2024
@tcharding
Copy link
Copy Markdown
Member Author

tcharding commented Aug 22, 2024

I created #3206 in case its easier for you to merge as a single PR. And I'm going to work now on doing #2861 on top of #3206.

@apoelstra
Copy link
Copy Markdown
Member

Annoyed at me, at yourself, or at our process?

Some combination of you and Kix that led to a "fix CI which is currently multiply broken on master" PR being mixed up with controversial non-CI changes.

apoelstra added a commit that referenced this pull request Aug 22, 2024
009e747 CI: Put fuzz last in CRATES list (Tobin C. Harding)
aed61bd Implement FromStr and serde impls for siphash (Tobin C. Harding)
f884610 siphash: Make functions inherent (Tobin C. Harding)
321d82c hashes: Pin in extra_test (Tobin C. Harding)
42c7617 Fix shchemars test (Tobin C. Harding)
b6fda65 Implement JsonSchema for siphash::Hash (Tobin C. Harding)
1af6ff4 hashes: Feature gate hash_reader unit test (Tobin C. Harding)
5230d33 Remove hash_reader from sha256t_hash_newtype (Tobin C. Harding)

Pull request description:

  This is just the bug fixes pulled out of #2861

  The root problem here is the bug described in rust-bitcoin/rust-bitcoin-maintainer-tools#10

  This PR works around the bug by putting `fuzz` last in the list.

ACKs for top commit:
  apoelstra:
    ACK 009e747; confirmed that local CI STILL does not pass after this; needs #3195 as well; but it is improved
  Kixunil:
    ACK 009e747 but I'd much rather see the siphash problems addressed by splitting up the macro. I can make a PR for that (but just that) later.

Tree-SHA512: f724570003b862e994787192b720c4698e6b904ececa7188770ee507dd9e12382ae64db3363fa15ab3f347341c99ed9a75938f4da0438ce3a0fa88717497a233
@apoelstra
Copy link
Copy Markdown
Member

#3200 is in. Can you rebase this?

@apoelstra
Copy link
Copy Markdown
Member

Rebased in #3220. Closing this in favor of that.

@apoelstra apoelstra closed this Aug 22, 2024
apoelstra added a commit that referenced this pull request Aug 23, 2024
d03e545 ci: fix `rev` to be `ref` when pinning bitcoin-maintainer-tools (Andrew Poelstra)
d04b6aa bitcoin: add a couple missing prelude imports (Andrew Poelstra)
30f6bd4 Add primitives to the CRATES list (Andrew Poelstra)
e1e0230 primitives: Enable alloc from serde (Tobin C. Harding)
ce36a37 primitives: Test ordered feature (Tobin C. Harding)

Pull request description:

  This is a rebase of #3195 on master (since #3200 was merged) and adds one commit which fixes a few missing imports, which it appears was missed in the other "fix CI" PRs.

ACKs for top commit:
  Kixunil:
    ACK d03e545
  tcharding:
    ACK d03e545

Tree-SHA512: 6f5dc5c2cbb42716e83e46e70dbf4d644f9e80f6b3d2e75a276e37d33d461f4a86e67eb5c32444dbd23bb9767abedc88bfe958b1beba39852c382cf73736711f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants