Just to document, this was one of the crates proposed in #550.

After a nights sleep I had a new idea, add a "crypto" feature to primitives and moves the bitcoin::crypto module into primitives. All the crypto stuff is then feature gated and bitcoin depends on primitives with the "crypto" feature enabled.

This leaves only a single NetworkExt trait in bitcoin.

Yeah, I think this is the right approach.

For future reviewers: the issue is that WitnessProgram and Script need constructors from public keys and signatures, but these are cryptographic objects which can't even be de/serialized without help from libsecp. If they live in a separate crate then we wind up needed extension traits or something to implement basic functionality, which will be undiscoverable and hard to maintain.

So I guess, instead we have primitives which contains the core Bitcoin data structures (blocks and everything in them), and within primitives we have a feature-gated crypto module where signatures and keys go.

We would also want to re-export secp256k1 from primitives I think.

Feature-gating this -- which we definitely want to do, because a design goal of primitives is that users can depend on it and get a smallish crate that implements all the "stupid, harmless" components of bitcoin -- would be the bulk of the work of getting an optional libsecp, something that WASM people have requested for a long time.

Then in the long term we would try to move some components out from under the feature gate. In particular I'm pretty sure we can serialize/deserialize signatures in Rust quite easily. Public keys are harder but may be worth doing.

While I still think this is the right approach, and getting "make secp optional" basically for free is super awesome I realized this has one massive drawback - it means that primitives will not stabalize any time soon at least with the "crypto" feature enabled. Its not as bad as it sounds because we don't update scep that much but its not nothing.

Ok, so where does that leave us?

• We need to be able to stabilize Script and WitnessProgram which are definitely primitives
• ...which need the ability to serialize (and maybe parse?) public keys and signatures
• ...which needs libsecp.

I guess, if all we need is serialization, that's pretty easy to implement in pure Rust. Do you think that's all we need for primitives? In this case we should refactor our PublicKey, CompressedPublicKey and Signature types to directly contain byte-blobs rather than containing rust-secp types.

However, we would still want the ability to convert these to the "underlying" rust-secp types (or do we?) because sometimes users want that. We could make them do it manually by serializing and deserializing but that's pretty expensive and would have an inaccessible error path on parsing. We should benchmark it at least.

Probably we need to pull this into a separate discussion thread.

I guess, if all we need is serialization, that's pretty easy to implement in pure Rust

Deserialization in Rust would have to be more dumb than what it is today, when we deserialize in secp256k1 we call into libsecp to do some validation on the slice. Or can we pull whatever the validation logic is out into Rust as well?

Do you think that's all we need for primitives?

I went over the whole primitives crate, and to the best of my knowledge all we need is, as you say, serialization and deserialization of keys and signatures.

Probably we need to pull this into a separate discussion thread.

I think we can move forward with this PR and make ser/deser of crypto feature gated stuff part of "stabalize primitives". Although I have no idea how we move forward with the current reviewers. I can either:

1. Tease this PR apart into somewhat reviewable pieces and we use the Andrew acks and either we get another ack or wait two weeks merge method or
2. I could put more work into the "alloc" feature gate first then do (1)

Deserialization in Rust would have to be more dumb than what it is today, when we deserialize in secp256k1 we call into libsecp to do some validation on the slice.

It's not just validation. We decompress compressed keys. If we wanted to avoid this, it would be pretty complicated because the internal representation would then need to be an enum.

Let's meet at the Nashville conference and try to do it in sync.

Ah good idea!

What about if we branch off at v0.32.2 and do primitives, then we can release it (v0.1.0-beta) as a drop in replacement for projects already using v0.32.2. This will make it easier to develop, no rebasing, and easier to test, because no usage changes.

Hi @klebs6, I see your bitcoin-primitives crate has not been updated in a while. Is there any chance you would be willing please to transfer the name to us?

You could re-release yours with a more project specific prefix?

Thanks in advance for the consideration.

This needs Address moving over to primitives as well.

I took a look at the crates in bdk/crates and it looks like the following ones will be able to depend directly on primitives

• bitcoind_rpc
• chain
• electrum
• esplora
• testenv

That's a win. cc @notmandatory

What about if we branch off at v0.32.2 and do primitives, then we can release it (v0.1.0-beta) as a drop in replacement for projects already using v0.32.2.

I think we should branch off of master .. projects already using 0.32.2 will already need to change their Cargo.toml to switch to primitives so I don't think it's a big deal to also make them tweak whatever minor breaking changes we've made to primitives since then (I'm unsure there are any).

In any case, I like this idea ... though it then means that we have two parallel implementations of primitives and we'd want to start cutting over ASAP to turning the bitcoin ones into a facade for the primitives ones.

Another idea, what about if I just rebase this branch haphazardly, then we can use this branch to test downstream, and also to get a state we want to achieve. Then in Nashville we freeze all merging and do the whole thing from scratch so no mistakes are introduced.

Yep, that sounds good to me.

Hi @klebs6, I see your bitcoin-primitives crate has not been updated in a while. Is there any chance you would be willing please to transfer the name to us?

You could re-release yours with a more project specific prefix?

Thanks in advance for the consideration.

certainly! i think i can take a look this weekend and make the transfer as we discussed before!

I'm out of loop here and I just quickly scrolled through comments. I think Address should have its own crate.

This is tempting. I think I agree (assuming it's possible; I did a quick check to see if there were methods on Script that depended on Address existing but it looks like no).

The address would have deps on base58, bech32, etc (which we can feature-gate) while primitives can maybe get away with no deps. Unsure. I think it may need hashes. As discussed above it would also need crypto but I think we can eventually feature-gate it.

For crypto I'd make yet another crate. FTR I co-own bitcoin-keys which I made with @dr-orlovsky to potentially replace the current types but IIRC I got tangled in something. If we can reconcile the ideas that'd be really nice.

My hope with crypto is that we can define the types and their serialization directly in primitives, and have a feature-gate for libsecp backing (which would provide the ability to actually sign and verify things).

As a first step though I think we should move everything into primitives since that's a big enough job as-is, and then work on moving stuff further out of primitives.

It seems that this primitives crate is not really that meaningfully testable without also having psbt and bip32 sorted out, then we can test with miniscript. Also I think that this crate smashing stuff is abstract/entangled enough that we need to see running code to talk. I'm going to forge forward adding those two additional crates so we have a base branch to base discussions off of. This should be considered a vision-of-the-future branch not a merge candidate.

Sounds good. And agreed on taking on bip32 or psbt first. Both will probably have a fair bit of API arguing but they should be far easier to test independently.

Oh they can't be first, can be at the same time though. bip32 depends then on primitives and psbt on primitives and bip32.

I might try splitting primitives myself later. I have some urgent work now but maybe in a week or two I could find the time for it.

greetings! i am almost done with refactoring (and preparing for transfer) the current bitcoin-primitives crate! should be there soon!

greetings! i am almost done with refactoring (and preparing for transfer) the current bitcoin-primitives crate! should be there soon!

Ready! I sent ownership invitations for the crate via crates.io.

Earlier, I factored the contents of the previous bitcoin-primitives crate into the following:
bitcoin-u256
bitcoin-u160
bitcoin-bitstream
bitcoin-blob
bitcoin-bigint
bitcoin-autofile
bitcoin-bufferedfile
bitcoin-vectorstream

Next, I removed the bitcoin_primitives dependency throughout the workspace and added a dependency pointing to the new crates where necessary.

bitcoin-primitives crate is now available!

Thank you!! Invitation accepted.

Madness, thanks @klebs6! Note we should be careful with the version number we use for bitcoin-primitives (the current usage of 0.1.0 is now wrong). How about we do 1.0.0-beta to signal that 1.0.0 is imminent (and we sort our shit out and make it happen)?

@tcharding please, let's not use x.y.z-foo style. It messes up version and is annoying to work with. Just select a high enough number and later we'll get proper 1.0.

Yeah, I think we should just use 0.100.0 or something. On the 5th iteration we're gonna be pretty mad at ourselves for using 1.0.0-beta.N or whatever. And that's assuming we even get the syntax right so that cargo update doesn't comingle them all.

Fair enough.

Replaced by #2883