r? @alexcrichton

(rust_highfive has picked a reviewer for you, use r? to override)

Just curious, would it be possible to implement Intern<T> to avoid all the duplication? Or maybe use one of the many crates like internment?

That's the next challenge: "intern" this cache infrastructure 😉

☔ The latest upstream changes (presumably #6469) made this pull request unmergeable. Please resolve the merge conflicts.

Out of curiosity, do we know a location where we clone these a lot or create a lot of duplicate ones? It's not as clear to me with source/package ids that Cargo would benefit from this

Paging @Eh2406 😄

@dwijnand good question why did I say that...

looking back at the original, it is not so much that we cloned it a lot. Rather they are larger structures then they seem, and we probably have the same values menny times. (How many crates have a version 0.1.0?) So it may save memory by deduping. Also we already are leaking some of them by PackageId being interned.

Now that I am saying it I am not entirely convinced. Maybe given that PackageId Is Interned it is redundant to inter this ass well? Or maybe on the flip side, If we intern this then PackageId is just 3 pointers, and maybe it should not be interned?

@ehuss this interning strategy was inspired by internment. Each time we have copied this code we have needed to override the impl of Hash, so I don't think we can use something off the shelf. But there is definitely room to remove some of the duplication.

@dwijnand would you be up for doing some measurements perhaps? Checking memory differences for resolving large crate graphs like rustc or servo?

Sure! How?

The "easiest" is probably Valgrind with massif on Linux perhaps?

I don't have the bandwidth. The PR is here and it's not going anywhere, so we can always reconsider this change down the line. I'll salvage some of the diffs in a new PR.