Skip to content

Resolve non-determistic behavior in preferences due to site-packages ordering#2780

Merged
zanieb merged 9 commits intomainfrom
zb/test-installed-multiple
Apr 2, 2024
Merged

Resolve non-determistic behavior in preferences due to site-packages ordering#2780
zanieb merged 9 commits intomainfrom
zb/test-installed-multiple

Conversation

@zanieb
Copy link
Copy Markdown
Member

@zanieb zanieb commented Apr 2, 2024

Originally a regression test for #2779 but we found out that there's some weird behavior where different anyio versions were preferred based on the platform.

@zanieb zanieb added the testing Internal testing of behavior label Apr 2, 2024
Comment thread crates/uv/tests/pip_install.rs Outdated

// Request the second anyio version again
// Should remove both previous versions and reinstall the second one
uv_snapshot!(context1.filters(), context1.install().arg("anyio==4.0.0"), @r###"
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I do not request 4.0.0, 3.7.0 is reinstalled without panic on my machine 🤯

@zanieb
Copy link
Copy Markdown
Member Author

zanieb commented Apr 2, 2024

Cherry-picked over to #2779 — we can discuss the weirdness of this test here if necessary but I'll merge the clear fix anyway.

@zanieb zanieb changed the title Add regression test for #2779 Resolve non-determistic behavior in preferences due to site-packages ordering Apr 2, 2024
Comment on lines +50 to +53
let mut entries = site_packages.collect::<Result<Vec<_>, std::io::Error>>()?;
// TODO(zanieb): Consider filtering to just directories to reduce the size of the sort
// Sort for determinism, `read_dir` is different per-platform
entries.sort_by_key(fs_err::DirEntry::path);
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't figure out how to write this as a single iterator operation because it's weird working with nested results.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean, not collecting the entries here at all?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We must collect them to sort them afaik. I just couldn't filter to directories here.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. You would need to filter prior to the collect above.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I tried that haha it's awkward because of the nested Result types, it's beyond my understanding of iterators in Rust.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @snowsignal maybe you have a suggestion

Copy link
Copy Markdown
Contributor

@snowsignal snowsignal Apr 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zanieb Here's one way you could do it:

let mut entries: Vec<_> = site_packages
                        .filter(|read_dir| match read_dir {
                            Ok(entry) => entry
                                .file_type()
                                .map(|file_type| file_type.is_dir())
                                .unwrap_or_default(),
                            Err(_) => true,
                        })
                        .collect::<std::io::Result<_>>()?;

Also, if you tweak the code to collect into a BTreeSet<PathBuf>, the sorting will happen as part of collect() and you don't have to call sort_by_key afterwards.

Copy link
Copy Markdown
Contributor

@snowsignal snowsignal Apr 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's how you could do that:

                    let paths: BTreeSet<_> = site_packages
                        .filter_map(|read_dir| match read_dir {
                            Ok(entry) => {
                                entry.file_type().ok()?.is_dir().then_some(Ok(entry.path()))
                            }
                            Err(err) => Some(Err(err)),
                        })
                        .collect::<Result<_>>()?;

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool thank you!

// Index all installed packages by name.
for entry in site_packages {
let entry = entry?;
if entry.file_type()?.is_dir() {
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the call we'd move into the iterator chain above

Comment on lines +3569 to +3571
// Request the anyio without a version specifier
// This is loosely a regression test for the ordering of the installation preferences
// from existing site-packages
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before sorting the site-packages, this preferred anyio==3.7.0 on macOS and anyio==4.0.0 on Ubuntu

@zanieb zanieb requested a review from charliermarsh April 2, 2024 18:08
@zanieb zanieb added bug Something isn't working and removed testing Internal testing of behavior labels Apr 2, 2024
@zanieb zanieb marked this pull request as ready for review April 2, 2024 18:13
@zanieb zanieb merged commit 1ac9672 into main Apr 2, 2024
@zanieb zanieb deleted the zb/test-installed-multiple branch April 2, 2024 18:48
zanieb added a commit that referenced this pull request Apr 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants