Skip to content

Bugfix: Correct checksum's sha256 when retrieve from remote#25831

Merged
alalazo merged 2 commits intospack:developfrom
tldahlgren:bugfix-checksum-sha256-from-remote
Sep 13, 2021
Merged

Bugfix: Correct checksum's sha256 when retrieve from remote#25831
alalazo merged 2 commits intospack:developfrom
tldahlgren:bugfix-checksum-sha256-from-remote

Conversation

@tldahlgren
Copy link
Copy Markdown
Contributor

@tldahlgren tldahlgren commented Sep 8, 2021

Fixes #24971
Fixes #24668

This PR ensures the proper URL when not given versions on the command line by using the URL associated/derived for versions listed in the package.

For example, the output without this fix is:

$ spack checksum pmdk
==> Found 11 versions of pmdk:
  
  1.11.0-rc2  https://github.com/pmem/pmdk/releases/download/1.11.0-rc2/pmdk-1.11.0-rc2.tar.gz
  1.11.0-rc1  https://github.com/pmem/pmdk/releases/download/1.11.0-rc1/pmdk-1.11.0-rc1.tar.gz
  1.11.0      https://github.com/pmem/pmdk/releases/download/1.11.0/pmdk-1.11.0.tar.gz
  1.10-rc1    https://github.com/pmem/pmdk/releases/download/1.10-rc1/pmdk-1.10-rc1.tar.gz
  1.10        https://github.com/pmem/pmdk/releases/download/1.10/pmdk-1.10.tar.gz
  1.9.2       https://github.com/pmem/pmdk/releases/download/1.9.2/pmdk-1.9.2.tar.gz
  1.9.1       https://github.com/pmem/pmdk/releases/download/1.9.1/pmdk-1.9.1.tar.gz
  1.9         https://github.com/pmem/pmdk/releases/download/1.9/pmdk-1.9.tar.gz
  1.8.1       https://github.com/pmem/pmdk/releases/download/1.8.1/pmdk-1.8.1.tar.gz
  ...
  1.5         https://github.com/pmem/pmdk/archive/1.5.tar.gz

==> How many would you like to checksum? (default is 1, q to abort) 10
==> Fetching https://github.com/pmem/pmdk/releases/download/1.11.0-rc2/pmdk-1.11.0-rc2.tar.gz
==> Fetching https://github.com/pmem/pmdk/releases/download/1.11.0-rc1/pmdk-1.11.0-rc1.tar.gz
==> Fetching https://github.com/pmem/pmdk/releases/download/1.11.0/pmdk-1.11.0.tar.gz
==> Fetching https://github.com/pmem/pmdk/releases/download/1.10-rc1/pmdk-1.10-rc1.tar.gz
==> Fetching https://github.com/pmem/pmdk/releases/download/1.10/pmdk-1.10.tar.gz
==> Fetching https://github.com/pmem/pmdk/releases/download/1.9.2/pmdk-1.9.2.tar.gz
==> Fetching https://github.com/pmem/pmdk/releases/download/1.9.1/pmdk-1.9.1.tar.gz
==> Fetching https://github.com/pmem/pmdk/releases/download/1.9/pmdk-1.9.tar.gz
==> Fetching https://github.com/pmem/pmdk/releases/download/1.8.1/pmdk-1.8.1.tar.gz
==> Fetching https://github.com/pmem/pmdk/releases/download/1.7.1/pmdk-1.7.1.tar.gz
==> Fetching https://github.com/pmem/pmdk/archive/1.5.tar.gz

    version('1.11.0-rc2', sha256='25885ea35d2bb25fc5193135045efacf97f64b26df12585e885a849972cd850d')
    version('1.11.0-rc1', sha256='0d0947ab079be50ff752c184d04f6001d2bc95f2083c59c488ff4304c45deaec')
    version('1.11.0',     sha256='2116f30b1fbb3ee11b208f21f6eb81e594649608d85f6741d597a7fbea36143b')
    version('1.10-rc1',   sha256='8dd6fdad1a870f7f861e760c40aa3cc4fb248b9eec5f98a869c05cbd96971b5f')
    version('1.10',       sha256='08dafcf94db5ac13fac9139c92225d9aa5f3724ea74beee4e6ca19a01a2eb20c')
    version('1.9.2',      sha256='38c0dc5cec1145b1a42d7c3de8fa726f246346c29256c6658ccb264a00db54f3')
    version('1.9.1',      sha256='3f31b75b07ec484028b8c00918b2dcb410fed5e3683e471f04bdf1e957f39b75')
    version('1.9',        sha256='956186faa3feb0156f58da5578143cce22501b9b4a00133a490f997594cb466f')
    version('1.8.1',      sha256='0ae98ace247e2417718e64e3ea2cf7959cf7dce0e686f95ebfe3853a4c22c00e')
    version('1.7.1',      sha256='cc48cc623fd36f9fb7898dde7406802f4e1a4ad7c84c091979ce20cd5a0e14b0')

Using the change in this PR:

$ spack checksum pmdk
==> Found 14 versions of pmdk:
  
  1.11.0-rc2  https://github.com/pmem/pmdk/releases/download/1.11.0-rc2/pmdk-1.11.0-rc2.tar.gz
  1.11.0-rc1  https://github.com/pmem/pmdk/releases/download/1.11.0-rc1/pmdk-1.11.0-rc1.tar.gz
  1.11.0      https://github.com/pmem/pmdk/archive/1.11.0.tar.gz
  1.10-rc1    https://github.com/pmem/pmdk/releases/download/1.10-rc1/pmdk-1.10-rc1.tar.gz
  1.10        https://github.com/pmem/pmdk/archive/1.10.tar.gz
  1.9.2       https://github.com/pmem/pmdk/archive/1.9.2.tar.gz
  1.9.1       https://github.com/pmem/pmdk/archive/1.9.1.tar.gz
  1.9         https://github.com/pmem/pmdk/archive/1.9.tar.gz
  1.8.1       https://github.com/pmem/pmdk/archive/1.8.1.tar.gz
  ...
  1.5         https://github.com/pmem/pmdk/archive/1.5.tar.gz

==> How many would you like to checksum? (default is 1, q to abort) 11
==> Fetching https://github.com/pmem/pmdk/releases/download/1.11.0-rc2/pmdk-1.11.0-rc2.tar.gz
==> Fetching https://github.com/pmem/pmdk/releases/download/1.11.0-rc1/pmdk-1.11.0-rc1.tar.gz
==> Fetching https://github.com/pmem/pmdk/archive/1.11.0.tar.gz
==> Fetching https://github.com/pmem/pmdk/releases/download/1.10-rc1/pmdk-1.10-rc1.tar.gz
==> Fetching https://github.com/pmem/pmdk/archive/1.10.tar.gz
==> Fetching https://github.com/pmem/pmdk/archive/1.9.2.tar.gz
==> Fetching https://github.com/pmem/pmdk/archive/1.9.1.tar.gz
==> Fetching https://github.com/pmem/pmdk/archive/1.9.tar.gz
==> Fetching https://github.com/pmem/pmdk/archive/1.8.1.tar.gz
==> Fetching https://github.com/pmem/pmdk/archive/1.8.tar.gz
==> Fetching https://github.com/pmem/pmdk/archive/1.7.1.tar.gz

    version('1.11.0-rc2', sha256='25885ea35d2bb25fc5193135045efacf97f64b26df12585e885a849972cd850d')
    version('1.11.0-rc1', sha256='0d0947ab079be50ff752c184d04f6001d2bc95f2083c59c488ff4304c45deaec')
    version('1.11.0',     sha256='bfbc82e6bfd788c8bcb380da76172b83732d12775a719c9c423eb2fadc78bb3a')
    version('1.10-rc1',   sha256='8dd6fdad1a870f7f861e760c40aa3cc4fb248b9eec5f98a869c05cbd96971b5f')
    version('1.10',       sha256='06edcd43ef267c4cc70754d5d1a5d88aeb9f2086bc014bf2594df4c5efd8cc4e')
    version('1.9.2',      sha256='6bca98ecf9e036603951024b7436d688cd1907b2d8c428373697fafff4096a4f')
    version('1.9.1',      sha256='3d0ea15099d6dc7b454a67ecd0ed04d7426ff05ee0331a221cb384e293d841f0')
    version('1.9',        sha256='2c8a148070f4bbf9f82e2ca63d2f84cb5101fc6e72c1ba93cc673ca3b7b95467')
    version('1.8.1',      sha256='ee4e93bbf29976eac2444e4eb04a862f38b8446f7400f8f7cdcf58febf6f6ba2')
    version('1.8',        sha256='a241ea76ef76d233cb92826b6823ed48091a2fb6963282a4fea848dbce68aa21')
    version('1.7.1',      sha256='7e98c0522a3e96b64822902c66024e24455f4742246c679cc0f46036ef4685bc')

The differences for the overlapping versions (PR vs. without PR):

    version('1.11.0-rc2', sha256='25885ea35d2bb25fc5193135045	    version('1.11.0-rc2', sha256='25885ea35d2bb25fc5193135045
    version('1.11.0-rc1', sha256='0d0947ab079be50ff752c184d04	    version('1.11.0-rc1', sha256='0d0947ab079be50ff752c184d04
    version('1.11.0',     sha256='bfbc82e6bfd788c8bcb380da761 |	    version('1.11.0',     sha256='2116f30b1fbb3ee11b208f21f6e
    version('1.10-rc1',   sha256='8dd6fdad1a870f7f861e760c40a	    version('1.10-rc1',   sha256='8dd6fdad1a870f7f861e760c40a
    version('1.10',       sha256='06edcd43ef267c4cc70754d5d1a |	    version('1.10',       sha256='08dafcf94db5ac13fac9139c922
    version('1.9.2',      sha256='6bca98ecf9e036603951024b743 |	    version('1.9.2',      sha256='38c0dc5cec1145b1a42d7c3de8f
    version('1.9.1',      sha256='3d0ea15099d6dc7b454a67ecd0e |	    version('1.9.1',      sha256='3f31b75b07ec484028b8c00918b
    version('1.9',        sha256='2c8a148070f4bbf9f82e2ca63d2 |	    version('1.9',        sha256='956186faa3feb0156f58da55781
    version('1.8.1',      sha256='ee4e93bbf29976eac2444e4eb04 |	    version('1.8.1',      sha256='0ae98ace247e2417718e64e3ea2
    version('1.7.1',      sha256='7e98c0522a3e96b64822902c660 |	    version('1.7.1',      sha256='cc48cc623fd36f9fb7898dde740

And the checksums listed in the package correspond to the versions with difference above, with or without this PR:

$ spack checksum pmdk 1.11.0 1.10 1.9.2 1.9.1 1.9 1.8.1 1.7.1
==> Found 7 versions of pmdk:
  
  1.11.0  https://github.com/pmem/pmdk/archive/1.11.0.tar.gz
  1.10    https://github.com/pmem/pmdk/archive/1.10.tar.gz
  1.9.2   https://github.com/pmem/pmdk/archive/1.9.2.tar.gz
  1.9.1   https://github.com/pmem/pmdk/archive/1.9.1.tar.gz
  1.9     https://github.com/pmem/pmdk/archive/1.9.tar.gz
  1.8.1   https://github.com/pmem/pmdk/archive/1.8.1.tar.gz
  1.7.1   https://github.com/pmem/pmdk/archive/1.7.1.tar.gz

==> Fetching https://github.com/pmem/pmdk/archive/1.11.0.tar.gz
==> Fetching https://github.com/pmem/pmdk/archive/1.10.tar.gz
==> Fetching https://github.com/pmem/pmdk/archive/1.9.2.tar.gz
==> Fetching https://github.com/pmem/pmdk/archive/1.9.1.tar.gz
==> Fetching https://github.com/pmem/pmdk/archive/1.9.tar.gz
==> Fetching https://github.com/pmem/pmdk/archive/1.8.1.tar.gz
==> Fetching https://github.com/pmem/pmdk/archive/1.7.1.tar.gz

    version('1.11.0', sha256='bfbc82e6bfd788c8bcb380da76172b83732d12775a719c9c423eb2fadc78bb3a')
    version('1.10',   sha256='06edcd43ef267c4cc70754d5d1a5d88aeb9f2086bc014bf2594df4c5efd8cc4e')
    version('1.9.2',  sha256='6bca98ecf9e036603951024b7436d688cd1907b2d8c428373697fafff4096a4f')
    version('1.9.1',  sha256='3d0ea15099d6dc7b454a67ecd0ed04d7426ff05ee0331a221cb384e293d841f0')
    version('1.9',    sha256='2c8a148070f4bbf9f82e2ca63d2f84cb5101fc6e72c1ba93cc673ca3b7b95467')
    version('1.8.1',  sha256='ee4e93bbf29976eac2444e4eb04a862f38b8446f7400f8f7cdcf58febf6f6ba2')
    version('1.7.1',  sha256='7e98c0522a3e96b64822902c66024e24455f4742246c679cc0f46036ef4685bc')

match the changes from this PR.

@tldahlgren tldahlgren added the bugfix Something wasn't working, here's a fix label Sep 8, 2021
@spackbot-app spackbot-app bot added the commands label Sep 8, 2021
@adamjstewart
Copy link
Copy Markdown
Member

Don't have time to test this thoroughly at the moment, but if this fixes the issue that would be amazing. I think there may be many more issues with the same problem as #24971, it's a very commonly reported bug.

@tldahlgren
Copy link
Copy Markdown
Contributor Author

tldahlgren commented Sep 8, 2021

Don't have time to test this thoroughly at the moment, but if this fixes the issue that would be amazing. I think there may be many more issues with the same problem as #24971, it's a very commonly reported bug.

I included the output in the description to show that this does fix #24971 . I checked other PRs (i.e., #24861) and this does not fix that problem. I still need to check #24668.

@tldahlgren
Copy link
Copy Markdown
Contributor Author

Don't have time to test this thoroughly at the moment, but if this fixes the issue that would be amazing. I think there may be many more issues with the same problem as #24971, it's a very commonly reported bug.

I included the output in the description to show that this does fix #24971 . I checked other PRs (i.e., #24861) and this does not fix that problem. I still need to check #24668.

Confirmed that this PR fixes the issue with #24668 so the py-setuptools-rust package can be updated per comments related to 0.10.6 and spack checksum py-setuptools-rust will yield the correct sha256 for that version.

@tldahlgren
Copy link
Copy Markdown
Contributor Author

@spackbot run pipeline

@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Sep 10, 2021

I've started that pipeline for you!

@tldahlgren
Copy link
Copy Markdown
Contributor Author

@mdorier @manuelakuhn Does this PR fix your issues?

@tldahlgren tldahlgren force-pushed the bugfix-checksum-sha256-from-remote branch from 04aad20 to 97a7932 Compare September 10, 2021 19:56
@mdorier
Copy link
Copy Markdown
Contributor

mdorier commented Sep 13, 2021

@tldahlgren if you have tested it with pmdk and it works, then yes it fixes it, since that's where I was having the issue. Thanks!

@alalazo
Copy link
Copy Markdown
Member

alalazo commented Sep 13, 2021

@spackbot run pipeline

@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Sep 13, 2021

I've started that pipeline for you!

@alalazo alalazo enabled auto-merge (squash) September 13, 2021 07:22
@alalazo alalazo merged commit a1d792a into spack:develop Sep 13, 2021
trws added a commit to trws/spack that referenced this pull request Mar 1, 2022
Checksum was only actually scraping when called with no versions.  It
now always scrapes and then selects URLs from the set of URLs known to
exist whenever possible.

fixes spack#25831
trws added a commit that referenced this pull request Mar 19, 2022
* lower priority of package-provided urls

This change favors urls found in a scraped page over those provided by
the package from `url_for_version`.  In most cases this doesn't matter,
but R specifically returns known bad URLs in some cases, and the
fallback path for a failed fetch uses `fetch_remote_versions` to find a
substitute.  This fixes that problem.

fixes #29204

* consider what links actually exist in all cases

Checksum was only actually scraping when called with no versions.  It
now always scrapes and then selects URLs from the set of URLs known to
exist whenever possible.

fixes #25831

* bow to the wrath of flake8

* test-fetch urls from package, prefer if successful

* Update lib/spack/spack/package.py

Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>

* reword as suggested

* re-enable mypy specific ignore and ignore pyflakes

* remove flake8 ignore from .flake8

* address review comments

* address comments

* add sneaky missing substitute

I missed this one because we call substitute on a URL that doesn't
contain a version component.  I'm not sure how that's supposed to work,
but apparently it's required by at least one mock package, so back in it
goes.

Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugfix Something wasn't working, here's a fix commands

Projects

None yet

Development

Successfully merging this pull request may close these issues.

spack checksum prefers version-specific url over global url spack checksum gives incorrect sha256 when discovering versions by itself

4 participants