503 errors when checking certain URLs #1261
Description
I'm getting some errors with planemo shed_lint when checking certain URLs, both in CI and local testing.
Example from this CI run:
Applying linter tool_urls... FAIL
.. ERROR: Error '503 Server Error: Service Temporarily Unavailable for url: https://pubs.acs.org/doi/10.1021/ct300849w' accessing https://doi.org/10.1021/ct300849w
.. ERROR: Error '503 Server Error: Service Temporarily Unavailable for url: https://pubs.acs.org/doi/10.1021/ct401002w' accessing https://doi.org/10.1021/ct401002w
.. ERROR: Error '503 Server Error: Service Temporarily Unavailable for url: https://pubs.acs.org/doi/10.1021/jp506557r' accessing https://doi.org/10.1021/jp506557r
.. ERROR: Error '503 Server Error: Service Temporarily Unavailable for url: https://pubs.acs.org/doi/10.1021/ct5009137' accessing https://doi.org/10.1021/ct5009137
.. ERROR: Error '503 Server Error: Service Temporarily Unavailable for url: https://onlinelibrary.wiley.com/doi/10.1002/pssb.200301886' accessing https://doi.org/10.1002/pssb.200301886
.. INFO: URL OK https://creativecommons.org/licenses/by-sa/4.0/legalcode
.. INFO: URL OK https://doi.org/10.1016/S0038-1098(99)00137-4
.. INFO: URL OK https://doi.org/10.1103/PhysRevB.64.085333
.. INFO: URL OK https://doi.org/10.1016/j.susc.2005.10.044
.. INFO: URL OK https://doi.org/10.1016/j.chemphys.2004.03.034
.. INFO: URL OK https://www.osti.gov/etdeweb/biblio/20249635
.. INFO: URL OK https://www.dftb.org/parameters/download/3ob/3ob-3-1-cc/
.. INFO: URL OK https://www.dftb.org/parameters/download/pbc/pbc-0-3-cc/
When I click through manually, the links work fine - both the full link and the DOI link. Not all the URLs raise errors, and the ones that fail do so consistently, so I don't think the service is 'temporarily unavailable' (this has been the same for months). Perhaps it's something specific that these websites are doing that blocks access, but then I'd expect a 403 or 401 like in #578 and #677.
Any thoughts on whether this is something that can be fixed on the Planemo side? If not I guess I'll just have to disable --urls in my CI checks.