Skip to content

deps.dev does not find some PyPI package github links #980

@art1f1c3R

Description

@art1f1c3R

When attempting to run macaron analyze on two common packages torch and tensorflow using PURLs pkg:pypi/torch and pkg:pypi/tensorflow respectively, a GitHub repository link was not found for either although one exists.
Example output for torch:

macaron -v analyze -purl pkg:pypi/torch
2025-02-07 14:24:39,878 [macaron.__main__:main:556] [INFO] Setting the output directory to /home/carl_flottmann/macaron/output
2025-02-07 14:24:39,879 [macaron.__main__:main:573] [INFO] The logs will be stored in debug.log
2025-02-07 14:24:39,882 [macaron.slsa_analyzer.analyzer:__init__:95] [INFO] The following checks are excluded based on the user configuration: []
2025-02-07 14:24:39,882 [macaron.slsa_analyzer.analyzer:__init__:99] [INFO] The following checks will be run: ['mcn_provenance_available_1', 'mcn_version_control_system_1', 'mcn_detect_malicious_metadata_1', 'mcn_provenance_derived_commit_1', 'mcn_provenance_level_three_1', 'mcn_build_tool_1', 'mcn_provenance_derived_repo_1', 'mcn_build_as_code_1', 'mcn_provenance_expectation_1', 'mcn_find_artifact_pipeline_1', 'mcn_build_script_1', 'mcn_scm_authenticity_1', 'mcn_provenance_verified_1', 'mcn_build_service_1', 'mcn_trusted_builder_level_three_1', 'mcn_provenance_witness_level_one_1']
2025-02-07 14:24:39,960 [macaron.repo_finder.provenance_finder:find_provenance:58] [DEBUG] Seeking provenance of: pkg:pypi/torch
2025-02-07 14:24:39,960 [macaron.repo_finder.provenance_finder:find_provenance:82] [DEBUG] Provenance finding not supported for PURL type: pypi
2025-02-07 14:24:39,960 [macaron.repo_finder.repo_finder:to_repo_path:153] [INFO] The PURL type of pkg:pypi/torch is not valid as a repository type.
2025-02-07 14:24:39,960 [macaron.repo_finder.repo_finder:find_repo:103] [DEBUG] Analyzing pkg:pypi/torch with Repo Finder: <class 'macaron.repo_finder.repo_finder_deps_dev.DepsDevRepoFinder'>
2025-02-07 14:24:39,961 [macaron.util:send_get_http_raw:160] [DEBUG] GET - https://api.deps.dev/v3alpha/purl/pkg%3Apypi%2Ftorch
2025-02-07 14:24:41,600 [macaron.util:send_get_http_raw:160] [DEBUG] GET - https://api.deps.dev/v3alpha/purl/pkg%3Apypi%2Ftorch%402.6.0
2025-02-07 14:24:42,220 [macaron.repo_finder.repo_finder_deps_dev:_read_json:228] [DEBUG] Could not extract 'version' or 'links' from deps.dev response.
2025-02-07 14:24:42,220 [macaron.repo_finder.repo_finder_deps_dev:find_repo:69] [DEBUG] Failed to extract repository URLs from json data: pkg:pypi/torch
2025-02-07 14:24:42,220 [macaron.repo_finder.repo_finder:find_repo:110] [DEBUG] Could not find repo for PURL: pkg:pypi/torch
2025-02-07 14:24:42,221 [macaron.util:send_get_http_raw:160] [DEBUG] GET - https://api.deps.dev/v3alpha/purl/pkg%3Apypi%2Ftorch
2025-02-07 14:24:42,833 [macaron.repo_finder.repo_finder:get_latest_purl_if_different:303] [DEBUG] Found new version of PURL: pkg:pypi/torch@2.6.0
2025-02-07 14:24:42,834 [macaron.util:send_get_http_raw:160] [DEBUG] GET - https://api.deps.dev/v3alpha/purl/pkg%3Apypi%2Ftorch%402.6.0
2025-02-07 14:24:43,461 [macaron.repo_finder.repo_finder_deps_dev:_read_json:228] [DEBUG] Could not extract 'version' or 'links' from deps.dev response.
2025-02-07 14:24:43,461 [macaron.repo_finder.repo_finder_deps_dev:find_repo:69] [DEBUG] Failed to extract repository URLs from json data: pkg:pypi/torch@2.6.0
2025-02-07 14:24:43,461 [macaron.repo_finder.repo_finder:find_repo:118] [DEBUG] Could not find repo from latest version of PURL: pkg:pypi/torch@2.6.0

And for tensorflow:

macaron -v analyze -purl pkg:pypi/tensorflow
2025-02-07 11:30:56,325 [macaron.__main__:main:556] [INFO] Setting the output directory to /home/carl_flottmann/macaron/output
2025-02-07 11:30:56,325 [macaron.__main__:main:573] [INFO] The logs will be stored in debug.log
2025-02-07 11:30:56,326 [macaron.slsa_analyzer.analyzer:__init__:95] [INFO] The following checks are excluded based on the user configuration: []
2025-02-07 11:30:56,327 [macaron.slsa_analyzer.analyzer:__init__:99] [INFO] The following checks will be run: ['mcn_provenance_available_1', 'mcn_version_control_system_1', 'mcn_detect_malicious_metadata_1', 'mcn_provenance_derived_commit_1', 'mcn_provenance_level_three_1', 'mcn_build_tool_1', 'mcn_provenance_derived_repo_1', 'mcn_build_as_code_1', 'mcn_provenance_expectation_1', 'mcn_find_artifact_pipeline_1', 'mcn_build_script_1', 'mcn_scm_authenticity_1', 'mcn_provenance_verified_1', 'mcn_build_service_1', 'mcn_trusted_builder_level_three_1', 'mcn_provenance_witness_level_one_1']
2025-02-07 11:30:56,387 [macaron.repo_finder.provenance_finder:find_provenance:58] [DEBUG] Seeking provenance of: pkg:pypi/tensorflow
2025-02-07 11:30:56,388 [macaron.repo_finder.provenance_finder:find_provenance:82] [DEBUG] Provenance finding not supported for PURL type: pypi
2025-02-07 11:30:56,388 [macaron.repo_finder.repo_finder:to_repo_path:153] [INFO] The PURL type of pkg:pypi/tensorflow is not valid as a repository type.
2025-02-07 11:30:56,388 [macaron.repo_finder.repo_finder:find_repo:103] [DEBUG] Analyzing pkg:pypi/tensorflow with Repo Finder: <class 'macaron.repo_finder.repo_finder_deps_dev.DepsDevRepoFinder'>
2025-02-07 11:30:56,388 [macaron.util:send_get_http_raw:160] [DEBUG] GET - https://api.deps.dev/v3alpha/purl/pkg%3Apypi%2Ftensorflow
2025-02-07 11:30:58,061 [macaron.util:send_get_http_raw:160] [DEBUG] GET - https://api.deps.dev/v3alpha/purl/pkg%3Apypi%2Ftensorflow%402.9.3
2025-02-07 11:30:58,694 [macaron.repo_finder.repo_finder_deps_dev:_read_json:228] [DEBUG] Could not extract 'version' or 'links' from deps.dev response.
2025-02-07 11:30:58,694 [macaron.repo_finder.repo_finder_deps_dev:find_repo:69] [DEBUG] Failed to extract repository URLs from json data: pkg:pypi/tensorflow
2025-02-07 11:30:58,694 [macaron.repo_finder.repo_finder:find_repo:110] [DEBUG] Could not find repo for PURL: pkg:pypi/tensorflow
2025-02-07 11:30:58,695 [macaron.util:send_get_http_raw:160] [DEBUG] GET - https://api.deps.dev/v3alpha/purl/pkg%3Apypi%2Ftensorflow
2025-02-07 11:30:59,308 [macaron.repo_finder.repo_finder:get_latest_purl_if_different:303] [DEBUG] Found new version of PURL: pkg:pypi/tensorflow@2.9.3
2025-02-07 11:30:59,309 [macaron.util:send_get_http_raw:160] [DEBUG] GET - https://api.deps.dev/v3alpha/purl/pkg%3Apypi%2Ftensorflow%402.9.3
2025-02-07 11:30:59,919 [macaron.repo_finder.repo_finder_deps_dev:_read_json:228] [DEBUG] Could not extract 'version' or 'links' from deps.dev response.
2025-02-07 11:30:59,919 [macaron.repo_finder.repo_finder_deps_dev:find_repo:69] [DEBUG] Failed to extract repository URLs from json data: pkg:pypi/tensorflow@2.9.3
2025-02-07 11:30:59,919 [macaron.repo_finder.repo_finder:find_repo:118] [DEBUG] Could not find repo from latest version of PURL: pkg:pypi/tensorflow@2.9.3

Source code links can be found by following the "Download" page on each project's PyPI page. For example, https://pypi.org/project/torch has a link under "Download" to its GitHub repository: https://github.com/pytorch/pytorch/tags. Similarly for tensorflow at https://pypi.org/project/tensorflow/, where "Download" links to https://github.com/tensorflow/tensorflow/tags.

Metadata

Metadata

Assignees

Labels

repository_finderThe issues related to the repository finder

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions