Skip to content

feat: add support for cloning GitLab repositories#316

Merged
nathanwn merged 19 commits into
stagingfrom
301-clone-gitlab
Jun 22, 2023
Merged

feat: add support for cloning GitLab repositories#316
nathanwn merged 19 commits into
stagingfrom
301-clone-gitlab

Conversation

@nathanwn

@nathanwn nathanwn commented Jun 15, 2023

Copy link
Copy Markdown
Member

Closes #301.

Description

This PR adds support for cloning GitLab repositories in Macaron. Specifically, GitLab repositories can now be cloned as either main targets or dependencies in a macaron analyze run.

This feature comes with the following additions/changes:

.ini configuration

  • The old [git] section in the .ini configuration will now be replaced by the new [git_service.*]  sections.
  • Each [git_service] section corresponds to a different git service with a domain key. This domain is used to determine if a Git service hosts a repo given the repo's Git URL.
  • If the domain key is not found in any [git_service.*]  section, the command macaron analyze  will error right away before any analysis is done.
  • Users can configure 2 different GitLab services in the [git_service.gitlab.*]  sections: a public one and a private one. The domain of the public GitLab service is gitlab.com by default.

Access Token Environment Variables

An access token for the public GitLab service is not required when it is enabled. However, an access token for the private GitLab service is always required when it is enabled.

We accept two environment variables storing the access tokens that correspond to the two GitLab services:

  • MCN_PUBLIC_GITLAB_TOKEN  for the public GitLab service.
  • MCN_PRIVATE_GITLAB_TOKEN  for the private GitLab service.

The MCN_PRIVATE_GITLAB_TOKEN environment variable must be set when the [git_service.gitlab.private] section is enabled in the .ini config. If this is not the case, the command macaron analyze  will error right away before any analysis is done.

@nathanwn nathanwn linked an issue Jun 15, 2023 that may be closed by this pull request
@oracle-contributor-agreement oracle-contributor-agreement Bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label Jun 15, 2023
@nathanwn nathanwn changed the base branch from main to staging June 15, 2023 05:44
Comment thread src/macaron/slsa_analyzer/git_service/__init__.py Outdated
Comment thread src/macaron/slsa_analyzer/git_service/__init__.py Outdated
Comment thread src/macaron/errors.py Outdated
nathanwn added 4 commits June 19, 2023 11:57
Signed-off-by: Nathan Nguyen <nathan.nguyen@oracle.com>
Signed-off-by: Nathan Nguyen <nathan.nguyen@oracle.com>
Signed-off-by: Nathan Nguyen <nathan.nguyen@oracle.com>
Signed-off-by: Nathan Nguyen <nathan.nguyen@oracle.com>
nathanwn added 3 commits June 19, 2023 18:26
Signed-off-by: Nathan Nguyen <nathan.nguyen@oracle.com>
Signed-off-by: Nathan Nguyen <nathan.nguyen@oracle.com>
Signed-off-by: Nathan Nguyen <nathan.nguyen@oracle.com>
@nathanwn nathanwn self-assigned this Jun 19, 2023
@nathanwn nathanwn marked this pull request as ready for review June 19, 2023 10:15
Signed-off-by: Nathan Nguyen <nathan.nguyen@oracle.com>
Comment thread src/macaron/__main__.py Outdated
Comment thread src/macaron/slsa_analyzer/git_service/base_git_service.py Outdated
Comment thread src/macaron/slsa_analyzer/git_service/gitlab.py Outdated
Comment thread src/macaron/slsa_analyzer/git_service/gitlab.py Outdated
Comment thread src/macaron/slsa_analyzer/git_service/gitlab.py Outdated
Comment thread src/macaron/slsa_analyzer/git_service/gitlab.py
Comment thread tests/slsa_analyzer/test_git_url.py Outdated
@behnazh-w

Copy link
Copy Markdown
Member

@nathanwn I know you have opened a ticket for integration tests. But please add at least one test for a public GitLab repository in this PR.

nathanwn added 2 commits June 20, 2023 15:36
Signed-off-by: Nathan Nguyen <nathan.nguyen@oracle.com>
Signed-off-by: Nathan Nguyen <nathan.nguyen@oracle.com>
Signed-off-by: Nathan Nguyen <nathan.nguyen@oracle.com>
Comment thread src/macaron/slsa_analyzer/git_service/base_git_service.py Outdated
Comment thread src/macaron/slsa_analyzer/git_url.py
Comment thread src/macaron/slsa_analyzer/git_service/gitlab.py
nathanwn added 2 commits June 21, 2023 10:58
Signed-off-by: Nathan Nguyen <nathan.nguyen@oracle.com>
Signed-off-by: Nathan Nguyen <nathan.nguyen@oracle.com>
@nathanwn

Copy link
Copy Markdown
Member Author

@nathanwn I know you have opened a ticket for integration tests. But please add at least one test for a public GitLab repository in this PR.

Integration test added in dbb2748.

nathanwn added 4 commits June 21, 2023 11:33
Signed-off-by: Nathan Nguyen <nathan.nguyen@oracle.com>
Signed-off-by: Nathan Nguyen <nathan.nguyen@oracle.com>
Signed-off-by: Nathan Nguyen <nathan.nguyen@oracle.com>
Signed-off-by: Nathan Nguyen <nathan.nguyen@oracle.com>

@tromai tromai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread tests/slsa_analyzer/test_git_url.py
Comment thread tests/slsa_analyzer/test_git_url.py Outdated
tmp_path: Path,
) -> None:
"""Test the get allowed git service domains function, in multi-config files scenario."""
default_filepath = tmp_path / "default.ini"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use os.path.join here and for all the path constructions.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder why couldn't we use the slash operator here as it is supported by pathlib - https://docs.python.org/3/library/pathlib.html#operators. The tmp_path is of type pathlib.Path so I think it's okay to use.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's to be consistent with the rest of the codebase and to be more explicit by using a function signature.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be fixed in 9425dfe.

Comment thread tests/slsa_analyzer/test_git_url.py
…ults config object

Signed-off-by: Nathan Nguyen <nathan.nguyen@oracle.com>
Comment thread tests/slsa_analyzer/test_git_url.py Outdated
Signed-off-by: Nathan Nguyen <nathan.nguyen@oracle.com>
@nathanwn nathanwn merged commit 073c08f into staging Jun 22, 2023
@nathanwn nathanwn deleted the 301-clone-gitlab branch June 22, 2023 04:43
@nathanwn nathanwn added the git_service The issues related to a Git service, such as Bitbucket label Jul 8, 2023
art1f1c3R pushed a commit that referenced this pull request Nov 29, 2024
Signed-off-by: Nathan Nguyen <nathan.nguyen@oracle.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

git_service The issues related to a Git service, such as Bitbucket OCA Verified All contributors have signed the Oracle Contributor Agreement.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for cloning from GitLab

3 participants