Skip to content

Convert SCP-style URLs (no explicit scheme) into proper SSH URLs #1061

Closed
Listener430 wants to merge 79 commits intomainfrom
DEV-2964_ssh_scp
Closed

Convert SCP-style URLs (no explicit scheme) into proper SSH URLs #1061
Listener430 wants to merge 79 commits intomainfrom
DEV-2964_ssh_scp

Conversation

@Listener430
Copy link
Collaborator

@Listener430 Listener430 commented Feb 13, 2025

what

Done:

  1. Sometimes vendoring urls are provided in a non-standard, SCP-style Git URLs formt which omits a scheme and use a colon for separation. In order Go’s URL parser can process them, they have to be converted into fully qualified URLs (using SSH or HTTPS).
  2. Vendoring now honors tokens for Gitlab and Bitbucket for https vendoring
  3. Masking of sensative data in debug statements in Custom Detector

This is a spin off of #984 that futher extends custom detector logic

Testing

Use this to run only test cases relevant for this PR
$ go test -v -run '^TestCLICommands/(atmos_vendor_pull_using_SSH|atmos_vendor_pull_with_custom_detector_and_handling_credentials_leakage)$' github.com/cloudposse/atmos/tests

non-standard SCP-style links handling
github ssh vendor pull

Token injections were tested wtih bitbucket and gitlab (http) for private and public repos + ssh vendoring for both.
Listing them here as there are no dedicated tests/repos available for testing at bitbucket/gitlab.

gitlab over ssh private repo
gitlab over https private repo with a token
bitbucket public repo over ssh
bitbucket private repo over ssh
bitbucket https public repo with token set and no token set works
bitbucket https private repo
gitlab over https public repo no auth

why

  1. Links without explicit scheme were indication were not handled correctly, e.g. this one failed
    git::git@github.com:cloudposse/terraform-null-label.git?ref={{.Version}}
  2. credentials for http vendoring were read from the token only for github, but not fot bitbucket and gitlab

references

Summary by CodeRabbit

  • New Features

    • Expanded support for Git repositories by enhancing URL detection and secure token injection for GitHub, Bitbucket, and GitLab.
    • Introduced new YAML configuration options for vendoring operations, offering flexible settings for both SSH and HTTPS protocols.
    • Added a new method for masking sensitive authentication credentials in URLs.
  • Documentation

    • Updated guides to provide clear instructions on SSH/HTTPS vendoring, authentication, and log handling.
  • Refactor

    • Improved logging and error messaging to enhance observability while ensuring sensitive credential data remains masked.
  • Bug Fixes

    • Enhanced error handling and logging for the vendoring process to improve user experience and traceability.

@Listener430 Listener430 added the enhancement New feature or request label Feb 13, 2025
@Listener430 Listener430 requested a review from osterman February 13, 2025 07:23
@Listener430 Listener430 self-assigned this Feb 13, 2025
@Listener430 Listener430 requested a review from a team as a code owner February 13, 2025 07:23
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Feb 13, 2025

📝 Walkthrough

Walkthrough

This pull request refactors the Git detection and vendoring process within Atmos. It renames and restructures the custom Git detector to support multiple hosting services, adds new helper methods for URL scheme validation, normalization, and token injection, and introduces a custom getter for post-clone cleanup. Several YAML configuration files, tests, and documentation updates are also included to better support SSH/HTTPS vendoring, new environment variables for Bitbucket and GitLab, and ensure secure handling of credentials.

Changes

File(s) Summary of Changes
internal/exec/go_getter_utils.go Renamed CustomGitHubDetector to CustomGitDetector; added methods (ensureScheme, rewriteSCPURL, normalizePath, injectToken, resolveToken, adjustSubdir); introduced CustomGitGetter for symlink removal; enhanced logging and error handling.
tests/snapshots/TestCLICommands_*_stderr.golden Added extensive debug, info, and warning logs documenting the vendoring process, custom token injection, and TTY detection across multiple CLI output logs.
tests/fixtures/scenarios/vendor-pulls-ssh/*.{yaml}
tests/fixtures/scenarios/vendor-creds-sanitize/*.{yaml}
Updated and added YAML config files to introduce new settings for base paths, token injection, components/stacks, and deployment scenarios for Atmos vendoring.
website/docs/cli/configuration/configuration.mdx Updated documentation with new environment variables (ATMOS_BITBUCKET_TOKEN, ATMOS_BITBUCKET_USERNAME, ATMOS_GITLAB_TOKEN) for Bitbucket and GitLab authentication.
website/docs/cli/commands/vendor/vendor-pull.mdx Expanded vendoring documentation to cover SSH and HTTPS approaches, detailing URL resolution, authentication methods, and token injection logic.
pkg/utils/url_utils.go Introduced the new function MaskBasicAuth to mask basic authentication credentials in URLs.
tests/test-cases/vendoring-ssh-dryrun.yaml Added new test cases for a dry-run of SSH vendoring and ensuring that sensitive credentials do not appear in logs.
internal/exec/go_getter_utils_test.go Added a comprehensive suite of unit tests covering URI validation, scheme enforcement, SCP URL rewriting, path normalization, token injection/resolution, symlink removal, file downloading, and detector registration.
internal/exec/{validate_stacks.go, vendor_model.go, yaml_func_include.go, vendor_component_utils.go} Modified function calls to pass atmosConfig as a pointer instead of a value/dereferenced copy, affecting configuration mutability in various vendoring functions.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant Detector as CustomGitDetector
    participant Parser as Git URL Parser
    participant Resolver as Token Resolver
    participant Getter as CustomGitGetter

    Client->>Detector: Detect(src)
    Detector->>Detector: ensureScheme(src)
    Detector->>Detector: rewriteSCPURL(src)
    Detector->>Parser: Parse & normalize URL
    Detector->>Resolver: Resolve token for host
    Resolver-->>Detector: Return token & username
    Detector->>Detector: injectToken(validated URL, host)
    Detector-->>Client: Return final URL, validity flag, error

    Client->>Getter: Get(dst, final URL)
    Getter->>Getter: Perform clone operation
    Getter->>Getter: removeSymlinks(root)
    Getter-->>Client: Return clone operation status
Loading

Possibly related PRs

Suggested labels

patch

Warning

There were issues while running some tools. Please review the errors and either fix the tool’s configuration or disable the tool if it’s a critical failure.

🔧 golangci-lint (1.62.2)

Error: can't load config: the Go language version (go1.23) used to build golangci-lint is lower than the targeted Go version (1.24.0)
Failed executing command with error: can't load config: the Go language version (go1.23) used to build golangci-lint is lower than the targeted Go version (1.24.0)

Tip

⚡🧪 Multi-step agentic review comment chat (experimental)
  • We're introducing multi-step agentic chat in review comments. This experimental feature enhances review discussions with the CodeRabbit agentic chat by enabling advanced interactions, including the ability to create pull requests directly from comments.
    - To enable this feature, set early_access to true under in the settings.
✨ Finishing Touches
  • 📝 Generate Docstrings

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary or @auto-summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai or @auto-title anywhere in the PR title to generate the title automatically.

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (5)
internal/exec/go_getter_utils.go (5)

15-15: Consider using a more descriptive alias
Short aliases like l might hamper clarity. Renaming to log or logger could improve readability.


70-70: Suggest clarifying the field name
source could be named more explicitly, such as sourceURI or originalSource, for improved code clarity.


90-112: Add optional support for custom ports
SCP-style URLs sometimes specify a custom port (e.g., git@host:port/repo). The current regex won’t match those.

- scpPattern := regexp.MustCompile(`^(([\w.-]+)@)?([\w.-]+\.[\w.-]+):([\w./-]+)(\.git)?(.*)$`)
+ scpPattern := regexp.MustCompile(`^(([\w.-]+)@)?([\w.-]+\.[\w.-]+)(:[0-9]+)?:([\w./-]+)(\.git)?(.*)$`)

136-138: Convert TBC comment into a TODO
Documenting pending enhancements is good. Consider adding a // TODO: or opening an issue.

Would you like me to open an issue for broadening token injection support?


139-164: Ensure consistent config toggles
Only GitHub token injection is governed by InjectGithubToken. Consider adding similar toggles for Bitbucket and GitLab for uniformity.

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 161c074 and a43fd72.

📒 Files selected for processing (1)
  • internal/exec/go_getter_utils.go (3 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (17)
  • GitHub Check: [mock-macos] tests/fixtures/scenarios/complete
  • GitHub Check: [mock-macos] examples/demo-vendoring
  • GitHub Check: [mock-macos] examples/demo-context
  • GitHub Check: [mock-macos] examples/demo-component-versions
  • GitHub Check: [mock-macos] examples/demo-atlantis
  • GitHub Check: [mock-windows] examples/demo-context
  • GitHub Check: [mock-windows] examples/demo-component-versions
  • GitHub Check: [mock-windows] examples/demo-atlantis
  • GitHub Check: [mock-linux] tests/fixtures/scenarios/complete
  • GitHub Check: [mock-linux] examples/demo-vendoring
  • GitHub Check: [mock-linux] examples/demo-context
  • GitHub Check: Acceptance Tests (windows-latest, windows)
  • GitHub Check: Acceptance Tests (ubuntu-latest, linux)
  • GitHub Check: Docker Lint
  • GitHub Check: [k3s] demo-helmfile
  • GitHub Check: [localstack] demo-localstack
  • GitHub Check: Summary
🔇 Additional comments (11)
internal/exec/go_getter_utils.go (11)

11-11: Looks good
No immediate concerns. The regexp import is necessary for the new SCP-style detection.


61-61: Confirmed
Adding git::ssh to the list of valid schemes aligns with go-getter usage.


66-68: Renaming is consistent
This rename clarifies support for multiple Git hosting platforms.


81-88: Excellent documentation
The inline comments clearly explain how SCP-style URLs are handled.


115-115: Duplicate concern: potential credential exposure
Similar to the previous logging of src, sensitive info may be leaked.


123-126: Clear error handling
Returning an error when no SSH agent is available is straightforward.


128-133: Non-standard host scenario well-handled
Skipping token injection for unrecognized hosts is logical.


166-175: Token injection logic is robust
Injecting the token only if credentials aren’t already present is correct.


177-185: Subdir detection
Appending //. for top-level repos is a known go-getter approach. Looks good.


187-197: Default shallow clone
Specifying depth=1 improves performance but may break use cases needing full history. Confirm this suits your workflows.


208-208: Registering the new CustomGitDetector
Ensuring the custom detector runs first is correct.

coderabbitai[bot]
coderabbitai bot previously approved these changes Feb 13, 2025
@osterman
Copy link
Member

To test vendoring of SSH style URLs, without SSH key, test instead —-dry-run with log level debug, and a snapshot. Note, tokens are not valid for SSH authentication.

@osterman
Copy link
Member

Windows tests are failing:

=== RUN   TestCLICommands/atmos_vendor_pull_(no_tty)
    cli_test.go:901: Stderr diff mismatch for "D:\\a\\atmos\\atmos\\tests\\snapshots\\TestCLICommands_atmos_vendor_pull_(no_tty).stdout.golden":
--- expected
+++ actual
@@ -1,7 +1,12 @@
 INFO Vendoring from 'vendor.yaml'
 WARN No TTY detected. Falling back to basic output. This can happen when no terminal is attached or when commands are pipelined.
-INFO ✓ github/stargazers (main)
-INFO ✓ weather (main)
-INFO ✓ ipinfo (main)
+ERRO Failed to vendor github/stargazers: error : failed to download package: subdir "examples%5Cdemo-library%5Cgithub%5Cstargazers" not found
+INFO x github/stargazers (main)
+ERRO Failed to vendor weather: error : failed to download package: subdir "examples%5Cdemo-library%5Cweather" not found
+INFO x weather (main)
+ERRO Failed to vendor ipinfo: error : failed to download package: subdir "examples%5Cdemo-library%5Cipinfo" not found
+INFO x ipinfo (main)
+INFO Vendored 0 components. Failed to vendor 3 components.
+

@Listener430
Copy link
Collaborator Author

To test vendoring of SSH style URLs, without SSH key, test instead —-dry-run with log level debug, and a snapshot. Note, tokens are not valid for SSH authentication.

Have added the test, but it is in disabled state now as current dry run implementation in internal\exec\vendor_component_utils.go exits before hitting custom detectors and go getters. Let me know there are any ojectiosn to adjust vendor_component_utils.go code for the dry run to operate differently.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
tests/test-cases/demo-vendoring.yaml (1)

58-59: Clean up YAML formatting.

Fix the following formatting issues:

  • Line 58: Remove trailing spaces
  • Line 59: Add newline at end of file
-        - "No SSH authentication method found"  
-      exit_code: 0
\ No newline at end of file
+        - "No SSH authentication method found"
+      exit_code: 0
+
🧰 Tools
🪛 YAMLlint (1.35.1)

[error] 58-58: trailing spaces

(trailing-spaces)


[error] 59-59: no new line character at the end of file

(new-line-at-end-of-file)

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between a43fd72 and ef4a300.

📒 Files selected for processing (3)
  • internal/exec/go_getter_utils.go (3 hunks)
  • tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh.stderr.golden (1 hunks)
  • tests/test-cases/demo-vendoring.yaml (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • internal/exec/go_getter_utils.go
🧰 Additional context used
🪛 YAMLlint (1.35.1)
tests/test-cases/demo-vendoring.yaml

[error] 58-58: trailing spaces

(trailing-spaces)


[error] 59-59: no new line character at the end of file

(new-line-at-end-of-file)

⏰ Context from checks skipped due to timeout of 90000ms (2)
  • GitHub Check: Build (windows-latest, windows)
  • GitHub Check: Summary
🔇 Additional comments (1)
tests/test-cases/demo-vendoring.yaml (1)

43-59: LGTM! Well-structured test case for SSH URL handling.

The test case effectively validates:

  • SSH-style URL handling
  • Dry-run behavior
  • Error messaging for missing SSH authentication
  • Debug logging output

This aligns perfectly with the PR objectives for SCP-style URL support.

🧰 Tools
🪛 YAMLlint (1.35.1)

[error] 58-58: trailing spaces

(trailing-spaces)


[error] 59-59: no new line character at the end of file

(new-line-at-end-of-file)

coderabbitai[bot]
coderabbitai bot previously approved these changes Feb 14, 2025
coderabbitai[bot]
coderabbitai bot previously approved these changes Mar 12, 2025
coderabbitai[bot]
coderabbitai bot previously approved these changes Mar 13, 2025
coderabbitai[bot]
coderabbitai bot previously approved these changes Mar 13, 2025
Co-authored-by: Erik Osterman (CEO @ Cloud Posse) <erik@cloudposse.com>
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (5)
website/docs/cli/commands/vendor/vendor-pull.mdx (5)

111-118: Clarify SSH Vendoring Behavior
The introduction clearly explains SSH support for non-public repositories. To further guide users, consider explicitly noting that while SSH is convenient for local development, credentials (such as tokens) are not injected for SSH sources. This distinction reinforces why automated environments should prefer HTTPS.


128-133: URL Rewriting Clarity
The example showing the rewritten URL is concise and correct. Consider adding a brief explanation—perhaps as an inline note—on why the ?depth=1 parameter is appended (e.g., to minimize clone time and optimize performance).


172-203: Authentication & Token Usage Details
The detailed tables for GitHub, Bitbucket, and GitLab authentication priorities are comprehensive and invaluable. As a minor improvement, consider using typographic quotes (as hinted by the static analysis) for values like "x-access-token" and "x-token-auth" to enhance readability and consistency.

🧰 Tools
🪛 LanguageTool

[typographical] ~189-~189: Consider using a typographic close quote here.
Context: ...e and does not accept dummy values like "x-access-token". | **Default Username ...

(EN_QUOTES)


[typographical] ~194-~194: Consider using a typographic close quote here.
Context: ...s set, it is used. - If neither is set, "x-token-auth" is used. ### GitLab | En...

(EN_QUOTES)


204-211: Git URL Resolution Rules
The rules for resolving Git URLs are clearly defined and align with the PR objectives. One minor nitpick: replacing the plain ellipsis with a typographic ellipsis (…​) in the example could improve stylistic consistency, as suggested by static analysis.

🧰 Tools
🪛 LanguageTool

[style] ~206-~206: Consider using a different verb to strengthen your wording.
Context: ...lved When resolving Git sources, Atmos follows these rules: 1. If a *full HTTPS URL...

(FOLLOW_OBEY)


[style] ~208-~208: Consider using the typographical ellipsis character here instead.
Context: ...1. If a full HTTPS URL is provided (git::https://github.com/...), it is used as-is. No token data is i...

(ELLIPSIS)


222-232: Examples Section
The examples provided are diverse and clearly demonstrate different usage scenarios. To further support users, you might consider including a specific example that shows the transformation of an SCP-style URL into its SSH format.

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 796be4d and 9526276.

📒 Files selected for processing (1)
  • website/docs/cli/commands/vendor/vendor-pull.mdx (1 hunks)
🧰 Additional context used
🪛 LanguageTool
website/docs/cli/commands/vendor/vendor-pull.mdx

[typographical] ~189-~189: Consider using a typographic close quote here.
Context: ...e and does not accept dummy values like "x-access-token". | **Default Username ...

(EN_QUOTES)


[typographical] ~194-~194: Consider using a typographic close quote here.
Context: ...s set, it is used. - If neither is set, "x-token-auth" is used. ### GitLab | En...

(EN_QUOTES)


[style] ~206-~206: Consider using a different verb to strengthen your wording.
Context: ...lved When resolving Git sources, Atmos follows these rules: 1. If a *full HTTPS URL...

(FOLLOW_OBEY)


[style] ~208-~208: Consider using the typographical ellipsis character here instead.
Context: ...1. If a full HTTPS URL is provided (git::https://github.com/...), it is used as-is. No token data is i...

(ELLIPSIS)

⏰ Context from checks skipped due to timeout of 90000ms (6)
  • GitHub Check: Build (windows-latest, windows)
  • GitHub Check: website-deploy-preview
  • GitHub Check: Analyze (go)
  • GitHub Check: Build (ubuntu-latest, linux)
  • GitHub Check: Lint (golangci)
  • GitHub Check: Summary
🔇 Additional comments (9)
website/docs/cli/commands/vendor/vendor-pull.mdx (9)

120-127: SCP-style Sources Explanation
This section effectively illustrates the traditional SCP-style format and its use of a colon as a separator. It might be helpful to add a brief note that Atmos automatically converts these sources into proper SSH URLs behind the scenes.


136-144: Explicit SSH Scheme Handling
This segment aptly differentiates between SCP-style and explicitly defined SSH URLs, noting that explicit ssh:// URLs bypass rewriting. Excellent clarity here.


145-152: Invalid URL Format Notice
The note regarding invalid URLs (e.g., using github.com:... without the proper prefix) is well-articulated and useful. It clearly warns users about go-getter’s misinterpretation of such formats.


153-158: Scheme Default Explanation
The explanation that URLs lacking an explicit scheme default to HTTPS—and trigger automatic credential injection—is clear and straightforward. No changes needed here.


159-171: HTTPS Vendoring Section
This section provides a clear walkthrough with examples on how HTTPS URLs are resolved and transformed (including the addition of ?depth=1). The content is consistent and informative.


212-216: Configuration Reference Note
The note directing users to further details on configuration is succinct and helpful.


218-220: Help Command Tip
The tip advising users to run atmos vendor pull --help is clear and useful.


234-245: Vendoring Manifest Selection Note
The explanation regarding how Atmos chooses between vendor.yaml and component.yaml is detailed and clear.

🧰 Tools
🪛 LanguageTool

[grammar] ~239-~239: The word ‘vendor’ is a noun or an adjective. A verb or adverb is missing or misspelled here, or maybe a comma is missing.
Context: ...componentis not specified, Atmos will vendor all the artifacts defined in thevendo...

(PRP_MD_NN)


[style] ~240-~240: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...efined in the vendor.yaml manifest. If the flag --component is passed in, At...

(ENGLISH_WORD_REPEAT_BEGINNING_RULE)


[grammar] ~240-~240: The word ‘vendor’ is a noun or an adjective. A verb or adverb is missing or misspelled here, or maybe a comma is missing.
Context: ... --component is passed in, Atmos will vendor only that component - If vendor.yaml...

(PRP_MD_NN)


[grammar] ~240-~240: Please add a punctuation mark at the end of paragraph.
Context: ... passed in, Atmos will vendor only that component - If vendor.yaml is not found, Atmos...

(PUNCTUATION_PARAGRAPH_END)


[style] ~242-~242: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...mlmanifest in the component's folder. Ifcomponent.yaml` is not found, an err...

(ENGLISH_WORD_REPEAT_BEGINNING_RULE)


[grammar] ~243-~243: Please add a punctuation mark at the end of paragraph.
Context: ... flag --component is required in this case ::: ## Flags | Flag | Descr...

(PUNCTUATION_PARAGRAPH_END)


247-255: Flags Table
The flags table is well-organized and comprehensively illustrates the available options for the command.

coderabbitai[bot]
coderabbitai bot previously approved these changes Mar 13, 2025
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (3)
website/docs/cli/commands/vendor/vendor-pull.mdx (3)

189-200: Bitbucket Authentication Details with Minor Typographic Suggestion
The Bitbucket section is well structured with a table, clearly listing the environment variables and the default username rules. As a minor improvement, consider using typographic (curly) quotes for the default username "x-token-auth" to align with our style guidelines.

🧰 Tools
🪛 LanguageTool

[typographical] ~196-~196: Consider using a typographic close quote here.
Context: ...e and does not accept dummy values like "x-access-token". | **Default Username ...

(EN_QUOTES)


201-208: GitLab Authentication Section Consistency
This section is clear and mirrors the style used for GitHub and Bitbucket. Similar to the Bitbucket section, you might consider replacing the straight quotes around "oauth2" with typographic quotes for improved readability.

🧰 Tools
🪛 LanguageTool

[typographical] ~201-~201: Consider using a typographic close quote here.
Context: ...s set, it is used. - If neither is set, "x-token-auth" is used. ### GitLab | En...

(EN_QUOTES)


209-217: Refinement of Git URL Resolution Rules
The numbered list explaining how Atmos resolves Git sources is informative. As a nitpick, consider rephrasing the introductory sentence (e.g., "Atmos resolves Git sources using the following rules:") to strengthen clarity. Also, replacing the sequence of three dots (...) in the HTTPS URL example with a typographical ellipsis ( … ) could enhance the presentation per our static analysis hints.

🧰 Tools
🪛 LanguageTool

[style] ~213-~213: Consider using a different verb to strengthen your wording.
Context: ...lved When resolving Git sources, Atmos follows these rules: 1. If a *full HTTPS URL...

(FOLLOW_OBEY)


[style] ~215-~215: Consider using the typographical ellipsis character here instead.
Context: ...1. If a full HTTPS URL is provided (git::https://github.com/...), it is used as-is. No token data is i...

(ELLIPSIS)

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 9526276 and 18f7a50.

📒 Files selected for processing (1)
  • website/docs/cli/commands/vendor/vendor-pull.mdx (1 hunks)
🧰 Additional context used
🪛 LanguageTool
website/docs/cli/commands/vendor/vendor-pull.mdx

[typographical] ~196-~196: Consider using a typographic close quote here.
Context: ...e and does not accept dummy values like "x-access-token". | **Default Username ...

(EN_QUOTES)


[typographical] ~201-~201: Consider using a typographic close quote here.
Context: ...s set, it is used. - If neither is set, "x-token-auth" is used. ### GitLab | En...

(EN_QUOTES)


[style] ~213-~213: Consider using a different verb to strengthen your wording.
Context: ...lved When resolving Git sources, Atmos follows these rules: 1. If a *full HTTPS URL...

(FOLLOW_OBEY)


[style] ~215-~215: Consider using the typographical ellipsis character here instead.
Context: ...1. If a full HTTPS URL is provided (git::https://github.com/...), it is used as-is. No token data is i...

(ELLIPSIS)

⏰ Context from checks skipped due to timeout of 90000ms (3)
  • GitHub Check: Build (windows-latest, windows)
  • GitHub Check: Analyze (go)
  • GitHub Check: Summary
🔇 Additional comments (8)
website/docs/cli/commands/vendor/vendor-pull.mdx (8)

111-118: Addition of the SSH Vendoring Introduction
The new section clearly explains that Atmos supports SSH-based vendoring for non-public Git repositories and establishes the context for local development. It provides a concise overview along with a helpful tip for automated systems.


120-127: Clear Explanation of SCP-style Sources
This section clearly details how traditional SCP-style sources (using a colon) are supported and provides a relevant example showing the expected syntax. The example helps readers understand the input format.


128-133: Documentation of URL Rewriting Behavior
The explanation of how Atmos rewrites the SCP-style URL into a proper SSH URL (adding the depth=1 parameter) is well presented. Ensure that the significance of depth=1 is documented elsewhere if not already.


134-134: Default Username Injection for GitHub
The note that Atmos automatically injects the default username git when none is provided for github.com is clear and aligns with our design.


135-143: Explicit SSH Scheme Usage
The section explaining that URLs already starting with the ssh:// scheme remain unmodified is straightforward and provides a concrete example. This helps clarify behavior for users preferring explicit schemes.


145-158: Important Notes on Invalid URL Formats
The warnings about using an invalid SCP-style URL (without the proper prefix) and about the default HTTPS behavior are precise and useful. This section efficiently prevents common pitfalls related to URL formatting.


159-171: HTTPS Vendoring Clarification
The explanation of how repository names without an explicit scheme are automatically resolved to HTTPS with credential injection is clear. The before-and-after examples effectively illustrate the transformation.


172-188: Comprehensive Authentication Instructions for GitHub
The GitHub authentication section uses a clean definition list to outline the priority of environment variables. It succinctly covers token usage and the default username strategy.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (3)
website/docs/cli/commands/vendor/vendor-pull.mdx (3)

111-159: New SSH Vendoring Section Clarity and Completeness
The newly added "Vendoring from SSH" section is well structured. It clearly differentiates between SCP-style sources and explicit SSH sources and provides concrete examples of URL rewriting. One suggestion is to include a brief note about SSH key requirements for local development to help users understand potential caveats when using SSH in automated pipelines.


216-216: Wording Suggestion for Git Source Resolution
Consider revising the sentence "When resolving Git sources, Atmos follows these rules:" to a more active voice—perhaps "Atmos resolves Git sources using the following rules:"—to strengthen the tone.

🧰 Tools
🪛 LanguageTool

[style] ~216-~216: Consider using a different verb to strengthen your wording.
Context: ...lved When resolving Git sources, Atmos follows these rules: 1. If a *full HTTPS URL...

(FOLLOW_OBEY)


218-218: Typographical Ellipsis
The example URL uses three consecutive dots (...). For improved readability and typographical polish, consider replacing them with the typographical ellipsis character ().

🧰 Tools
🪛 LanguageTool

[style] ~218-~218: Consider using the typographical ellipsis character here instead.
Context: ...1. If a full HTTPS URL is provided (git::https://github.com/...), it is used as-is. No token data is i...

(ELLIPSIS)

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 18f7a50 and a48376a.

📒 Files selected for processing (2)
  • internal/exec/vendor_utils.go (2 hunks)
  • website/docs/cli/commands/vendor/vendor-pull.mdx (1 hunks)
🧰 Additional context used
🪛 LanguageTool
website/docs/cli/commands/vendor/vendor-pull.mdx

[style] ~216-~216: Consider using a different verb to strengthen your wording.
Context: ...lved When resolving Git sources, Atmos follows these rules: 1. If a *full HTTPS URL...

(FOLLOW_OBEY)


[style] ~218-~218: Consider using the typographical ellipsis character here instead.
Context: ...1. If a full HTTPS URL is provided (git::https://github.com/...), it is used as-is. No token data is i...

(ELLIPSIS)

⏰ Context from checks skipped due to timeout of 90000ms (6)
  • GitHub Check: Build (windows-latest, windows)
  • GitHub Check: Build (ubuntu-latest, linux)
  • GitHub Check: Analyze (go)
  • GitHub Check: Lint (golangci)
  • GitHub Check: website-deploy-preview
  • GitHub Check: Summary
🔇 Additional comments (2)
internal/exec/vendor_utils.go (2)

22-23: Good addition of dedicated stderr logger.

Creating a separate logger for stderr output is a clean approach that helps maintain separation between different output streams. This will keep stdout clean of detailed debug messages during vendoring operations, making it easier to parse program output in scripts or CI/CD pipelines.


514-514: Updated log statement to use the new StderrLogger.

The log statement has been correctly updated to use the new StderrLogger.Debug instead of log.Debug, which aligns with the goal of separating detailed vendoring messages to stderr.

coderabbitai[bot]
coderabbitai bot previously approved these changes Mar 13, 2025
coderabbitai[bot]
coderabbitai bot previously approved these changes Mar 13, 2025
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (2)
pkg/utils/url_utils_test.go (2)

43-49: Proper error handling test.

This test correctly validates that the function returns an error when given an invalid URL format. Consider adding a check for the specific error message to ensure the exact expected error is returned.

 func TestMaskBasicAuth_InvalidURL(t *testing.T) {
 	input := "://invalid-url"
 	_, err := MaskBasicAuth(input)
 	if err == nil {
 		t.Errorf("Expected error for invalid URL, but got none")
+	} else if !strings.Contains(err.Error(), "parse") {
+		t.Errorf("Expected URL parsing error, got: %v", err)
 	}
 }

Don't forget to add strings to your imports if you implement this suggestion.


1-50: Consider adding test cases for SCP-style URLs.

Since the PR's main objective is to handle SCP-style URLs (like git@github.com:user/repo.git), consider adding specific test cases that verify the MaskBasicAuth function correctly handles these formats.

func TestMaskBasicAuth_SCPStyleURL(t *testing.T) {
	input := "git@github.com:user/repo.git"
	expected := "git@github.com:user/repo.git" // or whatever the expected masking should be
	masked, err := MaskBasicAuth(input)
	if err != nil {
		t.Fatalf("Expected no error, got %v", err)
	}
	if masked != expected {
		t.Errorf("Expected masked URL %q, got %q", expected, masked)
	}
}
📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 7d5731d and c8b6554.

📒 Files selected for processing (1)
  • pkg/utils/url_utils_test.go (1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (7)
  • GitHub Check: Build (macos-latest, macos)
  • GitHub Check: Build (windows-latest, windows)
  • GitHub Check: Build (ubuntu-latest, linux)
  • GitHub Check: Lint (golangci)
  • GitHub Check: website-deploy-preview
  • GitHub Check: Analyze (go)
  • GitHub Check: Summary
🔇 Additional comments (4)
pkg/utils/url_utils_test.go (4)

1-6: Good package structure and imports.

The test file is properly organized with the correct package name matching the implementation. The imports are kept minimal with just the testing package, which is sufficient for these tests.


7-17: Well-structured test for username and password masking.

This test effectively validates that URLs with both username and password have their credentials masked. The test includes proper error handling and clear assertion messages.


19-29: Good coverage for username-only scenario.

The test correctly handles the case where a URL contains only a username without a password, ensuring it gets properly masked.


31-41: Appropriate test for URLs without credentials.

This test verifies that URLs without any authentication information remain unchanged, which is an important edge case to verify.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

minor New features that do not break anything

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Vendor issue when using SSH-formatted Git URLs

3 participants