Skip to content

fix: Implement retry logic for image resolution processes#11

Merged
chmouel merged 1 commit intopipelines-as-code:mainfrom
chmouel:improve-registry-check-and-ko-retry
Feb 26, 2026
Merged

fix: Implement retry logic for image resolution processes#11
chmouel merged 1 commit intopipelines-as-code:mainfrom
chmouel:improve-registry-check-and-ko-retry

Conversation

@chmouel
Copy link
Copy Markdown
Collaborator

@chmouel chmouel commented Feb 26, 2026

Added a verification loop to check registry connectivity before starting the resolution process. Included a retry mechanism for the ko resolve command to mitigate failures caused by intermittent network issues or temporary registry unavailability. Prevented the script from proceeding with invalid configurations by ensuring it returned an error code upon repeated resolution failure.

Added a verification loop to check registry connectivity before starting
the resolution process. Included a retry mechanism for the ko resolve
command to mitigate failures caused by intermittent network issues or
temporary registry unavailability. Prevented the script from proceeding
with invalid configurations by ensuring it returned an error code upon
repeated resolution failure.

Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
@gemini-code-assist
Copy link
Copy Markdown

Summary of Changes

Hello @chmouel, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the robustness of the image resolution process by introducing proactive registry connectivity verification and a retry mechanism for the 'ko resolve' command. These changes aim to improve the reliability of deployments by mitigating failures caused by temporary network instability or registry unavailability, and by providing clearer error signaling when critical steps cannot be completed.

Highlights

  • Registry Connectivity Check: Implemented a pre-check using 'curl' to verify the accessibility of the configured image registry before attempting image resolution.
  • ko resolve Retry Mechanism: Added a retry loop for the 'ko resolve' command, allowing it to reattempt resolution up to 3 times with a 15-second delay between failures to handle transient issues.
  • Robust Error Handling: Ensured the script exits with an error code ('return 1') if 'ko resolve' consistently fails after all retry attempts, preventing continuation with an invalid state.
Changelog
  • startpaac
    • Added a pre-check for registry connectivity using 'curl'.
    • Implemented a retry loop for the 'ko resolve' command.
    • Modified the script to return an error code if 'ko resolve' fails after multiple retries.
Activity
  • No specific activity (comments, reviews, progress) has been recorded for this pull request yet.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@qodo-code-review
Copy link
Copy Markdown

ⓘ You are approaching your monthly quota for Qodo. Upgrade your plan

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 No relevant tests
🔒 Security concerns

TLS verification bypass:
the use of curl -k disables certificate verification when checking https://${REGISTRY}/v2/. This weakens transport security and could allow a MITM to spoof registry availability during the preflight check. Consider removing -k or gating it behind an explicit “insecure registry” configuration.

⚡ Recommended focus areas for review

TLS Bypass

The registry connectivity check uses curl -k, which disables TLS certificate verification. This can mask real connectivity/misconfiguration problems and may allow man-in-the-middle scenarios during the preflight check. Consider making TLS verification configurable (or defaulting to verification) and surfacing actionable errors when verification fails.

# Verify registry is accessible before ko resolve
echo "Verifying registry ${REGISTRY} is accessible..."
local reg_ok=false
for ((i = 1; i <= 5; i++)); do
  if curl -o/dev/null --fail -k -s "https://${REGISTRY}/v2/"; then
    reg_ok=true
    break
  fi
  echo "Registry not ready, retrying in 5s ($i/5)..."
  sleep 5
done
if [[ ${reg_ok} != true ]]; then
  echo "Warning: Registry ${REGISTRY} not responding, proceeding anyway..."
fi
Retry Robustness

The retry loops should ensure curl and ko resolve fail fast and predictably. Consider adding explicit timeouts for curl (connect/max time) to avoid hanging, and potentially validating error types (e.g., network vs. config) before retrying to prevent repeated retries on deterministic failures.

# Verify registry is accessible before ko resolve
echo "Verifying registry ${REGISTRY} is accessible..."
local reg_ok=false
for ((i = 1; i <= 5; i++)); do
  if curl -o/dev/null --fail -k -s "https://${REGISTRY}/v2/"; then
    reg_ok=true
    break
  fi
  echo "Registry not ready, retrying in 5s ($i/5)..."
  sleep 5
done
if [[ ${reg_ok} != true ]]; then
  echo "Warning: Registry ${REGISTRY} not responding, proceeding anyway..."
fi

# ko resolve with retry logic for intermittent registry failures
local ko_success=false
local ko_max_retries=3
for ((attempt = 1; attempt <= ko_max_retries; attempt++)); do
  if env KO_DOCKER_REPO="${REGISTRY}" ko resolve -f"${c}" "${extras[@]}" -B --sbom=none "${KO_EXTRA_FLAGS[@]}" >"${tmppac}"; then
    ko_success=true
    break
  fi
  if [[ ${attempt} -lt ${ko_max_retries} ]]; then
    echo "ko resolve failed (attempt ${attempt}/${ko_max_retries}), retrying in 15s..."
    sleep 15
  fi
done
Scope Leak

The loop variable attempt is not declared local, so it may leak/override a variable outside this function in bash. Declare it as local attempt (or use a different approach) to avoid unintended side effects.

local ko_success=false
local ko_max_retries=3
for ((attempt = 1; attempt <= ko_max_retries; attempt++)); do
  if env KO_DOCKER_REPO="${REGISTRY}" ko resolve -f"${c}" "${extras[@]}" -B --sbom=none "${KO_EXTRA_FLAGS[@]}" >"${tmppac}"; then
    ko_success=true
    break
  fi
  if [[ ${attempt} -lt ${ko_max_retries} ]]; then
    echo "ko resolve failed (attempt ${attempt}/${ko_max_retries}), retrying in 15s..."
    sleep 15
  fi
done

Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces retry logic for image resolution, which is a good improvement for handling intermittent network or registry issues. The changes include a pre-flight check for registry connectivity and a retry mechanism for the ko resolve command. My review focuses on improving the maintainability of the new retry logic by using constants for configuration and following shell scripting best practices.

Comment on lines +317 to +328
local reg_ok=false
for ((i = 1; i <= 5; i++)); do
if curl -o/dev/null --fail -k -s "https://${REGISTRY}/v2/"; then
reg_ok=true
break
fi
echo "Registry not ready, retrying in 5s ($i/5)..."
sleep 5
done
if [[ ${reg_ok} != true ]]; then
echo "Warning: Registry ${REGISTRY} not responding, proceeding anyway..."
fi
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The retry logic is a great addition. To improve maintainability and readability, consider extracting the magic numbers for retries and sleep intervals into named constants. Also, it's a good practice to quote variables within [[ ... ]] tests to prevent potential issues.

  local reg_ok=false
  local -r registry_check_retries=5
  local -r registry_check_interval=5
  for ((i = 1; i <= registry_check_retries; i++)); do
    if curl -o/dev/null --fail -k -s "https://${REGISTRY}/v2/"; then
      reg_ok=true
      break
    fi
    echo "Registry not ready, retrying in ${registry_check_interval}s ($i/${registry_check_retries})..."
    sleep "${registry_check_interval}"
  done
  if [[ "${reg_ok}" != "true" ]]; then
    echo "Warning: Registry ${REGISTRY} not responding, proceeding anyway..."
  fi

Comment on lines +331 to +346
local ko_success=false
local ko_max_retries=3
for ((attempt = 1; attempt <= ko_max_retries; attempt++)); do
if env KO_DOCKER_REPO="${REGISTRY}" ko resolve -f"${c}" "${extras[@]}" -B --sbom=none "${KO_EXTRA_FLAGS[@]}" >"${tmppac}"; then
ko_success=true
break
fi
if [[ ${attempt} -lt ${ko_max_retries} ]]; then
echo "ko resolve failed (attempt ${attempt}/${ko_max_retries}), retrying in 15s..."
sleep 15
fi
done
if [[ ${ko_success} != true ]]; then
echo "ko resolve failed after ${ko_max_retries} attempts"
return 1
fi
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Similar to the registry check, extracting the magic number for the retry interval into a named constant would improve maintainability. Also, quoting the variable in the if condition is a good practice.

  local ko_success=false
  local -r ko_max_retries=3
  local -r ko_retry_interval=15
  for ((attempt = 1; attempt <= ko_max_retries; attempt++)); do
    if env KO_DOCKER_REPO="${REGISTRY}" ko resolve -f"${c}" "${extras[@]}" -B --sbom=none "${KO_EXTRA_FLAGS[@]}" >"${tmppac}"; then
      ko_success=true
      break
    fi
    if [[ ${attempt} -lt ${ko_max_retries} ]]; then
      echo "ko resolve failed (attempt ${attempt}/${ko_max_retries}), retrying in ${ko_retry_interval}s..."
      sleep "${ko_retry_interval}"
    fi
  done
  if [[ "${ko_success}" != "true" ]]; then
    echo "ko resolve failed after ${ko_max_retries} attempts"
    return 1
  fi

@chmouel chmouel merged commit e1ad822 into pipelines-as-code:main Feb 26, 2026
3 checks passed
@chmouel chmouel deleted the improve-registry-check-and-ko-retry branch February 26, 2026 14:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant