Skip to content

Add local registry support for remote builders#1679

Merged
djmb merged 2 commits intomainfrom
remote-builder-local-registry
Oct 27, 2025
Merged

Add local registry support for remote builders#1679
djmb merged 2 commits intomainfrom
remote-builder-local-registry

Conversation

@djmb
Copy link
Collaborator

@djmb djmb commented Oct 21, 2025

When running with a remote builder and a local registry:

  1. Forward the local registry port to the remote builder
  2. Use host networking on the remote builder
  3. Use a different builder name as the settings are different

A remote builder can only have one port at a time forwarded, so concurrent builds on the same registry port will fail.

See: #1674 (comment)

cc @jeromedalbert, @adrienpoly

When running with a remote builder and a local registry:
1. Forward the local registry port to the remote builder
2. Use host networking on the remote builder
3. Use a different builder name as the settings are different

A remote builder can only have one port at a time forwarded, so
concurrent builds on the same registry port will fail.
@djmb djmb force-pushed the remote-builder-local-registry branch from ac0d598 to 0055338 Compare October 21, 2025 18:44
@darkamenosa
Copy link

Is this released in 2.8.1. I got the same issue, and here is my config:

# Credentials for your image host.
registry:
  server: "localhost:5555"

# Configure the image builder.
builder:
  # Build for the server's native architecture (aarch64 == arm64)
  arch: arm64
  # Enable a registry cache so apt/bundler layers can be reused across deploys.
  cache:
    type: registry
    # Use an arch-specific cache image to avoid mixing caches across platforms
    image: tuyenhx/async_example-buildcache-arm64
    options: mode=max,image-manifest=true,oci-mediatypes=true

  remote: ssh://root@46.224.21.86
  local: false

@adrienpoly
Copy link

thanks @djmb I tested this branch

It now works when I deploy from my mac but it fails when I deploy from the CI

To add more context
Mac: arm64
Host : amd64
CI: amd64

I tried both with local: true and false and the results where the same

Here is the draft PR rubyevents/rubyevents#1096

and the kamal config https://github.com/rubyevents/rubyevents/blob/main/config/deploy.staging.yml

let me know if you want me to run any more tests

@djmb
Copy link
Collaborator Author

djmb commented Oct 23, 2025

@adrienpoly - I only see remote builds failing in the action tab. I think the "connection reset by peer" error probably indicates that port forwarding was initially set up correctly, so possibly the SSH connection is being dropped?

I've added the keepalive configuration, hopefully that might fix it, but I've also added some logging for any errors in the port forwarding thread, if anything goes wrong there right now I think we are missing it.

Could you try building with this branch again when you have a chance?

@djmb djmb force-pushed the remote-builder-local-registry branch from 8befce9 to 058ba56 Compare October 23, 2025 17:34
- Include keepalive options when forwarding port for remote builder
- Log port forwarding errors.
@djmb djmb force-pushed the remote-builder-local-registry branch from 058ba56 to 10e40a7 Compare October 23, 2025 17:48
@djmb djmb merged commit 4bac779 into main Oct 27, 2025
16 of 17 checks passed
@djmb djmb deleted the remote-builder-local-registry branch October 27, 2025 08:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants