Skip to content

Multiple Deployment Blockers on Ubuntu 24.04: Nested Proxy Isolation, Image Pull Timeouts, and CLI Parsing Errors #802

Closed
Bug
Closed
Multiple Deployment Blockers on Ubuntu 24.04: Nested Proxy Isolation, Image Pull Timeouts, and CLI Parsing Errors#802
Bug
Assignees
prekshivyas
Labels
area: installInstall, setup, prerequisites, or uninstall flowarea: onboardingOnboarding FSM, provider setup, sandbox launch, or first-run flowarea: packagingPackages, images, registries, installers, or distributionfixed-on-latestVerified by verify-stale skill: bug not reproducible on latestneeds: infoWaiting on author for missing detailsplatform: containerAffects Docker, containerd, Podman, or imagesplatform: k8sAffects Kubernetes deploymentsplatform: ubuntuAffects Ubuntu Linux environments
@qing-alt

Description

@qing-alt
qing-alt
opened on Mar 24, 2026

Description

Describe the bug

I am reporting a series of critical deployment blockers encountered while installing NemoClaw v0.1.0 on an Ubuntu 24.04 server within a restricted corporate network (using a local HTTP proxy).

Despite multiple workarounds, the system fails primarily due to the nested K3s environment ignoring host configurations and CLI command inconsistencies.

System Environment

  • OS: Ubuntu 24.04 LTS (Noble Numbat)
  • Docker Version: 29.3.0
  • NemoClaw CLI: v0.1.0 (OpenShell 0.0.14)
  • Network: Restricted Intranet (Global Proxy configured via Docker systemd)

Critical Issues Encountered

1. Nested Networking Isolation (The "I/O Timeout" Loop)

The host Docker daemon has a working proxy, but the nested K3s cluster inside the openshell-cluster-nemoclaw container completely ignores host proxy settings. It attempts to pull rancher/mirrored-pause:3.6 directly from registry-1.docker.io, resulting in an infinite i/o timeout.

2. Self-Destructive Onboarding Logic

nemoclaw onboard unconditionally destroys the gateway container before starting. This makes "air-gap" workarounds impossible because manually imported images (docker save/load or ctr images import) are wiped before the K3s initialization begins.

3. Cgroup v2 Permission Denied

On Ubuntu 24.04, the internal K3s fails with: forbidden: User system:node... cannot get resource "pods". This appears to be a known compatibility issue with Cgroup v2 that setup-spark tries to fix but fails due to mandatory phone verification for the NVIDIA API Key.

4. CLI Parsing Bugs

The CLI (v0.0.14) does not recognize its own documented commands or arguments:

  • nemoclaw sandbox returns Unknown command.
  • openshell sandbox create --file returns unexpected argument '--file' found.
  • openshell shell returns unrecognized subcommand 'shell'.

Steps to Reproduce

  1. Install NemoClaw on Ubuntu 24.04 in a network requiring a proxy.
  2. Run sudo nemoclaw onboard.
  3. Observe the Initializing environment hang followed by K8s namespace timeout.

Suggested Fixes

  1. Proxy Passthrough: Allow the nested K3s to inherit HTTP_PROXY from the host.
  2. Persistence: Do not destroy existing gateway containers during onboard if they contain pre-loaded images.
  3. CLI Refactor: Ensure the CLI arguments match the documentation provided in the help menus.

Reproduction Steps

  1. Install NemoClaw v0.1.0 on Ubuntu 24.04.5 LTS in a restricted network environment.
  2. Configure HTTP_PROXY/HTTPS_PROXY for the host Docker daemon.
  3. Run sudo /usr/bin/nemoclaw onboard.
  4. The process successfully pulls the gateway image but fails at [2/7] Starting OpenShell gateway.
  5. Error logs show the nested K3s cluster ignores the host proxy and attempts to pull rancher/mirrored-pause:3.6 directly, resulting in an i/o timeout.
  6. Also, documented commands like nemoclaw sandbox or openshell shell return Unknown command or unrecognized subcommand.

Environment

  • OS: Ubuntu 24.04.5 LTS (Noble Numbat)
  • Node.js: v20.20.1
  • Docker: 29.3.0
  • NemoClaw: v0.1.0 (OpenShell CLI 0.0.14)
  • CPU: x86_64
  • Network: Restricted Intranet with Local Proxy (Port 39949)

Debug Output

Error:   × K8s namespace not ready
  ╰─▶ timed out waiting for namespace 'openshell' to exist: Error from server (NotFound): namespaces "openshell" not found

      container logs:
        E0324 09:22:29.988161     127 kuberuntime_manager.go:1558] "CreatePodSandbox for pod failed" err="rpc error: code = DeadlineExceeded desc = failed to start sandbox \"24908ae96f473cd0ee718ba48ca436048414f01592f06110067525a67885da91\": failed to get sandbox image \"rancher/
      mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": failed to do request: Head \"https://registry-1.docker.io/
      v2/rancher/mirrored-pause/manifests/3.6\": dial tcp 54.234.18.200:443: i/o timeout" pod="kube-system/coredns-7566b5ff58-zmq5p"
        E0324 09:22:29.988269     127 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"coredns-7566b5ff58-zmq5p_kube-system(3469b982-48b4-4969-90ab-f4d0c03c425b)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"coredns-
      7566b5ff58-zmq5p_kube-system(3469b982-48b4-4969-90ab-f4d0c03c425b)\\\": rpc error: code = DeadlineExceeded desc = failed to start sandbox \\\"24908ae96f473cd0ee718ba48ca436048414f01592f06110067525a67885da91\\\": failed to get sandbox image \\\"rancher/mirrored-pause:3.6\\\":
      failed to pull image \\\"rancher/mirrored-pause:3.6\\\": failed to pull and unpack image \\\"docker.io/rancher/mirrored-pause:3.6\\\": failed to resolve reference \\\"docker.io/rancher/mirrored-pause:3.6\\\": failed to do request: Head \\\"https://registry-1.docker.io/v2/
      rancher/mirrored-pause/manifests/3.6\\\": dial tcp 54.234.18.200:443: i/o timeout\"" pod="kube-system/coredns-7566b5ff58-zmq5p" podUID="3469b982-48b4-4969-90ab-f4d0c03c425b"
        E0324 09:22:30.104581     127 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = DeadlineExceeded desc = failed to start sandbox \"75f06196b5eecc3af70ff3ec5725f813ff2ae095d01d7d53d87ae27fd4baabda\": failed to get sandbox image \"rancher/mirrored-
      pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": failed to do request: Head \"https://registry-1.docker.io/v2/
      rancher/mirrored-pause/manifests/3.6\": dial tcp 54.234.18.200:443: i/o timeout"
        E0324 09:22:30.104631     127 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = DeadlineExceeded desc = failed to start sandbox \"75f06196b5eecc3af70ff3ec5725f813ff2ae095d01d7d53d87ae27fd4baabda\": failed to get sandbox image \"rancher/
      mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": failed to do request: Head \"https://registry-1.docker.io/
      v2/rancher/mirrored-pause/manifests/3.6\": dial tcp 54.234.18.200:443: i/o timeout" pod="kube-system/metrics-server-786d997795-c44dq"
        E0324 09:22:30.104657     127 kuberuntime_manager.go:1558] "CreatePodSandbox for pod failed" err="rpc error: code = DeadlineExceeded desc = failed to start sandbox \"75f06196b5eecc3af70ff3ec5725f813ff2ae095d01d7d53d87ae27fd4baabda\": failed to get sandbox image \"rancher/
      mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": failed to do request: Head \"https://registry-1.docker.io/
      v2/rancher/mirrored-pause/manifests/3.6\": dial tcp 54.234.18.200:443: i/o timeout" pod="kube-system/metrics-server-786d997795-c44dq"
        E0324 09:22:30.104730     127 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"metrics-server-786d997795-c44dq_kube-system(28e0d2c3-9594-4a42-9e3f-342ed4288266)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\
      \"metrics-server-786d997795-c44dq_kube-system(28e0d2c3-9594-4a42-9e3f-342ed4288266)\\\": rpc error: code = DeadlineExceeded desc = failed to start sandbox \\\"75f06196b5eecc3af70ff3ec5725f813ff2ae095d01d7d53d87ae27fd4baabda\\\": failed to get sandbox image \\\"rancher/mirrored-
      pause:3.6\\\": failed to pull image \\\"rancher/mirrored-pause:3.6\\\": failed to pull and unpack image \\\"docker.io/rancher/mirrored-pause:3.6\\\": failed to resolve reference \\\"docker.io/rancher/mirrored-pause:3.6\\\": failed to do request: Head \\\"https://registry-
      1.docker.io/v2/rancher/mirrored-pause/manifests/3.6\\\": dial tcp 54.234.18.200:443: i/o timeout\"" pod="kube-system/metrics-server-786d997795-c44dq" podUID="28e0d2c3-9594-4a42-9e3f-342ed4288266"
        E0324 09:22:33.829219     127 handler_proxy.go:143] error resolving kube-system/metrics-server: no endpoints available for service "metrics-server"
        E0324 09:22:42.738127     127 resource_quota_controller.go:460] "Error during resource discovery" err="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: stale GroupVersion discovery: metrics.k8s.io/v1beta1"
        I0324 09:22:42.818308     127 garbagecollector.go:792] "failed to discover some groups" groups="map[\"metrics.k8s.io/v1beta1\":\"stale GroupVersion discovery: metrics.k8s.io/v1beta1\"]"
        W0324 09:22:44.142825     127 handler_proxy.go:99] no RequestInfo found in the context
        E0324 09:22:44.142950     127 controller.go:102] "Unhandled Error" err=<
        loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to download v1beta1.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
        , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
        >
        I0324 09:22:44.142976     127 controller.go:109] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
        W0324 09:22:44.143919     127 handler_proxy.go:99] no RequestInfo found in the context
        E0324 09:22:44.143980     127 controller.go:113] "Unhandled Error" err="loading OpenAPI spec for \"v1beta1.metrics.k8s.io\" failed with: Error, could not get list of group versions for APIService"
        I0324 09:22:44.144008     127 controller.go:126] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
        E0324 09:23:09.737905     127 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = DeadlineExceeded desc = failed to start sandbox \"83d003e607560678730a073a23594917d7d9cdee09d4c550b0f85e1879154df6\": failed to get sandbox image \"rancher/mirrored-
      pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": failed to do request: Head \"https://registry-1.docker.io/v2/
      rancher/mirrored-pause/manifests/3.6\": dial tcp 54.234.18.200:443: i/o timeout"
        E0324 09:23:09.737991     127 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = DeadlineExceeded desc = failed to start sandbox \"83d003e607560678730a073a23594917d7d9cdee09d4c550b0f85e1879154df6\": failed to get sandbox image \"rancher/
      mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": failed to do request: Head \"https://registry-1.docker.io/
      v2/rancher/mirrored-pause/manifests/3.6\": dial tcp 54.234.18.200:443: i/o timeout" pod="kube-system/local-path-provisioner-6bc6568469-ln6xj"
        E0324 09:23:09.738042     127 kuberuntime_manager.go:1558] "CreatePodSandbox for pod failed" err="rpc error: code = DeadlineExceeded desc = failed to start sandbox \"83d003e607560678730a073a23594917d7d9cdee09d4c550b0f85e1879154df6\": failed to get sandbox image \"rancher/
      mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": failed to do request: Head \"https://registry-1.docker.io/
      v2/rancher/mirrored-pause/manifests/3.6\": dial tcp 54.234.18.200:443: i/o timeout" pod="kube-system/local-path-provisioner-6bc6568469-ln6xj"
        E0324 09:23:09.738155     127 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"local-path-provisioner-6bc6568469-ln6xj_kube-system(8ff1c18e-93dd-4078-9929-982b900e73d7)\" with CreatePodSandboxError: \"Failed to create sandbox for
      pod \\\"local-path-provisioner-6bc6568469-ln6xj_kube-system(8ff1c18e-93dd-4078-9929-982b900e73d7)\\\": rpc error: code = DeadlineExceeded desc = failed to start sandbox \\\"83d003e607560678730a073a23594917d7d9cdee09d4c550b0f85e1879154df6\\\": failed to get sandbox image \\
      \"rancher/mirrored-pause:3.6\\\": failed to pull image \\\"rancher/mirrored-pause:3.6\\\": failed to pull and unpack image \\\"docker.io/rancher/mirrored-pause:3.6\\\": failed to resolve reference \\\"docker.io/rancher/mirrored-pause:3.6\\\": failed to do request: Head \\
      \"https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.6\\\": dial tcp 54.234.18.200:443: i/o timeout\"" pod="kube-system/local-path-provisioner-6bc6568469-ln6xj" podUID="8ff1c18e-93dd-4078-9929-982b900e73d7"
        E0324 09:23:12.180068     127 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to start sandbox \"d4b5cf149ce51e968ec79c3c1f2e2d602ad6be5f2673cd1b2096818bd5acef32\": failed to get sandbox image \"rancher/mirrored-pause:3.6\":
      failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": failed to do request: Head \"https://registry-1.docker.io/v2/rancher/mirrored-
      pause/manifests/3.6\": dial tcp 162.125.18.133:443: i/o timeout"
        E0324 09:23:12.180163     127 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to start sandbox \"d4b5cf149ce51e968ec79c3c1f2e2d602ad6be5f2673cd1b2096818bd5acef32\": failed to get sandbox image \"rancher/mirrored-
      pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": failed to do request: Head \"https://registry-1.docker.io/v2/
      rancher/mirrored-pause/manifests/3.6\": dial tcp 162.125.18.133:443: i/o timeout" pod="kube-system/helm-install-openshell-74gwd"
        E0324 09:23:12.180222     127 kuberuntime_manager.go:1558] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to start sandbox \"d4b5cf149ce51e968ec79c3c1f2e2d602ad6be5f2673cd1b2096818bd5acef32\": failed to get sandbox image \"rancher/mirrored-
      pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": failed to do request: Head \"https://registry-1.docker.io/v2/
      rancher/mirrored-pause/manifests/3.6\": dial tcp 162.125.18.133:443: i/o timeout" pod="kube-system/helm-install-openshell-74gwd"
        E0324 09:23:12.180348     127 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"helm-install-openshell-74gwd_kube-system(2ac6b774-6551-4dff-a34b-b04aca273c10)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"helm-
      install-openshell-74gwd_kube-system(2ac6b774-6551-4dff-a34b-b04aca273c10)\\\": rpc error: code = Unknown desc = failed to start sandbox \\\"d4b5cf149ce51e968ec79c3c1f2e2d602ad6be5f2673cd1b2096818bd5acef32\\\": failed to get sandbox image \\\"rancher/mirrored-pause:3.6\\\":
      failed to pull image \\\"rancher/mirrored-pause:3.6\\\": failed to pull and unpack image \\\"docker.io/rancher/mirrored-pause:3.6\\\": failed to resolve reference \\\"docker.io/rancher/mirrored-pause:3.6\\\": failed to do request: Head \\\"https://registry-1.docker.io/v2/
      rancher/mirrored-pause/manifests/3.6\\\": dial tcp 162.125.18.133:443: i/o timeout\"" pod="kube-system/helm-install-openshell-74gwd" podUID="2ac6b774-6551-4dff-a34b-b04aca273c10"
        E0324 09:23:12.263299     127 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = DeadlineExceeded desc = failed to start sandbox \"440d80ad2d52cb938df732a85f7c595987cb5a06f2d8e681781073fcbb2974eb\": failed to get sandbox image \"rancher/mirrored-
      pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": failed to do request: Head \"https://registry-1.docker.io/v2/
      rancher/mirrored-pause/manifests/3.6\": dial tcp 54.234.18.200:443: i/o timeout"
        E0324 09:23:12.263391     127 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = DeadlineExceeded desc = failed to start sandbox \"440d80ad2d52cb938df732a85f7c595987cb5a06f2d8e681781073fcbb2974eb\": failed to get sandbox image \"rancher/
      mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": failed to do request: Head \"https://registry-1.docker.io/
      v2/rancher/mirrored-pause/manifests/3.6\": dial tcp 54.234.18.200:443: i/o timeout" pod="agent-sandbox-system/agent-sandbox-controller-0"
        E0324 09:23:12.263457     127 kuberuntime_manager.go:1558] "CreatePodSandbox for pod failed" err="rpc error: code = DeadlineExceeded desc = failed to start sandbox \"440d80ad2d52cb938df732a85f7c595987cb5a06f2d8e681781073fcbb2974eb\": failed to get sandbox image \"rancher/
      mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": failed to do request: Head \"https://registry-1.docker.io/
      v2/rancher/mirrored-pause/manifests/3.6\": dial tcp 54.234.18.200:443: i/o timeout" pod="agent-sandbox-system/agent-sandbox-controller-0"
        E0324 09:23:12.263580     127 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"agent-sandbox-controller-0_agent-sandbox-system(7f72633d-d333-4d82-a715-94eff50ed115)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\
      \"agent-sandbox-controller-0_agent-sandbox-system(7f72633d-d333-4d82-a715-94eff50ed115)\\\": rpc error: code = DeadlineExceeded desc = failed to start sandbox \\\"440d80ad2d52cb938df732a85f7c595987cb5a06f2d8e681781073fcbb2974eb\\\": failed to get sandbox image \\\"rancher/
      mirrored-pause:3.6\\\": failed to pull image \\\"rancher/mirrored-pause:3.6\\\": failed to pull and unpack image \\\"docker.io/rancher/mirrored-pause:3.6\\\": failed to resolve reference \\\"docker.io/rancher/mirrored-pause:3.6\\\": failed to do request: Head \\\"https://
      registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.6\\\": dial tcp 54.234.18.200:443: i/o timeout\"" pod="agent-sandbox-system/agent-sandbox-controller-0" podUID="7f72633d-d333-4d82-a715-94eff50ed115"
        E0324 09:23:12.705070     127 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = DeadlineExceeded desc = failed to start sandbox \"b8c07f4285c6867f0b57213a9dc1997a090e4732301e3919ad01bf420759cd50\": failed to get sandbox image \"rancher/mirrored-
      pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": failed to do request: Head \"https://registry-1.docker.io/v2/
      rancher/mirrored-pause/manifests/3.6\": dial tcp 162.125.18.133:443: i/o timeout"
        E0324 09:23:12.705129     127 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = DeadlineExceeded desc = failed to start sandbox \"b8c07f4285c6867f0b57213a9dc1997a090e4732301e3919ad01bf420759cd50\": failed to get sandbox image \"rancher/
      mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": failed to do request: Head \"https://registry-1.docker.io/
      v2/rancher/mirrored-pause/manifests/3.6\": dial tcp 162.125.18.133:443: i/o timeout" pod="kube-system/coredns-7566b5ff58-zmq5p"
        E0324 09:23:12.705151     127 kuberuntime_manager.go:1558] "CreatePodSandbox for pod failed" err="rpc error: code = DeadlineExceeded desc = failed to start sandbox \"b8c07f4285c6867f0b57213a9dc1997a090e4732301e3919ad01bf420759cd50\": failed to get sandbox image \"rancher/
      mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": failed to do request: Head \"https://registry-1.docker.io/
      v2/rancher/mirrored-pause/manifests/3.6\": dial tcp 162.125.18.133:443: i/o timeout" pod="kube-system/coredns-7566b5ff58-zmq5p"
        E0324 09:23:12.705221     127 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"coredns-7566b5ff58-zmq5p_kube-system(3469b982-48b4-4969-90ab-f4d0c03c425b)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"coredns-
      7566b5ff58-zmq5p_kube-system(3469b982-48b4-4969-90ab-f4d0c03c425b)\\\": rpc error: code = DeadlineExceeded desc = failed to start sandbox \\\"b8c07f4285c6867f0b57213a9dc1997a090e4732301e3919ad01bf420759cd50\\\": failed to get sandbox image \\\"rancher/mirrored-pause:3.6\\\":
      failed to pull image \\\"rancher/mirrored-pause:3.6\\\": failed to pull and unpack image \\\"docker.io/rancher/mirrored-pause:3.6\\\": failed to resolve reference \\\"docker.io/rancher/mirrored-pause:3.6\\\": failed to do request: Head \\\"https://registry-1.docker.io/v2/
      rancher/mirrored-pause/manifests/3.6\\\": dial tcp 162.125.18.133:443: i/o timeout\"" pod="kube-system/coredns-7566b5ff58-zmq5p" podUID="3469b982-48b4-4969-90ab-f4d0c03c425b"
        E0324 09:23:12.744807     127 resource_quota_controller.go:460] "Error during resource discovery" err="unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: stale GroupVersion discovery: metrics.k8s.io/v1beta1"
        I0324 09:23:12.829412     127 garbagecollector.go:792] "failed to discover some groups" groups="map[\"metrics.k8s.io/v1beta1\":\"stale GroupVersion discovery: metrics.k8s.io/v1beta1\"]"
        E0324 09:23:15.705699     127 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = DeadlineExceeded desc = failed to start sandbox \"6375472c6b9bcbdb8a4ee5ef2dc5ea50c20ede7a81df997b55a9a2d41d11db27\": failed to get sandbox image \"rancher/mirrored-
      pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": failed to do request: Head \"https://registry-1.docker.io/v2/
      rancher/mirrored-pause/manifests/3.6\": dial tcp 59.24.3.173:443: i/o timeout"
        E0324 09:23:15.705782     127 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = DeadlineExceeded desc = failed to start sandbox \"6375472c6b9bcbdb8a4ee5ef2dc5ea50c20ede7a81df997b55a9a2d41d11db27\": failed to get sandbox image \"rancher/
      mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": failed to do request: Head \"https://registry-1.docker.io/
      v2/rancher/mirrored-pause/manifests/3.6\": dial tcp 59.24.3.173:443: i/o timeout" pod="kube-system/metrics-server-786d997795-c44dq"
        E0324 09:23:15.705833     127 kuberuntime_manager.go:1558] "CreatePodSandbox for pod failed" err="rpc error: code = DeadlineExceeded desc = failed to start sandbox \"6375472c6b9bcbdb8a4ee5ef2dc5ea50c20ede7a81df997b55a9a2d41d11db27\": failed to get sandbox image \"rancher/
      mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": failed to do request: Head \"https://registry-1.docker.io/
      v2/rancher/mirrored-pause/manifests/3.6\": dial tcp 59.24.3.173:443: i/o timeout" pod="kube-system/metrics-server-786d997795-c44dq"
        E0324 09:23:15.705963     127 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"metrics-server-786d997795-c44dq_kube-system(28e0d2c3-9594-4a42-9e3f-342ed4288266)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\
      \"metrics-server-786d997795-c44dq_kube-system(28e0d2c3-9594-4a42-9e3f-342ed4288266)\\\": rpc error: code = DeadlineExceeded desc = failed to start sandbox \\\"6375472c6b9bcbdb8a4ee5ef2dc5ea50c20ede7a81df997b55a9a2d41d11db27\\\": failed to get sandbox image \\\"rancher/mirrored-
      pause:3.6\\\": failed to pull image \\\"rancher/mirrored-pause:3.6\\\": failed to pull and unpack image \\\"docker.io/rancher/mirrored-pause:3.6\\\": failed to resolve reference \\\"docker.io/rancher/mirrored-pause:3.6\\\": failed to do request: Head \\\"https://registry-
      1.docker.io/v2/rancher/mirrored-pause/manifests/3.6\\\": dial tcp 59.24.3.173:443: i/o timeout\"" pod="kube-system/metrics-server-786d997795-c44dq" podUID="28e0d2c3-9594-4a42-9e3f-342ed4288266"

  Command failed (exit 1): openshell gateway start --name nemoclaw

Logs



Checklist


    

  •  I confirmed this bug is reproducible
    • 

  •  I searched existing issues and this is not a duplicate
    • 

Metadata
Metadata
Assignees
Labels
area: installInstall, setup, prerequisites, or uninstall flowarea: onboardingOnboarding FSM, provider setup, sandbox launch, or first-run flowarea: packagingPackages, images, registries, installers, or distributionfixed-on-latestVerified by verify-stale skill: bug not reproducible on latestneeds: infoWaiting on author for missing detailsplatform: containerAffects Docker, containerd, Podman, or imagesplatform: k8sAffects Kubernetes deploymentsplatform: ubuntuAffects Ubuntu Linux environments
Type
Bug
Fields
No fields configured for Bug.
Projects
No projects
Milestone
No milestone
Relationships
None yet
Development
No branches or pull requests
Issue actions