feat(nodeadm): add support to configure http-proxy for al2023/nodeadm by shvbsle · Pull Request #2384 · awslabs/amazon-eks-ami

shvbsle · 2025-08-19T02:30:14Z

Issue #, if available:
#2128

Description of changes:
Nodeadm has two phases: 1. Config and 2. Run. Nodeadm has a set of outbound API calls (IMDS, AWS API calls) that could potentially need to be routed through a proxy. Currently, setting proxy configurations would happen after the config phase in user-data. This PR adds support to configure http-proxy in NodeConfigSpec.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Testing Done

Built an AL2023 AMI

make -d k8s=1.33 os_distro=al2023

Set up a proxy and added proxy details in the launch-template:

---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  instance:
    environment:
      default:
        HTTP_PROXY: "http://default-proxy:8080"
        HTTPS_PROXY: "https://default-proxy:8080"
        NO_PROXY: "localhost,127.0.0.1,169.254.169.254"
      kubelet:
        KUBELET_SPECIFIC_VAR: "kubelet-value"
        HTTP_PROXY: "http://kubelet-proxy:8080"  # Override default
      containerd:
        CONTAINERD_SPECIFIC_VAR: "containerd-value"
  cluster:
    apiServerEndpoint: https://example.com
    certificateAuthority: Y2VydGlmaWNhdGVBdXRob3JpdHk=
    cidr: 10.100.0.0/16
    name: my-cluster

Confirmed that:

Nodes join the cluster
IMDS calls don't go through proxy
nodeadm (config + run) outbound API calls go through the proxy
AWS EC2 API Calls go through the proxy
EKS API Server calls (gRPC) go through the proxy
Kubelet + containerd outbound calls go through the proxy

See this guide for recommended testing for PRs. Some tests may not apply. Completing tests and providing additional validation steps are not required, but it is recommended and may reduce review time and time to merge.

nodeadm/api/v1alpha1/nodeconfig_types.go

nodeadm/cmd/nodeadm/init/proxy_env.go

…picked up

…t variables

nodeadm/cmd/nodeadm/init/environment.go

nodeadm/cmd/nodeadm/init/init.go

nodeadm/internal/aws/imds/imds.go

nodeadm/cmd/nodeadm/init/environment.go

nodeadm/cmd/nodeadm/init/init.go

nodeadm/internal/aws/imds/imds.go

pattern. Handled escape quotes

quotes

shvbsle · 2025-09-16T18:16:00Z

/ci
+workflow:k8s_versions 1.33

github-actions · 2025-09-16T18:16:27Z

@shvbsle roger that! I've dispatched a workflow. 👍

github-actions · 2025-09-16T19:17:08Z

@shvbsle the workflow that you requested has completed. 🎉

AMI variant	Build	Test
1.33 / al2023	success ✅	success ✅

shvbsle · 2025-09-16T22:15:38Z

/ci

github-actions · 2025-09-16T22:16:02Z

@shvbsle roger that! I've dispatched a workflow. 👍

nodeadm/internal/system/environment.go

nodeadm/internal/aws/imds/imds.go

nodeadm/internal/system/environment.go

github-actions · 2025-09-16T23:20:54Z

@shvbsle the workflow that you requested has completed. 🎉

AMI variant	Build	Test
1.28 / al2023	success ✅	success ✅
1.29 / al2023	success ✅	success ✅
1.30 / al2023	success ✅	success ✅
1.31 / al2023	success ✅	success ✅
1.32 / al2023	success ✅	success ✅
1.33 / al2023	success ✅	success ✅

…v files separately

github-actions · 2025-09-30T00:17:31Z

@mselim00 the workflow that you requested has completed. 🎉

AMI variant	Build	Test
1.28 / al2023	success ✅	success ✅
1.29 / al2023	success ✅	success ✅
1.30 / al2023	success ✅	failure ❌
1.31 / al2023	success ✅	failure ❌
1.32 / al2023	success ✅	success ✅
1.33 / al2023	success ✅	failure ❌

shvbsle · 2025-09-30T01:14:06Z

/ci

github-actions · 2025-09-30T01:14:30Z

@shvbsle roger that! I've dispatched a workflow. 👍

github-actions · 2025-09-30T02:19:19Z

@shvbsle the workflow that you requested has completed. 🎉

AMI variant	Build	Test
1.28 / al2023	success ✅	success ✅
1.29 / al2023	success ✅	failure ❌
1.30 / al2023	success ✅	success ✅
1.31 / al2023	success ✅	failure ❌
1.32 / al2023	success ✅	failure ❌
1.33 / al2023	success ✅	failure ❌

shvbsle · 2025-09-30T18:48:29Z

I'm defining the following system aspect before the config phase is invoked:

	preConfigAspects := []system.SystemAspect{
		system.NewEnvironmentAspect(),
	}

This is because, enrichConfig calls the AWS EC2 API to get instance details and that call could potentially go through the proxy.

nodeadm/internal/system/environment.go

nodeadm/api/v1alpha1/nodeconfig_types.go

nodeadm/cmd/nodeadm/init/init.go

nodeadm/test/e2e/cases/nodeadm-environment-config/run.sh

nodeadm/test/e2e/helpers.sh

…e-equal directive for testcases

shvbsle · 2025-10-01T03:07:28Z

/ci

github-actions · 2025-10-01T03:07:59Z

@shvbsle roger that! I've dispatched a workflow. 👍

github-actions · 2025-10-01T04:11:31Z

@shvbsle the workflow that you requested has completed. 🎉

AMI variant	Build	Test
1.28 / al2023	success ✅	failure ❌
1.29 / al2023	success ✅	failure ❌
1.30 / al2023	success ✅	failure ❌
1.31 / al2023	success ✅	success ✅
1.32 / al2023	success ✅	failure ❌
1.33 / al2023	success ✅	success ✅

fletcherw · 2025-10-02T14:42:41Z

nodeadm/cmd/nodeadm/init/init.go

+		system.NewInstanceEnvironmentAspect(),
+	}
+
+	runAspects := []system.SystemAspect{


nit: should this be defined on line 140 just before it's used?

fletcherw · 2025-10-02T14:44:14Z

nodeadm/internal/system/environment.go

+
+const (
+	instanceEnvironmentAspectName = "instance-environment"
+	nodeamdEnvironmentAspectName  = "nodeadm-environment"


nit: is nodeadmd a typo?

fletcherw · 2025-10-02T14:48:20Z

nodeadm/internal/system/environment.go

+	}
+
+	// Reload systemd configuration once after all config files are written
+	zap.L().Info("Reloading systemd configuration")


this will still trigger daemon reload if all the maps within EnvironmentOptions are empty, it's alright if you don't fix this though.

fletcherw · 2025-10-02T14:49:04Z

nodeadm/internal/system/environment.go

+
+	zap.L().Info("All envOpts: ", zap.Any("=", envOpts))
+	for serviceName, envVars := range envOpts {
+		if len(envVars) > 0 {


nit: invert if condition:

if len(envVars) == 0 { continue }

to reduce indentation

fletcherw · 2025-10-02T14:55:19Z

nodeadm/internal/system/environment.go

+	var buf bytes.Buffer
+	if err := tmpl.Execute(&buf, configData); err != nil {
+		zap.L().Error("Failed to execute environment config template", zap.Error(err))
+		return ""


why log the error here vs returning it and logging at a higher level
by returning "" here you will write empty configs when you get an error.

shvbsle · 2025-10-02T21:34:55Z

/ci

github-actions · 2025-10-02T21:35:24Z

@shvbsle roger that! I've dispatched a workflow. 👍

github-actions · 2025-10-02T22:37:14Z

@shvbsle the workflow that you requested has completed. 🎉

AMI variant	Build	Test
1.28 / al2023	success ✅	success ✅
1.29 / al2023	success ✅	success ✅
1.30 / al2023	success ✅	success ✅
1.31 / al2023	success ✅	success ✅
1.32 / al2023	success ✅	success ✅
1.33 / al2023	success ✅	failure ❌
1.34 / al2023	success ✅	success ✅

shvbsle · 2025-10-02T23:06:39Z

CI is flaky but hoping its smoother now

/ci

edit: nvm: doesnt work after merging

github-actions · 2025-10-02T23:06:51Z

@shvbsle this PR is not currently mergeable, you'll need to rebase it first.

ndbaker1 · 2025-10-03T16:28:55Z

@shvbsle can you add example usage docs for this change?

feat(nodeadm): add support to configure http-proxy for al2023/nodeadm

b03bc80

cartermckinnon reviewed Aug 20, 2025

View reviewed changes

nodeadm/api/v1alpha1/nodeconfig_types.go Outdated Show resolved Hide resolved

nodeadm/cmd/nodeadm/init/proxy_env.go Outdated Show resolved Hide resolved

shvbsle added 7 commits August 20, 2025 09:23

generalize and allow setting of environment variables in NodeConfig

b9898a4

lazy loading of imds client to ensure that environment variables are …

18d9f78

…picked up

Create HTTP client with dynamic proxy function

b799e3a

simplify environment code and generate docs

43e1245

improve environment variable writes and address gosec scans

7439450

use DefaultEnvironment= directive to configure system wide environmen…

6595e37

…t variables

e2e test case for http-proxy-support

c20c4d9

shvbsle marked this pull request as ready for review August 26, 2025 21:02

shvbsle requested a review from cartermckinnon August 26, 2025 21:02

shvbsle changed the title ~~WIP: feat(nodeadm): add support to configure http-proxy for al2023/nodeadm~~ feat(nodeadm): add support to configure http-proxy for al2023/nodeadm Aug 26, 2025

cartermckinnon reviewed Aug 26, 2025

View reviewed changes

fletcherw reviewed Aug 26, 2025

View reviewed changes

nodeadm/cmd/nodeadm/init/environment.go Outdated Show resolved Hide resolved

nodeadm/cmd/nodeadm/init/init.go Outdated Show resolved Hide resolved

nodeadm/internal/aws/imds/imds.go Outdated Show resolved Hide resolved

shvbsle added 5 commits August 28, 2025 23:24

Moved environment.go to internal, and implemented as a SystemAspect

de5cff0

pattern. Handled escape quotes

add test-case complex environment variabels to test quite escaping

caf32af

clean up tests

92c8b2c

separated system aspects for config and run. Added test cases for escape

9ee4de6

quotes

fix linter

0e8016b

shvbsle requested review from cartermckinnon and fletcherw September 16, 2025 04:44

fletcherw approved these changes Sep 16, 2025

View reviewed changes

nodeadm/internal/system/environment.go Outdated Show resolved Hide resolved

nodeadm/internal/aws/imds/imds.go Outdated Show resolved Hide resolved

nodeadm/internal/system/environment.go Outdated Show resolved Hide resolved

shvbsle requested a review from ndbaker1 September 18, 2025 00:13

updated the NodeConfig to handle kubelet and containerd systemunit en…

4b6e132

…v files separately

generating dropin configs with go templates for better readbility

d411db0

move Environment aspect before enrichConfig call

7cc69bf

switch to a go standard library for escaping quotes

932827f

shvbsle commented Sep 30, 2025

View reviewed changes

shvbsle added 2 commits October 1, 2025 02:46

Simplify nodeconfig, separate envrionment configs by aspects, and fil…

6eef8fd

…e-equal directive for testcases

update docs

79b9bf3

shvbsle requested review from fletcherw and ndbaker1 October 1, 2025 03:13

fletcherw reviewed Oct 2, 2025

View reviewed changes

refactor code, propgate errors and handle edge cases

69024e9

fletcherw approved these changes Oct 2, 2025

View reviewed changes

shvbsle merged commit 6b33121 into awslabs:main Oct 2, 2025
12 checks passed

ndbaker1 mentioned this pull request Oct 9, 2025

feat(nodeadm): support for HTTP proxy #2128

Closed

mselim00 mentioned this pull request Dec 3, 2025

Bug: systemd-networkd races CNI on AL2023, causing missing routing rules & packet drops on secondary ENIs aws/amazon-vpc-cni-k8s#3524

Closed

inesshz mentioned this pull request Dec 4, 2025

[Feature] Support for new args for nodeadm eksctl-io/eksctl#8621

Closed

shvbsle deleted the http-proxy-support branch January 5, 2026 19:33

Conversation

shvbsle commented Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shvbsle commented Sep 16, 2025

Uh oh!

github-actions bot commented Sep 16, 2025

Uh oh!

github-actions bot commented Sep 16, 2025

Uh oh!

shvbsle commented Sep 16, 2025

Uh oh!

github-actions bot commented Sep 16, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Sep 16, 2025

Uh oh!

github-actions bot commented Sep 30, 2025

Uh oh!

shvbsle commented Sep 30, 2025

Uh oh!

github-actions bot commented Sep 30, 2025

Uh oh!

github-actions bot commented Sep 30, 2025

Uh oh!

shvbsle commented Sep 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shvbsle commented Oct 1, 2025

Uh oh!

github-actions bot commented Oct 1, 2025

Uh oh!

github-actions bot commented Oct 1, 2025

Uh oh!

fletcherw Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

fletcherw Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

fletcherw Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

fletcherw Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

fletcherw Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

shvbsle commented Oct 2, 2025

Uh oh!

github-actions bot commented Oct 2, 2025

Uh oh!

github-actions bot commented Oct 2, 2025

Uh oh!

Uh oh!

shvbsle commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 2, 2025

Uh oh!

ndbaker1 commented Oct 3, 2025

Uh oh!

Reviewers

shvbsle commented Aug 19, 2025 •

edited

Loading

shvbsle commented Oct 2, 2025 •

edited

Loading