Skip to content

fix(aws-eks): Support for http proxy in EKS onEvent lambda#16609

Merged
mergify[bot] merged 8 commits intomasterfrom
fix-node-lambda-http-proxy
Sep 23, 2021
Merged

fix(aws-eks): Support for http proxy in EKS onEvent lambda#16609
mergify[bot] merged 8 commits intomasterfrom
fix-node-lambda-http-proxy

Conversation

@ryparker
Copy link
Copy Markdown
Contributor

@ryparker ryparker commented Sep 22, 2021 •
edited
Loading

Summary

Currently when a user wants to route all of the EKS lambda's SDK requests through a proxy then they are instructed to configure an env var named HTTP_PROXY or http_proxy.

e.g.

const cluster = new eks.Cluster(this, 'hello-eks', {
  version: eks.KubernetesVersion.V1_21,
  clusterHandlerEnvironment: {
    'http_proxy': 'http://proxy.myproxy.com'
  }
});

However the JS SDK requires further configuration to enable proxy support.

This PR:

  • Adds a package.json with the dependency 'proxy-agent' to the cluster-resource-handler/ lambda bundle
  • Uses NodeJSFunction to install lambda dependencies and bundle.
  • Adds a condition that checks the environment for HTTP_PROXY or http_proxy values. If present then configures the aws-sdk to use that proxy (using proxy-agent).

Note: I placed the proxy-agent in the devDependencies of package.json. If the dependency is placed in the dependencies section then the CDK builder throws an error: NPM Package cluster-resources-handler inside jsii package '@aws-cdk/aws-eks', can only have devDependencies

Fixes: SIM D29159517, #12469

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@ryparker ryparker added p1 @aws-cdk/aws-eks Related to Amazon Elastic Kubernetes Service contribution/core This is a PR that came from AWS. labels Sep 22, 2021
@ryparker ryparker self-assigned this Sep 22, 2021
@gitpod-io
Copy link
Copy Markdown

gitpod-io bot commented Sep 22, 2021

@ryparker ryparker linked an issue Sep 22, 2021 that may be closed by this pull request
(aws-eks): Construct Library custom resources doesn't use proxy properly #12469
Closed
@ryparker
Copy link
Copy Markdown
Contributor Author

ryparker commented Sep 22, 2021

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
  • Commit ID: 483ad37
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Fix for tests is on the way. Deploying and validating integration test changes takes a while with EKS.

otaviomacedo
otaviomacedo reviewed Sep 23, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@otaviomacedo otaviomacedo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, but the integration tests of aws-stepfunctions-tasks are failing:

@aws-cdk/aws-stepfunctions-tasks: Error: Some stacks have changed. To verify that they still deploy successfully, run: 'yarn integ eks/integ.call.js'

@ryparker
Copy link
Copy Markdown
Contributor Author

ryparker commented Sep 23, 2021

Looks good to me, but the integration tests of aws-stepfunctions-tasks are failing:

@aws-cdk/aws-stepfunctions-tasks: Error: Some stacks have changed. To verify that they still deploy successfully, run: 'yarn integ eks/integ.call.js'

Done 👍🏻

@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Sep 23, 2021

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation
Copy link
Copy Markdown
Collaborator

aws-cdk-automation commented Sep 23, 2021

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
  • Commit ID: a998132
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify mergify bot merged commit cf22280 into master Sep 23, 2021
@mergify mergify bot deleted the fix-node-lambda-http-proxy branch September 23, 2021 11:26
@mergify
Copy link
Copy Markdown
Contributor

mergify bot commented Sep 23, 2021

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@ryparker ryparker restored the fix-node-lambda-http-proxy branch September 23, 2021 23:08
ryparker added a commit that referenced this pull request Sep 24, 2021
@ryparker
Revert "fix(aws-eks): Support for http proxy in EKS onEvent lambda (#…
5ef5cb9 
…16609)"

This reverts commit cf22280.
@ryparker
Copy link
Copy Markdown
Contributor Author

ryparker commented Sep 24, 2021

A bug was introduced by this PR that broke the onEvent EKS cluster handler lambda. ESBuild was inlining the node_module proxy-agent which was trying to read a file that did not exist (because all dependencies were bundled into a single file).

e.g.

var contextify = fs.readFileSync('/var/task/contextify.js');

Error:

ENOENT: no such file or directory, open '/var/task/contextify.js' Logs: /aws/lambda/test-fixed-nobundle-eks-wit-OnEventHandler42BEBAE0-s2cZwaWDW0xt at Object.openSync (fs.js:462:3) at Object.readFileSync (fs.js:364:35) at loadAndCompileScript (/var/task/index.js:29479:23) at ../aws-cdk/node_modules/vm2/lib/main.js (/var/task/index.js:29490:25) at __require (/var/task/index.js:26:44) at ../aws-cdk/node_modules/vm2/index.js (/var/task/index.js:30079:23) at __require (/var/task/index.js:26:44) at ../aws-cdk/node_modules/degenerator/dist/src/index.js (/var/task/index.js:30091:17) at __require (/var/task/index.js:26:44) at ../aws-cdk/node_modules/pac-resolver/dist/index.js (/var/task/index.js:30857:25) (RequestId: c44d1357-fbce-4f96-8c23-b865c2c3aaff)

I created a revert of this PR and a fresh PR with the corrected fix.

The updated fix:

  • Replaces proxy-agent with the smaller module http-proxy-agent (reduced bundle size from 1.5mb -> 18.5kb)
  • Adds configuration for the NodeJsFunction's bundling param to not inline http-proxy-agent.
  • Adds configuration for the NodeJsFunction's bundling param to exclude aws-sdk from the bundle.
  • Adds esbuild as a dev dependency of the aws-eks package (it is required by NodeJsFunction for tests).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@otaviomacedo otaviomacedo otaviomacedo approved these changes

Assignees

@otaviomacedo otaviomacedo

@ryparker ryparker

Labels

@aws-cdk/aws-eks Related to Amazon Elastic Kubernetes Service contribution/core This is a PR that came from AWS. p1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

(aws-eks): Construct Library custom resources doesn't use proxy properly

3 participants

@ryparker @aws-cdk-automation @otaviomacedo