Tag: aws-codebuild

How to separate serverless code from local server code

I currently have a static web app which is hosted on AWS S3 bucket hosting. The backend APIs are running on Lambda/API Gateway. I have a continuous development pipeline set up which automatically builds (using CodeBuild) and deploys (using CodeDeploy) my code from my GitHub repo onto S3.

The problem I have right now is that the code which is deployed serverlessly is different from the code I need to run on my local environment.

For example I want my local environment to call the API from localhost, but I want my production environment to call the API from a site like api.example.com. Also there is some code which is different in order to deploy to Lambda, which won’t run locally without reverting the change.

Another example: locally the APIs run on an Express server, but on AWS the code needs to be wrapped in
exports.handler = async (event, context, callback) => {...} to run serverlessly on Lambda.

My question is, how do I handle these differences between local and serverless in my Git repository?

Solution:

It would seem to me that using Codepipeline, CodeBuild and Cloudformation for this could be a good option. You didn’t specify what you are already using so I’m offering this as an example.

It will require some up-tooling to work. To get started you could have a look at CodeStar, which can get you set-up with a sample of this – that you could edit for your own purposes – in minutes. It can poll a github repo for changes on a particular branch to kick off builds.

In the pipeline you could have say 4 stages: Source (pull source from git), Build (CodeBuild to create the build artifacts), Dev/Test environment and finally Production.

The non-sensitive configuration you could keep in the git repo in different parameter files (for the cloudformation app stacks) for each different development and production configuration. You inject a different (json) parameter file for development and production (e.g. from the github repo) into the cloudformation stack as part of the deployment stage. Some of it you can also inject via parameter overrides (the come in handy where for instance you want to use “build tags” like commit id or whatever).

Consider using Secrets Manager for the sensitive bits like username/password pairs etc (or maybe all config – that’s an option too). Use a Secrets Manager key per environment. Send the Secrets Manager key (dev or prod) to the given app stack as an input parameter (cloudformation). Pull the Secrets Manager key/values from the app code (use an IAM role to grant privileges to for the Lambda’s to specific Secrets Manager keys) and use secret caching.

Also, as an aside – consider using AWS SAM (local) if you are not already.

The purpose of this is to:

  • Separate configuration from code
  • Automate your builds and deployments
  • Minimise scope for human error
  • Help reduce or outright eliminate downtime during deployments
  • Easy way to rollback (git revert for instance)

CodeBuild (AWS) from CodePipeline (AWS)

I’m trying to trigger multiple builds with CodePipeline (AWS) and when the pipeline trigger a CodeBuild, the CodeBuild fail with the next error:

[Container] 2018/02/07 19:30:20 Waiting for DOWNLOAD_SOURCE

Message: Access Denied

Extra information:

  • The source is coming from Github
  • If I start the CodeBuild manually works perfectly.

enter image description here

 

enter image description here

Solution:

I just discovered this the other day. I’m not sure if it’s documented anywhere, but it’s definitely not clear in the Code Pipeline UI.

Any CodeBuild project that CodePipeline initiates must have been created through the CodePipeline UI. It cannot be a “standalone” CodeBuild project.

When you create a CodeBuild project from the CodePipeline UI, the “Source Provider” setting is “AWS CodePipeline”, which is not an available choice when you create the CodeBuild project yourself.

CodePipeline retrieves it’s own source code from GitHub. It then passes that source code to your CodeBuild project. If your project is getting it’s own source code from GitHub, then that seems to cause the issue you describe:

[Container] 2018/02/06 14:58:37 Waiting for agent ping
[Container] 2018/02/06 14:58:37 Waiting for DOWNLOAD_SOURCE

To resolve this issue, you must edit your CodePipeline “build” stage, and choose “Create a new build project” under “AWS CodeBuild, Configure Your Project”. You can copy most settings from your existing project and reuse the buildspec.yml file in your source code.

Setting credentials for https git clone in AWS CodeBuild

I am running a CodeBuild on a project that has private requirements stored in CodeCommit.

My requriements.txt is

boto3==1.4.4
git+https://git-codecommit.us-west-2.amazonaws.com/v1/repos/my-req@master#egg=my-req

My IAM credentials are setup correctly and I have a username and password that works locally for both pip install -r requirements.txt and git clone git+https://....

I could use ssh but I don’t think that works with CodeBuild.

My buildspec.yml has

...
phases:
  install:
    commands:
      - pip install -r requirements.txt
...

I need to add a command in buildspec.yml that loads the https git credentials and then the git clone will work in pip install.

Currently it fails with:

fatal: could not read Username for 'https://git-codecommit.us-west-2.amazonaws.com': No such device or address

I can do git@git-code... but I still need the password. I don’t want to hard code it into requirements.txt.

What’s the best way to set https git credentials in buildspec.yml?

Solution:

Since the CodeBuild environment uses an IAM role for credentials (not a username and password), you will need to configure the CodeCommit credential helper in your buildspec:

- git config --global credential.helper '!aws codecommit credential-helper $@'
- git config --global credential.UseHttpPath true

Queue CodeBuild tasks

Is there are way to make a CodeBuild project execute build tasks one at a time (max concurrency = 1)?

I know one of the selling points of CodeBuild is that you can run builds concurrently and I like that feature.

However, for this one specific project, I NEED to make sure only one CodeBuild build task for this project executes at a time. If there is an incoming “startBuild” request while a previous request is still running, I want it to be queued and wait until the previous build task if finished.

As additional info on the project, this project runs integration tests across our various APIs (serverless APIs and legacy APIs on EC2) and running those tests concurrently may cause the tests to fail due to their setup and teardown procedures.

Solution:

I am from the AWS CodeBuild team. Thanks for your feedback. At this point the feature you requested isn’t supported. We’ll pass along the feedback to our product management team so they may consider adding it to our future roadmap.

However, you maybe able to implement something at your end by using CodeBuild’s Build notifications feature. At a high level, you could listen to the CloudWatch Events sent by CodeBuild to find when a build completes, and at that time “release” a new build from a queue that you maintain at your end.