DevTestProd AWS

This project was used as a learning tool to figure out how to develop a production-grade CI / CD workflow that would minimize the risk of publishing bad code into production AWS environments.

Comment

Inspiration

This hackathon project was used as a learning tool to figure out how to develop a production-grade CI / CD workflow that would minimize the risk of publishing bad code into production AWS environments.

What it does

Environment setup:

  1. I push my code changes into a dev branch which automatically runs 2 tests, unit-test and integration-test. The unit-test checks each piece of my code individually and makes sure that they are operating the way they're supposed to, independent from other interacting components. The integration-test checks that a given input to one piece of code will trigger the rest of the AWS architecture and other interacting components, and produce the desired end result, all running inside a localized AWS environment (i.e. not the real AWS cloud).
  2. If both tests succeed, I can proceed to merge my code from the dev branch into the main branch. This will then trigger the unit-test and integration-test workflows again to ensure that it'll still pass. If both tests still pass, then the final deploy workflow starts and deploys the AWS architecture onto the real AWS environment, returning a REST endpoint to interact with the application.
  3. The main branch also deploys a basic HTML page, hosted in GitHub pages, to interact with the deployed AWS application via the returned REST endpoint.

This is a diagram of the entire CI / CD pipeline.

Diagram

Learning goals:

  • Deploy Typescript Node.JS applications on Lambda
  • Run a basic AWS architecture on LocalStack (locally running AWS cloud)
  • Develop a production-grade CI / CD workflow for deploying AWS-based applications
    • GitHub actions
      • Unit tests via Bun
      • Integrations test via LocalStack
      • Deploy to AWS via SAM CLI

How I built it

Application

The backend application is a simple set & get color API. Here is an explanation of how the AWS architecture diagram works. The intention of over-complicating the AWS architecture for the application was to test that all the services would work when deployed to a LocalStack environment, which is a locally deployed AWS environment for testing.

Save color:

  1. The user sends a POST request of the color they want to set to the API's main interface, Main Function, which is a Lambda function.
  2. The request is forwarded to an SQS buffer that forwards the data to the Worker Function.
  3. Worker Function takes the event data and saves the color for the user in a DynamoDB table.

Get color:

  1. The user sends a GET request of the color they saved.
  2. Main Function looks up the corresponding user's data in the DynamoDB table and retrieves the saved color.

AWS Architecture

Additional Notes:

  • The concurrent executions allowed are set to 2 intentionally to prevent spam requests
  • All Lambdas and associated tests were written in Typescript
  • Both Lambdas utilize a utility script located in a shared Lambda layer
  • The DynamoDB table automatically deletes all items 1 hour after they were last updated
  • Main Function code is located here
  • Worker Function code is located here
  • Shared Lambda layer found here
  • CloudFormation template used to deploy AWS architecture (either locally or for real) found here

GitHub Actions

These 3 workflows run every time a commit is pushed onto either the dev branch and after a successful pull request merge into the main branch.

unit-test steps

  1. Grabs a set of AWS credentials to satisfy AWS SDK usage requirements (not used by code)
  2. Runs unit tests which test each Lambda's application logic and mocks the AWS components

Notes:

  • This action runs in a GitHub cloud-hosted runner
  • Workflow found here
  • Unit tests found here

integration-test steps

  1. Spins up a LocalStack Docker container service
  2. Create an S3 bucket to store the CloudFormation template, Lambda code files, & Lambda layer code
  3. Install CLI tools for deployment
  4. Builds the Lambda layer (converts TS files to JS files & installs dependencies)
  5. Builds CloudFormation template (converts TS files to JS files & prepares final CloudFormation deployment template)
  6. Deploy the final CloudFormation template onto LocalStack's local AWS environment
  7. Record the REST endpoint to interact with the application in the integration tests
  8. Run integration tests which test that interacting with the Main Function should trigger the Worker Function and store the color in the DynamoDB table correctly. One test interacts with the Lambdas directly, and another interacts with the REST endpoint instead.

Notes:

deploy steps (only applies to main branch)

  1. Waits for unit-test & integration-test to succeed
  2. Install CLI tools for deployment
  3. Grab the AWS credentials needed to deploy to the correct AWS account
  4. Builds the Lambda layer (converts TS files to JS files & installs dependencies)
  5. Builds CloudFormation template (converts TS files to JS files & prepares final CloudFormation deployment template)
  6. Deploy the final CloudFormation template onto the real AWS environment

Notes:

  • This action runs in a GitHub cloud-hosted runner
  • Workflow found here

GitHub Pages

I created a basic HTML page that would allow the user to interact with the backend service. It is hosted on GitHub pages and can be accessed at this link: https://riskyfrisky.github.io/DevTestProdAWS. When you set the color, you may need to wait a moment to get back the stored color since the request has to wake up the Lambda instances from a cold start, so you might get "null" as a response if you try to get back your color too quickly.

Challenges I ran into

  • Problem: LocalStack would run well on my local computer as well as in a GitHub action test environment via the act library but wouldn't run in the GitHub action because the internal network couldn't be accessed (i.e. can't access Lambda endpoint/function URL). I spent numerous hours on this problem and hadn't figured it out for many days.
    • Solution: Run a GitHub custom-hosted runner on my local computer instead of using the cloud-hosted ubuntu machine. It allows for more control and gets the LocalStack deployment to expose its internal Docker network for the integration tests.
  • Problem: Couldn't access application REST endpoint on GitHub page due to CORS issues.
    • Solution: Enabled CORS for the function URL configuration in the CloudFormation template, allowing access to the GitHub page origin only.

Accomplishments that I'm proud of

  • Wrote tests that check for bugs for each Lambda function as well as an integration test that invokes the entire system, testing the whole AWS architecture
  • Developed a CI / CD workflow that would minimize the risk of pushing bad code to a production environment

What I learned

  • Learned how to deploy Typescript Node.JS applications to AWS Lambdas
  • Learned how to deploy, run, and test an AWS architecture in LocalStack
  • Learned how to create and run GitHub action workflows
  • Learned about custom GitHub action runners and when to use them
  • Learned more about pull requests and git merging
  • Learned how to grab AWS credentials without storing AWS keys in GitHub environmental variables in workflows

What's next for DevTestProdAWS

  • Test larger/more advanced AWS architectures in this CI / CD pipeline
  • Cache installation of node dependencies via Bun
  • Cache LocalStack Docker image pull

Built With

Share this project:

Updates