DEV Community: Ryan Dsouza

Canary deployment of Lambdas using CDK Pipelines

Ryan Dsouza — Wed, 25 Aug 2021 15:18:15 +0000

In this post, we shall perform a Canary deployment of our Lambdas. We will be using CDK pipelines for automated deployments.

Canary is a deployment strategy that releases an application incrementally to a subset of users. This is done to limit the blast radius and for an easy rollback in case of a failure.

What we would like to achieve is:

Automate our API deployment using CDK Pipelines
Use CodeDeploy's Deployment Groups to perform a Canary deployment using a Lambda alias. This will perform a weighted routing between the current function and the previous function version.
Create an alarm to check for any errors in the Lambda and rollback if any.
Perform a simple load test to check if everything works.

Here's the repo for those who want to dive right in or follow along with the walkthrough.

ryands17 / lambda-canary-deployments

API Gateway and Lambda with weighted routing to the latest function deployed

Prerequisites

CDK prerequisites like bootstrapping and setting up AWS CLI with the default profile is assumed.
As our repository will be on GitHub, we need to create an access token for CodePipeline to fetch our repository. To perform this, create a Personal Access Token on GitHub with the repo and admin:repo_hook options checked.

Then, you need to create a Secrets Manager secret that will store this token and we will fetch this later in our CDKPipelines construct.

We're done with the prerequisites. Let's move on to creating the API.

API Stack

This stack will contain an API Gateway REST API with a Lambda Proxy integration. We will also add a CodeDeploy Deployment Group that will perform the required traffic shifting from the current to the latest deployed version.

In case our deployment is erroneous for some reason, CodeDeploy should rollback to the current version. For this, we will use CloudWatch Alarms that will check if our Lambda give any errors and if the alarm is in an Alarm state, CodeDeploy will rollback.

Let's start with the Lambda:



// lib/api-stack.ts

const aliasName = 'stage'

const handler = new Lambda(this, 'apiHandler')
const stage = new lambda.Alias(this, 'apiHandlerStage', {
  aliasName,
  version: handler.currentVersion,
})

We create a Lambda function named apiHandler and an alias named apiHandlerStage which we will point to the current version. When we deploy a new version, CodeDeploy will perform a weighted routing using the alias that will point both to the current version and the latest deployed version.

Next, we will create the REST API.



// lib/api-stack.ts

const api = new apiGw.LambdaRestApi(this, 'restApi', {
  handler: stage,
  deployOptions: { stageName: 'staging' },
})

CDK provides us with a neat construct named LambdaRestApi that automatically routes any request arriving to the Lambda that we specify using Lambda Proxy integration. And here we have specified stage which is actually an alias.

Moving on to the important step, i.e configuring an alarm for rollbacks in case of errors.



// lib/api-stack.ts

const failureAlarm = new cw.Alarm(this, 'lambdaFailure', {
  alarmDescription: 'The latest deployment errors > 0',
  metric: new cw.Metric({
    metricName: 'Errors',
    namespace: 'AWS/Lambda',
    statistic: 'sum',
    dimensionsMap: {
      Resource: `${handler.functionName}:${aliasName}`,
      FunctionName: handler.functionName,
    },
    period: cdk.Duration.minutes(1),
  }),
  threshold: 1,
  evaluationPeriods: 1,
})

Let's break this down. First, we create a description for this alarm named lambdaFailure.

We then specify a metric based on which we want the alarm to react to. The metric here is an AWS provided metric named Errors under the AWS/Lambda namespace.

We want the observe the total number of errors so we specify sum as the statistic. The time period over which we want this statistic to apply is specified in period and we set that to be 1 minute.

The dimensions that we need to specify are FunctionName i.e. our Lambda function name and Resource which will be our Lambda alias name in this case. The alias name will always be the functionName:aliasName. We will be watching the Error metric of this function specifically.

We then specify the threshold which in simple terms means that how many errors should occur before the alarm goes in an Alarm state. Even if we encounter 1 error, we would like to trigger the alarm in this case.

Finally, we specify evaluationPeriods which is the number of periods over which the statistic is compared to the threshold. We have set this to 1 because what we want is to trigger the alarm in a period of 1 minute if the Lambda errors 1 or more times.

We created the alarm, now let's use this in our Deployment Group.



// lib/api-stack.ts

new cd.LambdaDeploymentGroup(this, 'canaryDeployment', {
  alias: stage,
  deploymentConfig: cd.LambdaDeploymentConfig.CANARY_10PERCENT_5MINUTES,
  alarms: [failureAlarm],
})

We create a CodeDeploy Deployment Group specifying our Lambda alias and a Canary deployment of 10% in 5 minutes.

So for the first 5 minutes, we will be serving 90% of our current Lambda version and 10% of the newly deployed Lambda version. After 5 minutes, the entire traffic will be shifted over to the newly deployed Lambda version and that will become the current version. We also provided the created alarm in alarms. Note that we can specify more than one alarm.

Finally, let's look at our Lambda function:



// functions/apiHandler.ts

import { ProxyHandler } from 'aws-lambda'

export const handler: ProxyHandler = async (event) => {
  return {
    body: JSON.stringify({
      message: 'API version 1 has been deployed!',
      path: event.path,
    }),
    headers: { 'Content-Type': 'application/json' },
    statusCode: 200,
  }
}

This is a simple Lambda function that returns a 200 with a message. Now let's look at creating a Stage for our pipeline that will deploy our API.

Stage Stack

We need to define an application stage for our pipeline. A pipeline can have multiple stages like dev, staging, and production. In this case, we will define a staging stage.



// lib/stages.ts

import * as cdk from '@aws-cdk/core'
import { ApiStack } from './api-stack'

export class StagingStage extends cdk.Stage {
  constructor(scope: cdk.Construct, id: string, props?: cdk.StageProps) {
    super(scope, id, props)

    new ApiStack(this, 'ApiStackStaging')
  }
}

We create a new stage named StagingStage and create our an instance of the ApiStack here. This stage will be bootstrapping our API and Lambda function and we will use this stage in our pipeline.

CDK Pipelines

Let's start by creating our CDK Pipeline that will contain values for our repo, artifacts, and our synth step.



// lib/pipeline-stack.ts

const sourceArtifact = new codepipeline.Artifact()
const cloudAssemblyArtifact = new codepipeline.Artifact()

const pipeline = new pipelines.CdkPipeline(this, 'deployApi', {
  cloudAssemblyArtifact,
  sourceAction: new codepipelineActions.GitHubSourceAction({
    actionName: 'GH',
    output: sourceArtifact,
    oauthToken: cdk.SecretValue.secretsManager('github-token'),
    owner: 'ryands17',
    repo: 'lambda-canary-deployments',
    branch: 'main',
  }),
  synthAction: pipelines.SimpleSynthAction.standardYarnSynth({
    cloudAssemblyArtifact,
    sourceArtifact,
  }),
})

Let's break this down:

First we have our artifacts that will be stored in S3.
Then we specify a GitHubSourceAction with the above created sourceArtifact, oAuthToken that we created as a prerequisite, repo, owner, and branch that CodePipeline will pull from.
Finally, we specify a synth action and here CDK automatically provides us with a standardYarnSynth that installs the dependencies and runs the synth command to create the corresponding CloudFormation template. If you're using NPM, you need to use standardNpmSynth.

Moving on, let's add the Staging stage to this pipeline.



// lib/pipeline-stack.ts

const stagingStage = new StagingStage(this, 'staging', {
  env: { region: process.env.region || 'us-east-2' },
})

pipeline.addApplicationStage(stagingStage)

We create an instance of our StagingStage and add it to our pipeline using the addApplicationStage method. This will deploy our REST API (ApiStack) that we created in the StagingStage.

Deploying the app

We're done with the constructs. Now let's deploy the app using yarn cdk deploy.

Note: If you're using your own repository for this instead of mine, then you need to first push this code to your repo and then run yarn cdk deploy otherwise it won't find your repository.

After deploying, we can see the pipeline being run for the first time.

After this is completed, head over to CloudFormation and fetch the API Gateway URL from the Outputs section of your stack.

On opening this, we see the message we sent from our Lambda successfully.

Let's change the message in our Lambda to API version 2. On performing a commit and push, we can see that CodePipeline automatically fetches the source and continues with the pipeline.

On checking our Lambda function, we can see that the alias is performing a weighted routing to our current version and the newly deployed one. If you try the API URL in your browser, you will see both messages, API version 1 and API version 2 on refreshing multiple times.

Here version 1 is our current version (API version 1) and version 2 is our newly deployed version (API version 2).

We can see that our Deployment Group shifted the traffic successfully after 5 minutes as there were no errors.

Finally, let's simulate an error by adding an explicit error to the function in the hopes of triggering our CloudWatch alarm:



// functions/apiHandler.ts
import { ProxyHandler } from 'aws-lambda'

export const handler: ProxyHandler = async (event) => {
  if (Math.random() > 0.5) throw Error('an unexpected error occured!')

  return {
    body: JSON.stringify({
      message: 'API version 2 has been deployed!',
      path: event.path,
    }),
    headers: { 'Content-Type': 'application/json' },
    statusCode: 200,
  }
}

On pushing this code, we can see that the pipeline is triggered, and now we shall load test our API using a tool called artillery.



artillery quick -c 30 -n 100 -d 10 $API_URL

As you can see, a lot of 502 responses from the API. Let's check on our alarm now.

Voila! The alarm is triggered due to Lambda erroring out. On checking CodePipeline, we can see that the deployment failed and our original API version 2 is back. Let's run artillery again to see if our API works.

And we get all 200! Let's fix the nasty error and commit updating the message to API version 3. This will again run the pipeline and the message API version 3 will be displayed after a successful deployment.

When not to Canary

I had a discussion with Sheen Brisals about a point where Canary deployments are not recommended and that is when you're updating Lambda permissions.

In this case, we don't want to have a state where there's a permission mismatch and errors due to this will always trigger the alarm and rollback.

In this case, it would be better to replace Canary with All at Once in your Deployment Group as follows:



new cd.LambdaDeploymentGroup(this, 'canaryDeployment', {
  alias: stage,
  // deploymentConfig: cd.LambdaDeploymentConfig.CANARY_10PERCENT_5MINUTES,

  deploymentConfig: cd.LambdaDeploymentConfig.ALL_AT_ONCE,
  alarms: [failureAlarm],
})

So whenever there's a configuration change i.e. change in IAM permissions, you perform an ALL_AT_ONCE deployment and switch to Canary for the next deployment.

Conclusion

Here's the repo again for those who haven't checked it out yet.

ryands17 / lambda-canary-deployments

API Gateway and Lambda with weighted routing to the latest function deployed

Also don't forget to destroy the stack using yarn cdk destroy and also delete the StagingStack from the CloudFormation console to not incur extra charges.

And we're done! Thanks for reading this and I would love to hear your thoughts on this in the comments! If you liked this post, do give it a like and share, and follow me on Twitter. Until next time!

Magic links 🪄 with Cognito using the CDK

Ryan Dsouza — Mon, 02 Aug 2021 17:47:55 +0000

This is a post on how to perform Passwordless authentication 🪄 using Cognito. The resources will be created using the CDK (TypeScript).

The basic workflow in this would be:

User enters their email on the sign-in page.
We create a user in Cognito and store the secret/auth-challenge along with the creation timestamp.
We construct a link to our webapp with the user's email and secret and email it to them
Finally, when they open the sign-in link, we verify the secret + expiration and authenticate the user.

Prerequisites

To follow along, I would suggest cloning the repo and installing the dependencies using yarn.

ryands17 / passwordless-auth

Allows a user to login directly via email without a need for entering passwords using Cognito

Note: This is a monorepo that has both the backend resources and simple UI to sign-in. This will be located in the packages folder under backend and frontend respectively.

Constructs

Let's have a look at the constructs required to make this work.

Cognito User Pool and App Client

We need to create a Cognito User Pool and App Client that will help us register the users and execute our sign in flow.



// packages/backend/lib/passwordless-login-stack.ts

import * as cg from '@aws-cdk/aws-cognito'

const userPool = new cg.UserPool(this, 'users', {
  standardAttributes: { email: { required: true, mutable: true } },
  customAttributes: {
    authChallenge: new cg.StringAttribute({ mutable: true }),
  },
  passwordPolicy: {
    requireDigits: false,
    requireUppercase: false,
    requireSymbols: false,
  },
  accountRecovery: cg.AccountRecovery.NONE,
  selfSignUpEnabled: true,
  signInAliases: { email: true },
  lambdaTriggers: {
    preSignUp: lambda(this, 'preSignup'),
    createAuthChallenge: lambda(this, 'createAuthChallenge'),
    defineAuthChallenge: lambda(this, 'defineAuthChallenge'),
    verifyAuthChallengeResponse: lambda(this, 'verifyAuthChallenge'),
    postAuthentication,
  },
  removalPolicy: cdk.RemovalPolicy.DESTROY,
})

const webClient = userPool.addClient('webAppClient', {
  authFlows: { custom: true },
})

First, we create a UserPool. This will have a standard attribute email that we will use for signing up and so is required. We have also specified a custom attribute that will be used to store the authentication challenge code along with the current timestamp.

We disable all password policies because we will not be using it. We also disable account recovery and set email as an alias so signing in can be done via the email. There are some lambdaTriggers that we will be discussing in the next section.

For sending emails, you need to verify your email address in SES as we are using SES in Sandbox mode.

We then create a client where we enable a custom auth flow. This is needed because we won't be using a username/password flow but sending a sign-in link which is technically a custom authentication flow in Cognito.

Lambda triggers

Let's have a look at all the triggers used in the lambdaTriggers section:

preSignUp

This function will be called when we signup the user after they enter their email.



// packages/backend/functions/preSignup.ts

export const handler: PreSignUpTriggerHandler = async (event) => {
  event.response.autoConfirmUser = true
  return event
}

What we do here is confirm the user so that we don't need to verify them on signup. Cognito's default flow needs a user to be verified on signup and that would defeat the purpose of a link based sign-in workflow.

createAuthChallenge

This function will create the challenge and set the public and private challenge parameters respectively.



// packages/backend/functions/createAuthChallenge.ts

export const handler: CreateAuthChallengeTriggerHandler = async (event) => {
  // This is sent back to the client app
  event.response.publicChallengeParameters = {
    email: event.request.userAttributes.email,
  }

  // Add the secret login code to the private challenge parameters
  // so it can be verified by the "Verify Auth Challenge Response" trigger
  event.response.privateChallengeParameters = {
    challenge: event.request.userAttributes['custom:authChallenge'],
  }

  return event
}

The public challenge parameter i.e. the email will be sent to the user whereas the private challenge parameter i.e. our secret will be added to the event which will be used to verify from the sign-in link later on.

defineAuthChallenge

This function checks if the parameters for our Passwordless auth (Cognito's custom challenge) are correct.

Note: I haven't included the functions used here to keep the code concise but you can easily view them here or in the file mentioned in the comments if you have cloned the repo :)



// packages/backend/functions/defineAuthChallenge.ts

export const handler: DefineAuthChallengeTriggerHandler = async (event) => {
  if (notCustomChallenge(event)) {
    // We only accept custom challenges; fail auth
    event.response.issueTokens = false
    event.response.failAuthentication = true
  } else if (tooManyFailedAttempts(event)) {
    // The user provided a wrong answer 3 times; fail auth
    event.response.issueTokens = false
    event.response.failAuthentication = true
  } else if (successfulAnswer(event)) {
    // The user provided the right answer; succeed auth
    event.response.issueTokens = true
    event.response.failAuthentication = false
  } else {
    // The user did not provide a correct answer yet; present challenge
    event.response.issueTokens = false
    event.response.failAuthentication = false
    event.response.challengeName = 'CUSTOM_CHALLENGE'
  }

  return event
}

This function checks for four things:

If the challenge was not a custom challenge, then fail the authentication process as our sign-in link only works with a custom challenge.
If the user has already made three attempts at the incorrect answer i.e. if the link was tampered with, fail the auth.
If the answer was successful, issue the tokens which means that the user will login successfully.
If all of the above conditions don't match, it means that the user hasn't got the link yet so set the challenge and wait for the user to click the sign-in link.

Think of this as the pre-validation function. It will check the parameters before we validate the link and return event with the appropriate values.

verifyAuthChallengeResponse

This is the main function that verifies the link opened by the user. We call the sendCustomChallengeAnswer method from the Amplify JS library that would internally call this Lambda.



// packages/backend/functions/verifyAuthChallenge.ts

const MAGIC_LINK_TIMEOUT = 3 * 60 * 1000

export const handler: VerifyAuthChallengeResponseTriggerHandler = async (
  event
) => {
  const [authChallenge, timestamp] = (
    event.request.privateChallengeParameters.challenge || ''
  ).split(',')

  // fail if any one of the parameters is missing
  if (!authChallenge || !timestamp) {
    event.response.answerCorrect = false
    return event
  }

  // is the correct challenge and is not expired
  if (
    event.request.challengeAnswer === authChallenge &&
    Date.now() <= Number(timestamp) + MAGIC_LINK_TIMEOUT
  ) {
    event.response.answerCorrect = true
    return event
  }

  event.response.answerCorrect = false
  return event
}

Here, we set the validity of our link to be 3 minutes. We then check if the authChallenge (secret code) and timestamp are available and fail if they are missing.

In the next step, we verify if the link has the correct code (not tampered in any way) and the user has opened the link in time. If that's valid, we set the answerCorrect property to true which means that the user can successfully sign-in. If not, we will show a nice error message in the UI stating that the sign-in link was invalid.

postAuthentication

This is the final function which will be called after a user has been successfully authenticated.



// packages/backend/functions/postAuthentication.ts

import { CognitoIdentityServiceProvider } from 'aws-sdk'

const cisp = new CognitoIdentityServiceProvider()

export const handler: PostAuthenticationTriggerHandler = async (event) => {
  if (event.request.userAttributes?.email_verified !== 'true') {
    await cisp
      .adminUpdateUserAttributes({
        UserPoolId: event.userPoolId,
        UserAttributes: [
          {
            Name: 'email_verified',
            Value: 'true',
          },
        ],
        Username: event.userName,
      })
      .promise()
  }
  return event
}

We check if the user's email is verified and if it isn't, we verify it using the adminUpdateUserAttributes method from CognitoIdentityServiceProvider.

This was the entire flow of validating the link sent to the user and signing them in without a password.

You must be wondering that we have covered all this but where do we send the email to user with the sign-in link? For this, we will be creating a signIn API using API Gateway and Lambda proxy which will use SES to send an email to the user.

API

Let's start with creating the resources for API Gateway and its method.



// packages/backend/lib/passwordless-login-stack.ts

const api = new apiGw.RestApi(this, 'authApi', {
  endpointConfiguration: { types: [apiGw.EndpointType.REGIONAL] },
  defaultCorsPreflightOptions: { allowOrigins: ['*'] },
  deployOptions: { stageName: 'dev' },
})

const signInMethod = new apiGw.LambdaIntegration(signIn)
api.root.addMethod('POST', signInMethod)

This snippet creates a REST API with Lambda Proxy integration and a function named signIn. This will be added as on the POST method of our endpoint. We will be passing the email in the body of this request.

The Lambda resource would look something like this:



// packages/backend/lib/passwordless-login-stack.ts

const signIn = lambda(this, 'signIn')
  .addEnvironment('SES_FROM_ADDRESS', process.env.SES_FROM_ADDRESS)
  .addEnvironment('USER_POOL_ID', userPool.userPoolId)

signIn.addToRolePolicy(
new iam.PolicyStatement({
  effect: iam.Effect.ALLOW,
  actions: ['ses:SendEmail'],
  resources: ['*'],
})
)

signIn.addToRolePolicy(
new iam.PolicyStatement({
  effect: iam.Effect.ALLOW,
  actions: ['cognito-idp:AdminUpdateUserAttributes'],
  resources: [userPool.userPoolArn],
})
)

Here, we are creating a function that needs two environment variables, one for sending the email and the other for updating the authChallenge (secret) in our user's custom attribute.

Due the these operations, we would also need to add permissions for SES and Cognito. The permissions sendEmail and AdminUpdateUserAttributes are for sending the email and updating the custom attribute respectively.

Now let's have a look at what the Lambda function contains:



// packages/backend/functions/signIn.ts

const cisp = new CognitoIdentityServiceProvider()
const ses = new SES({ region: process.env.AWS_REGION })

export const handler: APIGatewayProxyHandler = async (event) => {
  try {
    const { email } = JSON.parse(event.body || '{}')
    if (!email) throw Error()

    // set the code in custom attributes
    const authChallenge = randomDigits(6).join('')
    await cisp
      .adminUpdateUserAttributes({
        UserAttributes: [
          {
            Name: 'custom:authChallenge',
            Value: `${authChallenge},${Date.now()}`,
          },
        ],
        UserPoolId: process.env.USER_POOL_ID,
        Username: email,
      })
      .promise()

    await sendEmail(email, authChallenge)

    return {
      statusCode: 200,
      headers: {
        'Access-Control-Allow-Origin': '*',
      },
      body: JSON.stringify({
        message: `A link has been sent to ${email}`,
      }),
    }
  } catch (e) {
    console.error(e)
    return {
      statusCode: 400,
      headers: {
        'Access-Control-Allow-Origin': '*',
      },
      body: JSON.stringify({
        message: `Couldn't process the request. Please try after some time.`,
      }),
    }
  }
}

const BASE_URL = `http://localhost:3000/verify`

async function sendEmail(emailAddress: string, authChallenge: string) {
  const MAGIC_LINK = `${BASE_URL}?email=${emailAddress}&code=${authChallenge}`

  const html = `
    <html><body>
    <p>Here's your link:</p>
    <h3>
      <a target="_blank" rel="noopener noreferrer" href="${MAGIC_LINK}">Click to sign-in</a>
    </h3>
    </body></html>
  `.trim()

  const params: SES.SendEmailRequest = {
    Destination: { ToAddresses: [emailAddress] },
    Message: {
      Body: {
        Html: {
          Charset: 'UTF-8',
          Data: html,
        },
        Text: {
          Charset: 'UTF-8',
          Data: `Here's your link (copy and paste in the browser): ${MAGIC_LINK}`,
        },
      },
      Subject: {
        Charset: 'UTF-8',
        Data: 'Login link',
      },
    },
    Source: process.env.SES_FROM_ADDRESS,
  }
  await ses.sendEmail(params).promise()
}

First, we create a cryptographically random secret that will be inserted in the sign-up link. We set the secret and the creation timestamp in the user's custom attribute. Finally, we call the sendEmail method that accepts the secret and email of the user trying to sign in.

We created a BASE_URL constant that will actually to be the link to our webapp set to localhost currently for development. We then construct the magic link as follows:



const MAGIC_LINK = `${BASE_URL}?email=${emailAddress}&code=${authChallenge}`

This will append the email and code as querystring params which we be fetching in our webapp to validate the sign-in request. The final part is just constructing the email to be sent where we embed the link and the user can directly click the link to open it in the browser.

Deploying the stack

The final step to add in our stack is outputs that our webapp will use to perform authentication against the User Pool. This is done using the Amplify JS library.



new cdk.CfnOutput(this, 'userPoolId', {
  value: userPool.userPoolId,
})

new cdk.CfnOutput(this, 'clientId', {
  value: webClient.userPoolClientId,
})

We set the userPoolId and clientId to be used by Amplify. CDK automatically outputs the API Gateway endpoint URL so we don't need to add it explicitly. Sweet!

Now let's deploy the stack! As we are using workspaces, we need to run the deploy command in the backend workspace.



yarn workspace backend cdk deploy

As an alternative, you can also move to the backend directory and simply run yarn cdk deploy.

This will deploy all our resources and create a JSON file with the outputs in our frontend/src folder. We generate the outputs file in the frontend/src folder on purpose as we need to use this in our React app. This app is created using create-react-app.

Note: You will need to edit the URL key the api file before proceeding further as CDK will generate a new resource name for your stack.

Testing the login flow

After successful deployment, we can run the webapp locally to test the login flow using the following command:



yarn workspace frontend dev

This will fire up the app on http://localhost:3000/ and will redirect to the signIn page as we aren't currently authenticated.

On adding our email address here, we will get a link on our email address.

Here's the link I received that includes the email and authChallenge (secret).

On clicking the link, a verify page is opened that will perform the sign-in and call the sendCustomChallengeAnswer method which will successfully verify the user and be redirected to the home page :)

Conclusion

So that was it! This is how we can perform a truly Passwordless sign-in with Cognito. Here's the repo again for those who would like to experiment:

ryands17 / passwordless-auth

Allows a user to login directly via email without a need for entering passwords using Cognito

Also don't forget to destroy this stack so that you do not incur unnecessary charges using the following command:



yarn workspace backend cdk destroy

The resources I used to help create this are:

https://aws.amazon.com/blogs/mobile/implementing-passwordless-email-authentication-with-amazon-cognito/

https://schof.co/cognito-magic-links/

Thanks for reading this and I would love to hear your thoughts on this in the comments! If you liked this post, do give it a like and share, and follow me on Twitter. Until next time!

Sentiment Analysis with Step Functions using the CDK

Ryan Dsouza — Fri, 23 Jul 2021 18:40:43 +0000

In this walkthrough, we will learn to perform sentiment analysis on user feedback with Step Functions and API Gateway created with CDK (TypeScript).

The example is taken from the docs in which a user submits a feedback form to an endpoint created by API Gateway and we fire a Synchronous Step function that performs the following steps:

Detect the sentiment of user feedback using Amazon Comprehend.
Generate a unique reference ID for the feedback.
Saves the feedback form data to DynamoDB.
Checks if the sentiment is positive or negative and if negative, sends an email to the desired email address with the feedback and sentiment.

We will walk through all the resources created with the CDK and finally see the API in action! We will skip the actual frontend feedback form and interact directly with the API.

Here's the repo that you can clone for this walkthrough:

ryands17 / step-function-sync-workflow

Synchronous Express Workflows for AWS Step Functions via API Gateway

Step Function Resources

Let's look at the resources created for the step function that performs sentiment analysis on text provided as input. Our final step function workflow will look something like this:

The first part in our step function is detectSentiment where we use Comprehend in a Lambda to detect if the feedback was positive or negative.

// lib/sync-sf-stack.ts

const detectSentimentFn = createFn(this, 'detectSentimentFn')
detectSentimentFn.addToRolePolicy(
  new iam.PolicyStatement({
    effect: iam.Effect.ALLOW,
    actions: ['comprehend:DetectSentiment'],
    resources: ['*'],
  })
)

const detectSentiment = new tasks.LambdaInvoke(this, 'detectSentiment', {
  lambdaFunction: detectSentimentFn,
  resultPath: '$.sentimentResult',
})

Here, we create an aws-lambda-nodejs function via a helper named createFn. We then add a role stating that it has permissions to detect sentiment using Comprehend. Finally, we create a new LambdaInvoke task that accepts this function and stores the result in a key named sentimentResult.

We use resultPath to store each step's results on the final result object which makes it easier to pass between steps and perform custom processing.

Note: Step functions automatically stores the result of each step in a key named Payload under the key that we specify. For e.g. If we set the result path as $.data, then the output of that step will be available under:

{
  "data": {
    "Payload": "result-here"
  }
}

Here's the function for detecting sentiment:

// functions/detectSentimentFn.ts

import { Comprehend } from 'aws-sdk'

const cp = new Comprehend({ apiVersion: '2017-11-27' })

type Event = {
  message: string
}

export const handler = async (event: Event) => {
  const data = await cp
    .detectSentiment({
      LanguageCode: 'en',
      Text: event.message,
    })
    .promise()

  return data
}

This uses Comprehend's detectSentiment on the feedback that we receive from our endpoint. We will be looking at how to send feedback from the created API in the final section.

The next step is to generate a unique reference ID for the feedback message. This will then be stored as a record along with the message.

// lib/sync-sf-stack.ts

const generateReferenceNumber = new tasks.LambdaInvoke(
  this,
  'generateReferenceNumber',
  {
    lambdaFunction: createFn(this, 'generateReferenceNumberFn'),
    resultPath: '$.ticketId',
  }
)

Here we create a LambdaInvoke task that will generate a reference ID for the feedback and store it in a key named ticketId. The function simply does this:

// functions/generateReferenceNumberFn.ts
import { ulid } from 'ulid'

export const handler = async () => {
  return ulid()
}

This function returns a unique identifier using ulid that will be used as the feedback reference ID.

Moving to the next step, we save the feedback data along with the reference ID to a DynamoDB table.

// lib/sync-sf-stack.ts

const formData = new ddb.Table(this, 'formData', {
  partitionKey: { name: 'formId', type: ddb.AttributeType.STRING },
  billingMode: ddb.BillingMode.PAY_PER_REQUEST,
  removalPolicy: cdk.RemovalPolicy.DESTROY,
})

const saveCustomerMessage = new tasks.DynamoPutItem(
  this,
  'saveCustomerMessage',
  {
    table: formData,
    item: {
      formId: tasks.DynamoAttributeValue.fromString(
        sfn.JsonPath.stringAt('$.ticketId.Payload')
      ),
      customerMessage: tasks.DynamoAttributeValue.fromString(
        sfn.JsonPath.stringAt('$.message')
      ),
      sentiment: tasks.DynamoAttributeValue.fromString(
        sfn.JsonPath.stringAt('$.sentimentResult.Payload.Sentiment')
      ),
    },
    resultPath: '$.formDataRecord',
  }
)

First, we create a DynamoDB table with our ticket ID being the partition key.

Then we use a DynamoPutItem task as Step functions can directly interact with DynamoDB without a need for a Lambda. This will save a record in the table we specify with the following values:

The formId which is taken from the output path stored in ticketId.
The customerMessage that directly comes from the frontend.
The sentiment result obtained from the very first step.

Note: The sfn.JsonPath extracts the value present currently in the output of our Step Function workflow.

Our next step is to check for the feedback's sentiment. If the sentiment is negative, we would like to send an email for analytics stating the message and sentiment.

// lib/sync-sf-stack.ts

const notifyOfNegativeSentimentFn = createFn(
  this,
  'notifyOfNegativeSentimentFn'
)
  .addEnvironment('SENDER', process.env.SENDER)
  .addEnvironment('RECEIVER', process.env.RECEIVER)

notifyOfNegativeSentimentFn.addToRolePolicy(
  new iam.PolicyStatement({
    effect: iam.Effect.ALLOW,
    actions: ['ses:SendEmail'],
    resources: ['*'],
  })
)

const checkSentiment = new sfn.Choice(this, 'checkSentiment')
.when(
  sfn.Condition.stringEquals(
    '$.sentimentResult.Payload.Sentiment',
    'NEGATIVE'
  ),
  new tasks.LambdaInvoke(this, 'notifyOfNegativeSentiment', {
    lambdaFunction: notifyOfNegativeSentimentFn,
    resultPath: '$.notifyViaEmail',
  })
)
.otherwise(new sfn.Succeed(this, 'positiveSentiment'))

What we did here is use a Choice state that checks for a specific condition and performs operations when that condition matches. Think of this as a branching (if/else) statement.

Here we check if the sentiment that we calculated in the very first step is NEGATIVE. If so, we initiate a Lambda function that sends an email via AWS SES to the intended recipient. If the sentiment is not NEGATIVE, do nothing and succeed.

This is where the CDK shines as we get to develop our workflow using simple methods like when and otherwise.

The lambda function created above adds two environment variables SENDER and RECEIVER. These need to be verified in SES for now so that you can send/receive messages.

We have also allowed Lambda access to send an email using SES and it uses SESV2 from aws-sdk to send an email.

// functions/notifyOfNegativeSentimentFn.ts

import { SESV2 } from 'aws-sdk'

const ses = new SESV2({ region: process.env.AWS_REGION })

export const handler = async (event: any) => {
  const Data = `Sentiment analysis: ${event.sentimentResult.Payload.Sentiment}
Feedback from customer: ${event.message}`

  await ses
    .sendEmail({
      FromEmailAddress: process.env.SENDER,
      Destination: {
        ToAddresses: [process.env.RECEIVER],
      },
      Content: {
        Simple: {
          Subject: { Charset: 'UTF-8', Data: 'Feedback form submission' },
          Body: { Text: { Charset: 'UTF-8', Data } },
        },
      },
    })
    .promise()

  return {
    body: 'Feedback submitted successfully!',
  }
}

This uses the values calculated from previous steps and constructs an email to send to the configured RECEIVER.

Finally, we draft the Step Function workflow definition:

// lib/sync-sf-stack.ts

const definition = detectSentiment
  .next(generateReferenceNumber)
  .next(saveCustomerMessage)
  .next(checkSentiment)

this.sentimentAnalysis = new sfn.StateMachine(this, 'sentimentAnalysis', {
definition,
stateMachineType: sfn.StateMachineType.EXPRESS,
timeout: cdk.Duration.seconds(30),
logs: {
  destination: new logs.LogGroup(this, 'sentimentAnalysisLogs', {
    retention: logs.RetentionDays.ONE_WEEK,
  }),
},
})

formData.grantWriteData(this.sentimentAnalysis)

The definition here maps the entire workflow of the step function and will look like this as we saw before.

Then we pass this definition to a StateMachine we named sentimentAnalysis and defined the type of the State Machine to be EXPRESS.

Note: The reason we chose EXPRESS is that Synchronous workflows can only be performed on Express workflows and not Standard.

Lastly, we grant write permissions to our State Machine on our DynamoDB table to insert our feedback message along with the sentiment.

This concludes our Step Function setup. Let's move on to creating an API that we can interact with using a client like Insomnia.

API Gateway Resources

Before we start with the resources in the API Gateway CDK stack, we shall see how to reference the State Machine created above in this stack.

// bin/sync-sf.ts

const app = new cdk.App()
const env = { region: process.env.CDK_REGION || 'us-east-1' }

const sfn = new SyncSfStack(app, 'SyncSfStack', { env })
new ApiStack(app, 'ApiGwStack', { env, sfn: sfn.sentimentAnalysis })

We create the SyncSfStack that includes the State Machine and pass it as props to the ApiStack which contains the API Gateway resource.

Moving on to the resources for the API, let's start with creating the API itself.

// lib/api-gw-stack.ts

const apiLogs = new logs.LogGroup(this, 'myApiLogs', {
  removalPolicy: cdk.RemovalPolicy.DESTROY,
  retention: logs.RetentionDays.ONE_WEEK,
})

const api = new apiGw.RestApi(this, 'myApi', {
  endpointTypes: [apiGw.EndpointType.REGIONAL],
  deployOptions: {
    stageName: 'dev',
    loggingLevel: apiGw.MethodLoggingLevel.ERROR,
    accessLogDestination: new apiGw.LogGroupLogDestination(apiLogs),
  },
  defaultCorsPreflightOptions: {
    allowOrigins: ['*'],
  },
})

Here we configure a REST API that we set as a REGIONAL endpoint for this example, but you can use EDGE as well. We also configured CORS to enable access from anywhere and Logging so that we can view the access logs via CloudWatch.

Next, let's create a POST method that accepts a feedback message from the customer.

// lib/api-gw-stack.ts

api.root.addMethod('POST', sfIntegration, {
  operationName: 'Submit Feedback Form',
  requestValidatorOptions: { validateRequestBody: true },
  requestModels: {
    'application/json': new apiGw.Model(this, 'feedbackFormPayload', {
      restApi: api,
      schema: {
        schema: apiGw.JsonSchemaVersion.DRAFT4,
        title: 'Feedback Form Payload',
        type: apiGw.JsonSchemaType.OBJECT,
        required: ['message'],
        properties: {
          message: {
            type: apiGw.JsonSchemaType.STRING,
            minLength: 1,
          },
        },
      },
    }),
  },
  methodResponses: [{ statusCode: '200' }],
})

The above snippet adds a POST method that accepts message in the body. This will be validated as we have specified the requestModels for what our method accepts. This makes sure the data is sanitised before reaching our Step Function.

Note: API Gateway is extremely powerful and request validation is one of its most useful features.

Notice we have added something called sfIntegration to our method. This would be the direct integration from API Gateway to Step Functions.

// lib/api-gw-stack.ts

const sfIntegrationRole = new iam.Role(this, 'asyncApiApigRole', {
  assumedBy: new iam.ServicePrincipal('apigateway.amazonaws.com'),
})
sfIntegrationRole.addToPolicy(
  new iam.PolicyStatement({
    resources: [props.sfn.stateMachineArn],
    actions: ['states:StartSyncExecution'],
  })
)

// Step Functions integration
const sfIntegration = new apiGw.AwsIntegration({
  service: 'states',
  action: 'StartSyncExecution',
  options: {
    credentialsRole: sfIntegrationRole,
    passthroughBehavior: apiGw.PassthroughBehavior.NEVER,
    requestParameters: {
      'integration.request.header.Content-Type': `'application/json'`,
    },
    requestTemplates: {
      'application/json': JSON.stringify({
        input: `$util.escapeJavaScript($input.json('$'))`,
        stateMachineArn: props.sfn.stateMachineArn,
      }),
    },
    integrationResponses: [
      {
        statusCode: '200',
        responseTemplates: {
          'application/json': `
          #set($output = $input.path('$.output'))
          #set($root = $util.parseJson($output))
          {
            "ticketId": "$root.ticketId.Payload",
            "message": "Feedback submitted successfully!"
          }`,
        },
      },
    ],
  },
})

Let's break this snippet down.

First of all, we assign a role to this integration where we allow it to perform a synchronous execution of our State Machine. We passed this resource via props to our ApiStack and we assign the resource's arn specifically to this role.

Next, we configure an AWSIntegration where we specify the service name. states is the service for Step Functions just like sqs is for Simple Queue Service. Then we configure the action to be StartSyncExecution, similar to what we gave it permission for.

Let's now go through the options one by one.

We pass the above created role in credentialsRole to allow this integration access to executing the Step Function. The passthroughBehaviour makes sure that any request that doesn't match the header's Content-Type gets discarded. The requestParameters are parameters we would like to send when we make the API call to our Step Function. The only thing we set here is the Content-Type as Step Functions accepts JSON.

Next, we configure the parameters need to be sent in requestTemplates. We need to send two values in our JSON body.

The input obtained from the frontend with the message. The entire input is escaped properly and sent. This is done using VTL that has utilities to perform such operations. $ here means the entire request body.
The stateMachineArn which we obtain via props from the State Machine stack.

Finally, we set our response to be sent to the frontend using integrationResponses. This also uses VTL to parse the response returned by the Step Function where we return the ticketId along with a success message.

That's all we needed to configure our entire app. Now let's deploy this to see it in action!

Deploying the app

To deploy this entire app, run:

yarn cdk deploy --all

This will deploy the two stacks we created and return the API Gateway endpoint as the output:

Now we'll test this API in Insomnia with a positive feedback and check the output:

It ran successfully and I didn't receive an email as expected.

Let's test this API again but now with a negative feedback:

After the API is completed, I receive an email:

Conclusion

So this was the entire application where the endpoint created by API Gateway directly invokes a Synchronous Step function that checks the sentiment of the review, generates a ticket id, saves it to the database and informs the business in case of a negative review.

Here's the repo again for those who would like to experiment:

ryands17 / step-function-sync-workflow

Synchronous Express Workflows for AWS Step Functions via API Gateway

Also don't forget to destroy this stack so that you do not incur unnecessary charges using the following command:

yarn cdk destroy --all

Thanks for reading this and I would love to hear your thoughts on this in the comments! If you liked this post, do give it a share and follow me on Twitter :)

Rust on Lambda using the CDK

Ryan Dsouza — Mon, 05 Apr 2021 18:19:05 +0000

Introduction

AWS introduced a Rust runtime for Lambda and since then you can directly run Rust code in your Lambda function.

In this post, we will explore how to run a Rust function on Lambda which will be deployed using the CDK.

Prerequisites

AWS CLI configured with your profile. I am using default in this case.
Rust installed to build our function.
Cloning the repo and configuring the build platform to follow along with the post.

ryands17 / rust-lambda

This is a CDK project that deploys a Rust function using the Rust runtime for Lambda. This creates a basic handler that logs some data sent as input

Constructs

In this application, we just have a single construct in which we create a Lambda function with a custom runtime.

// lib/rust-lambda-stack.ts

new lambda.Function(this, 'rust-hello', {
  description: 'Deploying a Rust function on Lambda using the custom runtime',
  code: lambda.Code.fromAsset(
    'resources/target/x86_64-unknown-linux-musl/release/lambda'
  ),
  runtime: lambda.Runtime.PROVIDED_AL2,
  handler: 'not.required',
  environment: {
    RUST_BACKTRACE: '1',
  },
  logRetention: RetentionDays.ONE_WEEK,
})

The above snippet creates a new Lambda function where we pass the code from a specific folder. We will see this folder in the section where we deploy the application.

The runtime here is the Custom Runtime provided by Lambda that helps us run our Rust application.

Our code will be an executable file created by Rust, so we do not need to specify the handler specifically as Lambda will execute the file based on what we are providing to the function. Which is why we provide something random like not.required to handler as that's a required prop.

The rest of the options are the environment and logRetention respectively. The RUST_BACKTRACE variable is specified in environment so that we get the entire stack trace of where the error occurred if there was one.

Let's move on to defining the Rust handler where we install necessary dependencies and create the function.

Rust setup

The Cargo.toml file is where we mention the dependencies required and some other metadata. Think of this as a package.json for Rust.

# resources/Cargo.toml

[dependencies]
lambda_runtime = "0.3.0"
log = "0.4.14"
serde_json = "1.0.64"
simple_logger = "1.11.0"
tokio = {version = "1", features = ["full"]}

[[bin]]
name = "bootstrap"
path = "src/main.rs"

In this file, we specify what dependencies we require to run our application and the lambda_runtime dependency is something that will run the handler that we pass to it. We shall also see the rest of the dependencies used in the function.

In the bin section, we denote the name our executable will have after it's built. We name our executable bootstrap as that's what Lambda expects.

Now let's look at the file which contains the function that will be executed.

// resources/src/main.rs

use lambda_runtime::{handler_fn, Context, Error};
use log::LevelFilter;
use serde_json::{json, Value};
use simple_logger::SimpleLogger;

#[tokio::main]
async fn main() -> Result<(), Error> {
  SimpleLogger::new()
    .with_level(LevelFilter::Info)
    .init()
    .unwrap();

  let func = handler_fn(handler);
  lambda_runtime::run(func).await?;
  Ok(())
}

This is the main function that will be the entrypoint in the Lambda. This will either return an empty result (()) or a Lambda Error as defined in the return type.

We have also added an attribute on top of our main function using the tokio crate that helps us make our main function asynchronous.

Then we initialise a logger via simple_logger that helps us log our messages in a better format. These logs will be stored in CloudWatch logs.

The final lines create the handler via handler_fn and pass in a function which we will look at below and then use the lambda_runtime::run that accepts this handler and runs our function.

Notice the await? after the run call and the async before our main function. You could compare this with the async/await flow that we have in Node.js that allows us to wait for Promises in a synchronous manner. This is possible due to the Future trait in Rust.

Finally, let's look at the handler that we pass above to handler_fn.

// resources/src/main.rs

async fn handler(event: Value, _: Context) -> Result<Value, Error> {
  let message = event["message"].as_str().unwrap_or("world");
  let first_name = event["firstName"].as_str().unwrap_or("Anonymous");

  let response = format!("Hello, {}! Your name is {}", message, first_name);
  log::info!("{}", response);

  Ok(json!({ "response": response }))
}

This is a function that returns a Value that is simply JSON which has a response field. The handler takes in parameters that any normal Lambda function does like event and context.

We use the serde_json library to serialise JSON. The event parameter is of type Value that indicates that it is a valid JSON object that could contain any value.

We set the message and first_name variables to the message and firstName values received from event respectively. We check if the value passed is a string and return the string via as_str() and the unwrap_or is a mechanism where we provide a default value in case the value passed by the user was not a string.

We then create a response via the format! macro that creates a string with the values obtained and logs that to the console. This will be visible in CloudWatch under our function's log group.

The final line returns a response object via the json! macro which constructs a JSON object from the values we pass to it.

Deploying the application

And that was it for our application so let's deploy this app via the command yarn deploy or npm run deploy.

This command just runs cdk deploy but before that it builds our Rust application via the following script:

#!/bin/bash

cd resources
cargo build --release --target x86_64-unknown-linux-musl
(cd target/x86_64-unknown-linux-musl/release && mkdir -p lambda && cp bootstrap lambda/)

This runs the cargo build with the release flag so that our executable is optimised and copies it into a folder named lambda. This lambda folder is referenced when creating our Lambda construct as follows:

// lib/rust-lambda-stack.ts

code: lambda.Code.fromAsset(
  'resources/target/x86_64-unknown-linux-musl/release/lambda'
)

On running yarn deploy we will be able to see our function in the console.

Testing the Lambda

Let's run the function by creating a sample test event.

We pass in the message and firstName and create the test event. Let's invoke this function and check if it has succeeded.

We can see that the response is returned to us correctly and it's logged in the Log Output as well which will also be visible in CloudWatch Logs for this function.

Let's edit the test event and remove the message field.

On invoking the function, we will see that it will take the default value that we specified for the message field and return the output as follows:

Conclusion

This is how we can run our Rust application on Lambda with the help of a custom runtime and deployed using the CDK.

If you have deployed this stack, do not forget to delete this via yarn cdk destroy.

ryands17 / rust-lambda

This is a CDK project that deploys a Rust function using the Rust runtime for Lambda. This creates a basic handler that logs some data sent as input

Here's the repo if you haven't cloned it again and let me know your thoughts in the comments. Thanks for reading!

Using Nested Stacks with AWS CDK

Ryan Dsouza — Fri, 19 Mar 2021 12:05:14 +0000

Intro

CloudFormation Nested Stacks are great when you want to separate logical resources in your stack and reference those resources in other nested stacks.

An example of this would be creating the base resources like the VPC, Subnets, Security Groups in one nested stack and application resources like EC2 instances and Lambda functions in another. We can then reference the resources like the VPC in the EC2 and Lambda stack.

Prerequisites

This post assumes a little bit of familiarity with the CDK.

Also it's required to install the aws-cli and creating a default profile by running aws configure.

To follow with this post, you can clone the repo and also deploy it to see it in action!

ryands17 / cdk-nested-stacks

A basic example of how Nested Stacks work in AWS CDK

I am skipping the imports, to make the snippets shorter but they can be easily viewed in the file mentioned in the snippet.

Constructs

Base Resources

Let's start by creating the stack which will contain the base resources like the VPC and Security Group.



// lib/nested-stacks.ts

class BaseResources extends cdk.NestedStack {
  vpc: ec2.Vpc
  applicationSg: ec2.SecurityGroup

  constructor(scope: cdk.Construct, id: string, props?: cdk.NestedStackProps) {
    super(scope, id, props)

    this.vpc = new ec2.Vpc(this, 'app-vpc', {
      cidr: '10.0.0.0/20',
      natGateways: 0,
      maxAzs: 2,
      enableDnsHostnames: true,
      enableDnsSupport: true,
      subnetConfiguration: [
        {
          cidrMask: 22,
          name: 'public',
          subnetType: ec2.SubnetType.PUBLIC,
        },
        {
          cidrMask: 22,
          name: 'private',
          subnetType: ec2.SubnetType.ISOLATED,
        },
      ],
    })

    this.applicationSg = new ec2.SecurityGroup(this, 'application-sg', {
      vpc: this.vpc,
      securityGroupName: 'application-sg',
    })

    this.applicationSg.addIngressRule(ec2.Peer.anyIpv4(), ec2.Port.tcp(80))
  }
}

Here, we have a class named BaseResources that extends from cdk.NestedStack as this will be the Nested Stack for our main resources. We have created a couple of properties such as vpc and applicationSg which will be used by the other Nested Stack i.e. our application stack.

The first resource that we create is the VPC. We have specified the CIDR 10.0.0.0/20 and deployed two Public and two Isolated subnets.
- The advantage of using this construct is that the public subnets will automatically be configured with a route to the Internet Gateway (IGW) and be associated to the VPC which we would have need to specified if using plain CloudFormation YAML.
The next resource we create is the Security Group (SG). This SG has Ingress allowed on port 80 as this will be required by our application that created in another nested stack.

Application

Now, we shall create another nested stack that contains our EC2 instance that displays a webpage via Apache.



// lib/nested-stacks.ts

interface AppResourcesProps extends cdk.NestedStackProps {
  vpc: ec2.Vpc
  applicationSg: ec2.SecurityGroup
}

class AppResources extends cdk.NestedStack {
  constructor(scope: cdk.Construct, id: string, props: AppResourcesProps) {
    super(scope, id, props)

    // The EC2 instance using Amazon Linux 2
    const instance = new ec2.Instance(this, 'simple-server', {
      vpc: props.vpc,
      instanceName: 'simple-server',
      instanceType: ec2.InstanceType.of(
        ec2.InstanceClass.T2,
        ec2.InstanceSize.MICRO
      ),
      machineImage: ec2.MachineImage.latestAmazonLinux({
        generation: ec2.AmazonLinuxGeneration.AMAZON_LINUX_2,
      }),
      vpcSubnets: { subnetType: ec2.SubnetType.PUBLIC },
      securityGroup: props.applicationSg,
    })

    // Display a simple webpage
    instance.addUserData(
      'yum install -y httpd',
      'systemctl start httpd',
      'systemctl enable httpd',
      'echo "<h1>Hello World from $(hostname -f)</h1>" > /var/www/html/index.html'
    )

    // Add the policy to access EC2 without SSH
    instance.role.addManagedPolicy(
iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonSSMManagedInstanceCore')
    )
  }
}

We create another class AppResources that extends cdk.NestedStack and we also specify an interface of the values that we will be receiving from the BaseResources stack.

The first snippet creates an EC2 instance from an Amazon Linux 2 image that it automatically selects. For this instance, we pass the VPC it belongs to and we will see in the final part how we connected these nested stacks.
- We have then specified it to be a type of t2.micro and mentioned the Security Group created in the base stack.
- Lastly, we make sure that this instance is deployed in a public subnet by specifying the vpcSubnets property as we need to view the application we will install on this next.
The next snippet adds some User Data to this instance. This installs httpd and echoes some text in an h1 tag that will be displayed when we visit the public IP of this instance.
The final snippet adds the AmazonSSMManagedInstanceCore managed policy that will allow us to SSH into the instance without any KeyPair or allowing inbound SSH access.

Final resource

We need to combine these nested stacks and for that we need a parent stack. Let's create the final resource needed to make this work.



// lib/nested-stacks.ts

export class MainApp extends cdk.Stack {
  baseResources: BaseResources
  appResources: AppResources

  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props)

    this.baseResources = new BaseResources(this, 'base-resources')
    const { vpc, applicationSg } = this.baseResources

    this.appResources = new AppResources(this, 'app-resources', {
      vpc,
      applicationSg,
    })

    this.appResources.addDependency(this.baseResources)
  }
}

Here, we create instances of both the BaseResources and AppResources nested stack respectively.

In the AppResources stack, we pass the vpc and applicationSg from the baseResource instance to the appResources props so in this way, the EC2 instance will be able to access the VPC and Security Group.
We have also stated that appResources has a dependency on baseResources as we need the VPC and Security Group from BaseResources for the instance to reference.

Deploying the stack

And we're done! Now let's deploy this stack with the following command to see it in action.



yarn cdk deploy

During deployment, if you open the CloudFormation console, you will see three stacks, one would be the main stack and the other two would be the base and application stack respectively.

Also note that the BaseResources stack will be created first due to the dependency we created via the AppResources stack.

After deployment, let's go to our instance and open it via its public IP. We will be greeted with the following text that we added via the User Data.

Conclusion

In this post, we saw how we can use Nested Stacks to separate our resources and how we can pass resources from one nested stack to another. Here's the repo again for anybody who hasn't viewed it yet!

ryands17 / cdk-nested-stacks

A basic example of how Nested Stacks work in AWS CDK

Do not forget to destroy this stack after you're done via:



yarn cdk destroy

Thank you all for reading and let me know your thoughts in the comments :)

Bundling Prisma with the CDK using aws-lambda-nodejs

Ryan Dsouza — Thu, 11 Mar 2021 13:56:39 +0000

Introduction

In this post, we shall see how we can bundle Prisma using aws-lambda-nodejs with the CDK.

Prerequisites

This is a walkthrough of the steps required to bundle Prisma with CDK and so cloning the repo below and following along would be ideal.

ryands17 / prisma-lambda

Integrating Prisma with `aws-lambda-nodejs` to run Prisma on Lambda

Constructs

In this project, we have a simple Lambda function that uses Prisma to query data from a User table and logs to the console. The only resource we create here is the Lambda function so let's look at that.

I'm skipping the imports in these snippets to make it smaller but if you want to view those you can check those out in the file mentioned in the comments.

// lib/prisma-lambda-stack.ts

new ln.NodejsFunction(this, 'prisma', {
  runtime: lambda.Runtime.NODEJS_14_X,
  handler: 'handler',
  entry: lambdaFn,
  timeout: cdk.Duration.seconds(10),
  environment: {
    DB_URL: process.env.DB_URL || '',
  },
  bundling: {
    nodeModules: ['@prisma/client', 'prisma'],
    commandHooks: {
      beforeBundling(inputDir: string, outputDir: string): string[] {
        return []
      },
      beforeInstall(inputDir: string, outputDir: string) {
        return [`cp -R ../prisma ${outputDir}/`]
      },
      afterBundling(inputDir: string, outputDir: string): string[] {
        return [
          `cd ${outputDir}`,
          `yarn prisma generate`,
          `rm -rf node_modules/@prisma/engines`,
        ]
      },
    },
  },
})

Here, we create a simple Node.js Lambda function and the parameters like runtime, handler, entry, timeout and environment are the basic Lambda parameters that we pass.

aws-lambda-nodejs uses esbuild under the hood so your function is bundled using esbuild. There are a couple of important parameters that we need to set for the bundling process.

The first is nodeModules. This will instruct esbuild to not include @prisma/client and prisma into our function file and treat them as node_modules.
The other section to be configured is commandHooks. The main hooks we need here are beforeInstall and afterBundling.
- In beforeInstall, we run the command cp -R ../prisma ${outputDir}/. This makes sure we have schema.prisma present in the bundle. You can also directly copy schema.prisma instead of the entire directory if you have the migrations folder which is not required in the bundle.
- The other hook we use is afterBundling. This is to generate PrismaClient and to remove the engines folder to reduce function size.

The rest is a simple Prisma setup with a schema.prisma and a User model.

Now when we run cdk synth, the function will be bundled and ready for deployment.

Conclusion

So this is how we can package Prisma with aws-lambda-nodejs.

Here is the repo again for those who haven't checked it out:

ryands17 / prisma-lambda

Integrating Prisma with `aws-lambda-nodejs` to run Prisma on Lambda

Generating video thumbnails with S3 and Fargate using the CDK

Ryan Dsouza — Mon, 01 Mar 2021 07:17:54 +0000

Introduction

This post is inspired from the following write-up which shows how to generate a thumbnail from a video using S3, Lambda and AWS Fargate. We will be creating all these resources using the CDK.

Setup

To follow along, I would suggest cloning the repo and installing the dependencies.

ryands17 / thumbnail-creator

A CDK project to create a thumbnail using Fargate from a video uploaded to S3

CDK Constructs

Let's start with the resources that we need to create to run this. I'm excluding the imports here to make the snippets smaller, but you can easily find them in the above repo.

First, we will create the VPC in which we want to run our Fargate task.

// lib/thumbnail-creator-stack.ts

const vpc = new ec2.Vpc(this, 'serverless-app', {
  cidr: '10.0.0.0/21',
  natGateways: 0,
  maxAzs: 2,
  enableDnsHostnames: true,
  enableDnsSupport: true,
  subnetConfiguration: [
    {
      cidrMask: 23,
      name: 'public',
      subnetType: ec2.SubnetType.PUBLIC,
    },
    {
      cidrMask: 23,
      name: 'private',
      subnetType: ec2.SubnetType.ISOLATED,
    },
  ],
})

This will create our VPC with 4 subnets (2 Public and 2 Isolated) in each availability zone with the cidr specified. We will be launching our Fargate task in public subnets as it requires internet access.

Now, let's create our ECS Cluster that our Fargate task will run in, and the S3 bucket which will be storing our uploaded videos and the generated thumbnail.

// lib/thumbnail-creator-stack.ts

const cluster = new ecs.Cluster(this, 'FargateCluster', { vpc })

const imagesBucket = new s3.Bucket(this, bucketName, {
  removalPolicy: cdk.RemovalPolicy.DESTROY,
  autoDeleteObjects: true,
})

The first line creates an ECS Cluster and we pass the VPC created above. The next line creates an S3 bucket to store all our videos and thumbnails.

Note: We have specified removalPolicy and autoDeleteObjects to make it easier to delete this stack, but this is not recommended in production as it will cause data loss.

Now come the exciting parts. Let's continue further by creating the task definition and container that will be run when we execute the Fargate task.

// lib/thumbnail-creator-stack.ts

const taskDefinition = new ecs.FargateTaskDefinition(
  this,
  'GenerateThumbnail',
  { memoryLimitMiB: 512, cpu: 256 }
)
imagesBucket.grantReadWrite(taskDefinition.taskRole)

taskDefinition.addContainer('ffmpeg', {
  image: ecs.ContainerImage.fromRegistry(
    'ryands1701/thumbnail-creator:1.0.0'
  ),
  logging: new ecs.AwsLogDriver({
    streamPrefix: 'FargateGenerateThumbnail',
    logRetention: RetentionDays.ONE_WEEK,
  }),
  environment: {
    AWS_REGION: this.region,
    INPUT_VIDEO_FILE_URL: '',
    POSITION_TIME_DURATION: '00:01',
    OUTPUT_THUMBS_FILE_NAME: '',
    OUTPUT_S3_PATH: '',
  },
})

The first line creates a Fargate task definition where we provide the name GenerateThumbnail along with memory and CPU limits which should be enough for generating thumbnails.

Now as we need to read the video from the bucket and write the thumbnail back to the same bucket, we need permissions on that bucket. The line:

imagesBucket.grantReadWrite(taskDefinition.taskRole)

is a construct provided by CDK and is a handy way of saying grant this Task definition's taskRole read and write access to the bucket we created above. This is one of the reasons why I love writing CDK constructs :)

Finally we have a snippet that adds a container to the taskDefinition. This contains 3 parts:

Image: The image that we will be pulling from DockerHub to create the thumbnail. This is an extension of the ffmpeg image and I will explain what's inside this image in a later section.
Logging: This will create a Log group in CloudWatch where we can see the output emitted by our container which is great for debugging.
Environment: There are some environment variables the container needs which are as follows:

AWS_REGION: The region that our S3 bucket is in to download the video and upload the thumbnail.
INPUT_VIDEO_FILE_URL: This is the URL of our video that will be uploaded by the user.
POSITION_TIME_DURATION: The frame of the video at mm:ss required to generate the thumbnail.
OUTPUT_THUMBS_FILE_NAME: The name of the thumbnail.
OUTPUT_S3_PATH: The path of the S3 bucket where the thumbnail will be stored.

Note: Apart from AWS_REGION, all the variables will be provided at runtime by the Lambda function. In our S3 bucket, we will be storing the videos under the video prefix and the thumbnails under the thumbnails prefix.

The next part is creating the Lambda function which the S3 bucket will trigger on object creation.

// lib/thumbnail-creator-stack.ts

const initiateThumbnailGeneration = createLambdaFn(
  this,
  'initiateThumbnailGeneration',
  {
    reservedConcurrentExecutions: 10,
    environment: {
      ECS_CLUSTER_NAME: cluster.clusterName,
      ECS_TASK_DEFINITION: taskDefinition.taskDefinitionArn,
      VPC_SUBNETS: vpc.publicSubnets.map(s => s.subnetId).join(','),
      VPC_SECURITY_GROUP: vpc.vpcDefaultSecurityGroup,
    },
  }
)

This will create the Lambda function using the createLambdaFn helper function that just uses the aws-lambda-nodejs module under the hood. We have specified some environment variables here as well that we need to look at.

ECS_CLUSTER_NAME: We pass the cluster's name that we created above as we need to run the Fargate task in this cluster.
ECS_TASK_DEFINITION: The task definition that we created above which we will be running to generate the thumbnail.
VPC_SUBNETS: I had mentioned before that we will be running our task in a public subnet as it needs internet access. So here we pass our public subnet id's that we fetch from the VPC created above.
VPC_SECURITY_GROUP: We pass the security group that the VPC creates by default as Fargate requires one to run.

Now, this Lambda function needs to run the Fargate task and also needs to pass the role to the task that will be running. For this, we need to add a couple of permissions to the function's role.

// lib/thumbnail-creator-stack.ts

initiateThumbnailGeneration.addToRolePolicy(
  new iam.PolicyStatement({
    effect: iam.Effect.ALLOW,
    actions: ['ecs:RunTask'],
    resources: [taskDefinition.taskDefinitionArn],
  })
)
initiateThumbnailGeneration.addToRolePolicy(
  new iam.PolicyStatement({
    effect: iam.Effect.ALLOW,
    actions: ['iam:Passrole'],
    resources: [
      taskDefinition.taskRole.roleArn,
      taskDefinition.executionRole?.roleArn || '',
    ],
  })
)

The first role allows the function to run the Fargate task that we have created above by passing the taskDefinitionArn to the resources.

The second role is special. This makes sure that when Lambda creates the ECS task and runs it, it has the permission to pass the taskRole and executionRole to the running task. iam:Passrole is a safeguard by AWS that makes sure no resource can pass elevated privileges that it's own.

Note: For those who do not know what iam:Passrole does, here's a great article by Rowan Udell on what the use case for this is and I would highly recommend reading this!

The final snippet is adding a trigger to fire the function we created above whenever a new object is created in S3.

// lib/thumbnail-creator-stack.ts

imagesBucket.addObjectCreatedNotification(
  new s3Notif.LambdaDestination(initiateThumbnailGeneration),
  { prefix: 'videos/', suffix: '.mp4' }
)

We listen for objects being created but with specific constraints. The object should have the videos/ prefix, or as per the S3 console be in the videos folder and it should have an .mp4 extension.

Note: It's needed to have a prefix when you are going to add something back to the same bucket or else the Lambda will trigger infinitely! So the best way is listening for objects in a specific prefix and creating objects in another prefix if working in the same bucket.

Lambda function implementation

Now as we're done with the constructs, let's look at what the Lambda function has. This is available in the functions directory.

First, we define our environment variables required.

// functions/config.ts

export const envs = {
  AWS_REGION: process.env.AWS_REGION,
  ECS_CLUSTER_NAME: process.env.ECS_CLUSTER_NAME,
  ECS_TASK_DEFINITION: process.env.ECS_TASK_DEFINITION,
  VPC_SUBNETS: process.env.VPC_SUBNETS?.split(','),
  VPC_SECURITY_GROUP: process.env.VPC_SECURITY_GROUP,
}

These are all the environment variables needed that we defined while constructing our Lambda function and which we will be passing to the Fargate task.

Next, let's move on to where the function is defined.

// functions/initiateThumbnailGeneration.ts

const ecs = new ECS()

export const handler = async (event: S3Event) => {
  let { bucket, object } = event.Records[0].s3
  let videoURL = `s3://${bucket.name}/${object.key}`
  let thumbnailName = `${object.key.replace('videos/', '')}.png`
  let framePosition = '01:32'

  await generateThumbnail({
    videoURL,
    thumbnailName,
    framePosition,
    bucketName: bucket.name,
  })
}

We initialise the ECS instance and this is our main handler where we get the event (S3Event) in which we get all the details of the uploaded object or video in our case.

In this handler, we extract all the values needed like the bucket name, object key and send them all to the generateThumbnail method which we will view below.

Note: I have provided a static frame position here, but to make this dynamic, it can come from the file name or an external API.

// functions/initiateThumbnailGeneration.ts

const generateThumbnail = async ({
  videoURL,
  thumbnailName,
  framePosition = '00:01',
  bucketName,
}: GenerateThumbnail) => {
  let params: ECS.RunTaskRequest = {
    taskDefinition: envs.ECS_TASK_DEFINITION,
    cluster: envs.ECS_CLUSTER_NAME,
    count: 1,
    networkConfiguration: {
      awsvpcConfiguration: {
        assignPublicIp: 'ENABLED',
        subnets: envs.VPC_SUBNETS,
        securityGroups: [envs.VPC_SECURITY_GROUP],
      },
    },
    launchType: 'FARGATE',
    overrides: {
      containerOverrides: [
        {
          name: 'ffmpeg',
          environment: [
            { name: 'AWS_REGION', value: envs.AWS_REGION },
            { name: 'INPUT_VIDEO_FILE_URL', value: videoURL },
            { name: 'OUTPUT_THUMBS_FILE_NAME', value: thumbnailName },
            { name: 'POSITION_TIME_DURATION', value: framePosition },
            { name: 'OUTPUT_S3_PATH', value: `${bucketName}/thumbnails` },
          ],
        },
      ],
    },
  }
  try {
    let data = await ecs.runTask(params).promise()
    console.log(
      `ECS Task ${params.taskDefinition} started: ${JSON.stringify(
        data.tasks,
        null,
        2
      )}`
    )
  } catch (error) {
    console.error(
      `Error processing ECS Task ${params.taskDefinition}: ${error}`
    )
  }
}

The first block in this function defines the parameters to run the ECS task. The gist of this is that we want to create run just a single instance of this Fargate task with the given task definition inside the cluster that runs the VPC and public subnets obtained from environment variables.

We also add something called containerOverrides where we pass the container name that needs the environment variables. In these variables, we pass all the values we constructed in the Lambda handler above like the uploaded videoURL, the new thumbnailName and framePosition for which we want the thumbnail.

Finally, in the try/catch block, we run our ECS task via ecs.runTask and log the success and error states which will be visible in CloudWatch.

Dockerfile creation

The Dockerfile creation is quite simple. You can either create your own and deploy it on DockerHub or use the one I created available here.

We have two files here that will constitute our image. Let's have a look at the Dockerfile first.

# docker/Dockerfile

FROM jrottenberg/ffmpeg:4.3-ubuntu

RUN apt-get update && \
  apt-get install -y curl unzip && \
  curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

RUN unzip awscliv2.zip && \ 
  ./aws/install && \
  aws --version

WORKDIR /files

COPY ./copy_thumbs.sh /files

ENTRYPOINT ./copy_thumbs.sh

What we do here is use the jrottenberg/ffmpeg image and install the AWS CLI. This is required to download the video and upload the thumbnail that ffmpeg will process.

The last two lines are important here. We copy a certain copy_thumbs.sh file and add it to our ENTRYPOINT that will be executed when our container is run. Let's look at this file now.

# docker/copy_thumbs.sh

#!/bin/bash

echo "Downloading ${INPUT_VIDEO_FILE_URL}..."
aws s3 cp ${INPUT_VIDEO_FILE_URL} video.mp4
ffmpeg -i video.mp4 -ss ${POSITION_TIME_DURATION} -vframes 1 -vcodec png -an -y ${OUTPUT_THUMBS_FILE_NAME}

echo "Copying ${OUTPUT_THUMBS_FILE_NAME} to S3 at ${OUTPUT_S3_PATH}/${OUTPUT_THUMBS_FILE_NAME}..."
aws s3 cp ./${OUTPUT_THUMBS_FILE_NAME} s3://${OUTPUT_S3_PATH}/${OUTPUT_THUMBS_FILE_NAME} --region ${AWS_REGION}

This file firstly downloads the provided file URL as video.mp4. Then we run the ffmpeg binary to create a thumbnail from the video on the frame position we specified and outputs it to the name we specified in OUTPUT_THUMBS_FILE_NAME. Finally, we upload the thumbnail to S3 in our thumbnails prefix/folder and this is again done by the CLI command aws s3 cp.

Deploying the application

As we're done with the constructs and functionality, let's deploy this CDK project and test it out. we need to run yarn cdk deploy or yarn cdk deploy --profile profileName if you are using a different AWS profile.

Now we have deployed the application, you should be able to see a bucket created like this. The name would be different in your environment.

Let's create the videos folder and upload a video to check if our thumbnail generation task is working. I have uploaded quite a famous video used widely for testing that you can use too.

After this upload is done, you should see a success message from S3 in this way:

This should kick off our Lambda function, so let's check the CloudWatch log group for our Lambda function.

On inspecting the logs, we see that the ECS task has run successfully.

Let's also check our Fargate task logs to confirm if the thumbnail has been created.

This means that our task has completed successfully! Finally let's check our S3 bucket and confirm. Voila! The thumbnail has been generated successfully and the output is exactly at the provided frame :)

So this is how we can generate thumbnails for our video in a truly serverless fashion by using a combination of Lambda and Fargate.

Note: Do not forget to delete this stack after completion by running yarn cdk destroy or yarn cdk destroy --profile profileName if you're using a custom profile.

Again the link to the repo if you haven't checked it out yet :)

ryands17 / thumbnail-creator

A CDK project to create a thumbnail using Fargate from a video uploaded to S3

I hope you liked this post, so it would be great if you could like, share, and also provide some feedback if I have missed something. Thanks for reading!

Prisma Migrate with AWS Aurora Serverless

Ryan Dsouza — Tue, 02 Feb 2021 15:17:23 +0000

Introduction

In the previous post, we saw how we could deploy a GraphQL API using AWS AppSync and Prisma. Querying the database with Prisma Client worked great, but there was a minor issue with the overall workflow: We had to create the tables manually in the RDS Query Editor and sync them manually with the Prisma model in schema.prisma.

In this post, we will look at how we can use Prisma Migrate to create our tables with migrations. These migrations then can be directly run on any other setup with a single command.

Before we move forward...

I am assuming that you have read the previous post in this series. We will be adding snippets to the current application that we have and deploy it. It's not necessary to deploy the application beforehand for this and so you would be good to go by just cloning the repo below and following along.

ryands17 / graphql-api-cdk-serverless-postgres

A basic example of AppSync + Lambda resolvers with AWS Aurora Serverless and Prisma

Setup

In this section, we will see how to make our Aurora Serverless database accessible on our local machine via an EC2 instance and SSH.

Intro

Amazon Aurora PostgreSQL Serverless databases are not accessible locally. They can only be accessed from the VPC in which they are deployed. So to add the table to our database locally, we will use a Jump Box (basically an EC2 instance) to forward our local connection to our remote database in the VPC.

Creating an SSH key

For this we first need to create an SSH key. So in the AWS Console, navigate to EC2 and select the Key Pairs section from the left hand side of the menu. We will see a UI like this where we need to select the Create key pair option.

I have already one named "prisma" which I will be using for the rest of this walkthrough.

Next, let's add the name of this key to our cdk.context.json file like this:

"keyName": "prisma"

Adding CDK resources

Now let's add some snippets to our CDK file.

The very first snippet will tell AWS to fetch us the latest Ubuntu Server:

// appsync-cdk-rds-stack.ts

const ami = new ec2.LookupMachineImage({
  name: 'ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*',
  filters: { 'virtualization-type': ['hvm'] },
  // Canonical AWS Account ID
  owners: ['099720109477'],
})

This will search for the latest Ubuntu image (20.04) which we will be using for our EC2 instance and set it to the ami variable.

The next snippet is for allowing SSH access to our EC2 instance and also to make sure that it can access our database.

// appsync-cdk-rds-stack.ts

const publicSg = new ec2.SecurityGroup(this, 'public-sg', {
  vpc,
  securityGroupName: 'public-sg',
})
publicSg.addIngressRule(
  ec2.Peer.anyIpv4(),
  ec2.Port.tcp(22),
  'allow SSH access'
)

privateSg.addIngressRule(
  publicSg,
  ec2.Port.tcp(5432),
  'allow Aurora Serverless Postgres access'
)

We have created a public security group so that we can port forward our database locally via the EC2 instance, which is why we have also opened port 22 on our public security group.

We have also given access to port 5432 which is our database port to our public SG from our private SG so that we can successfully access the database on our local machine.

Creating our EC2 instance (Jump Box)

This last snippet is for creating our EC2 instance assigining all the above parameters:

// appsync-cdk-rds-stack.ts

new ec2.Instance(this, 'jump-box', {
  vpc,
  securityGroup: publicSg,
  vpcSubnets: { subnetType: ec2.SubnetType.PUBLIC },
  instanceType: ec2.InstanceType.of(
    ec2.InstanceClass.T2,
    ec2.InstanceSize.MICRO
  ),
  machineImage: ec2.MachineImage.genericLinux({
    [this.region]: ami.getImage(this).imageId,
  }),
  keyName: this.node.tryGetContext('keyName'),
})

Finally, we create our EC2 instance where we assign it our vpc, public security group and subnets. We have selected a t2.micro instance here to save costs as we just need to port forward our database. Finally, we have assigned the AMI's imageId to the machineImage which means we will be using Ubuntu 20.04 as our OS.

Deploying to AWS

We're done with the setup so let's deploy our current stack using the following command:

yarn deploy

SSH forwarding

After deploying, all our resources including our EC2 instance will be created. Now we shall move on to SSH forwarding. The basic command used will like this:

ssh -N -L 5432:aurora-serverless-hostname:5432 ubuntu@ipAddress -i keyName.pem -v

This command forwards our instance locally to port 5432 so now we can access our Serverless database as if it were present locally i.e. localhost.

By looking at the above command, we need the host name of the newly created database and the IP address of our EC2 instance.

The hostname for our database can be obtained by visiting Secrets Manager from the AWS Console and selecting the secret created.

The IP address of the EC2 Instance can be obtained from the instances menu in the EC2 section of the AWS Console

Now, we can run the SSH command specified above and replace the placeholder values with the ones we found from the steps above. After running this command you will see something like this in the console, which means that our port forwarding setup has been successfully completed.

Let's leave this running as is and move on to the next section.

Creating and applying migrations with Prisma Migrate

Finally we have setup port forwarding so let's move on to creating the table by implementing our migrations with the Prisma CLI.

Currently our schema.prisma file look something like this:

generator client {
  provider      = "prisma-client-js"
  binaryTargets = ["native", "rhel-openssl-1.0.x"]
}

datasource db {
  provider = "postgresql"
  url      = env("DB_URL")
}

model Post {
  id      String  @id
  title   String?
  content String?

  @@map("posts")
}

Looking at the schema.prisma file we have in our lambda-fns/prisma directory, we can see that we already have the model Post and our database URL in the datasource block which would need to be specified via the DB_URL environment variable.

So let's create an .env file in the lambda-fns directory and the placeholder values would look something like this:

DB_URL="postgresql://username:password@localhost:5432/BlogDB"

In the above snippet, we only need to replace the username and password which can be obtained in the same place we fetched the database host i.e. from our secret created in Secrets Manager.

Let's run yarn db:save. This command invokes the Prisma CLI's prisma migrate dev --create-only command that will create our migration stored in an SQL file. Which will look something like this after you accept the defaults in the prompt:

And the created SQL migration:

-- CreateTable
CREATE TABLE "posts" (
    "id" TEXT NOT NULL,
    "title" TEXT,
    "content" TEXT,

    PRIMARY KEY ("id")
);

Now to create our table in the database, let's run yarn db:migrate which will again invoke Prisma CLI's prisma migrate dev which will run the SQL migrations created onto the database and our table will be created.

We can verify this by going to RDS in the AWS Console and selecting Query Editor from the left menu.

Selecting our secrets and entering the database name, we can now execute the following query:

select * from "posts"

And this will give us the following output:

Yayy! We can finally see the database created with the migrations via Prisma Migrate! You can now play around with the AppSync API in a similar manner as the previous post.

Conclusion

In this post, we created our database migrations with Prisma Migrate and applied them on our Aurora Serverless database using an EC2 instance with SSH port forwarding that helps connect to our database which is not publicly accessible otherwise.

Here's the link to the repo again for those who want to dive in.

ryands17 / graphql-api-cdk-serverless-postgres

A basic example of AppSync + Lambda resolvers with AWS Aurora Serverless and Prisma

Thanks a lot for reading this post, and if you have implemented this stack, please DO NOT FORGET TO DELETE THIS STACK via

yarn cdk destroy

Deploy a GraphQL API with Prisma, AWS AppSync, Aurora Serverless & CDK

Ryan Dsouza — Mon, 21 Dec 2020 10:18:16 +0000

Intro

Recently, Nader Dabit performed a livestream of using AWS AppSync with an Amazon Aurora PostgreSQL Serverless database.

I replicated the same setup using Prisma as the developer experience that you get while writing queries is quite amazing and also fits in perfectly with TypeScript. The entire stack is composed of the following components:

Here is a high level overview of the architecture that we will be deploying:

Before we move forward...

Here's the entire repository for those who directly want to dive in! To follow along with this blog post, open the link to the repo below and implement points 1-4.

ryands17 / graphql-api-cdk-serverless-postgres

A basic example of AppSync + Lambda resolvers with AWS Aurora Serverless and Prisma

This repo also contains a frontend that uses Next.js and AWS Amplify, but this article is just about deploying the backend.

This tutorial is an explanation for all the resources created. You do not need to create any of these snippets as I have provided a working repo above that you can directly deploy. You can clone the repo, follow along, and also make changes as you see fit.

The following stack that we will be deploying is NOT FREE. This stack will incur costs so please do not leave this running and delete it after you're done. If you do not want to incur any charges then feel free just to follow along by cloning the repo and understanding how this works rather than deploying the entire stack.

To delete the stack, run the following command at the end of the tutorial.

yarn cdk destroy

Prerequisites

This post assumes a little bit of familiarity with Prisma and the AWS CDK.

Also it's required to install the aws-cli and creating a default profile by running aws configure.

Once the repo has been cloned, create a cdk.context.json file in the root of this project and specify region and accountID which will contain the region and AWS account respectively where the stack will be deployed.as follows:

{
  "region": "us-east-1",
  "accountID": "123456789012"
}

The accountID is a placeholder value so it needs to be replaced with your AWS Account ID.

Getting started

Let's start by creating the basic resources we require for deploying our application. All the snippets for creating resources are located in lib/appsync-cdk-rds-stack.ts.

Here we create an AppSync GraphQL API from a schema.graphql file. This contains our Query, Mutation and Subscription types, basically CRUD for posts.

// lib/appsync-cdk-rds-stack.ts

const api = new appsync.GraphqlApi(this, 'Api', {
  name: 'cdk-blog-appsync-api',
  schema: appsync.Schema.fromAsset('graphql/schema.graphql'),
  authorizationConfig: {
    defaultAuthorization: {
      authorizationType: appsync.AuthorizationType.API_KEY,
      apiKeyConfig: {
        expires: cdk.Expiration.after(cdk.Duration.days(365)),
      },
    },
  },
})

And this is the referenced GraphQL file:

# graphql/schema.graphql

type Post {
  id: String!
  title: String!
  content: String!
}

input CreatePostInput {
  id: String
  title: String!
  content: String!
}

input UpdatePostInput {
  id: String!
  title: String
  content: String
}

type Query {
  listPosts: [Post]
  getPostById(postId: String!): Post
}

type Mutation {
  createPost(post: CreatePostInput!): Post
  deletePost(postId: String!): Post
  updatePost(post: UpdatePostInput!): Post
}

type Subscription {
  onCreatePost: Post @aws_subscribe(mutations: ["createPost"])
  onUpdatePost: Post @aws_subscribe(mutations: ["updatePost"])
  onDeletePost: Post @aws_subscribe(mutations: ["deletePost"])
}

The following snippet creates a VPC, and a SecurityGroup and a SubnetGroup for our Aurora database cluster. This database will not be publicly accessible so only resources created inside the VPC (like Lambda functions) will be able to access the database.

// lib/appsync-cdk-rds-stack.ts

const vpc = new ec2.Vpc(this, 'BlogAppVPC', {
  cidr: '10.0.0.0/20',
  natGateways: 0,
  maxAzs: 2,
  enableDnsHostnames: true,
  enableDnsSupport: true,
  subnetConfiguration: [
    {
      cidrMask: 22,
      name: 'public',
      subnetType: ec2.SubnetType.PUBLIC,
    },
    {
      cidrMask: 22,
      name: 'private',
      subnetType: ec2.SubnetType.ISOLATED,
    },
  ],
})

const privateSg = new ec2.SecurityGroup(this, 'private-sg', {
  vpc,
  securityGroupName: 'private-sg',
})
privateSg.addIngressRule(
  privateSg,
  ec2.Port.allTraffic(),
  'allow internal SG access'
)

const subnetGroup = new rds.SubnetGroup(this, 'rds-subnet-group', {
  vpc,
  subnetGroupName: 'aurora-subnet-group',
  vpcSubnets: { subnetType: ec2.SubnetType.ISOLATED },
  removalPolicy: cdk.RemovalPolicy.DESTROY,
  description: 'An all private subnets group for the DB',
})

Next, lets create the database.

This following is an Aurora Serverless PostgreSQL database cluster with Postgres 10. Here, we pass the database name as BlogDB, and we have specified the vpc, subnetGroup and securityGroup that we created above.

// lib/appsync-cdk-rds-stack.ts

const cluster = new rds.ServerlessCluster(this, 'AuroraBlogCluster', {
  engine: rds.DatabaseClusterEngine.AURORA_POSTGRESQL,
  // Set the engine to Postgres
  parameterGroup: rds.ParameterGroup.fromParameterGroupName(
    this,
    'ParameterGroup',
    'default.aurora-postgresql10'
  ),
  defaultDatabaseName: 'BlogDB',
  enableDataApi: true,
  vpc,
  subnetGroup,
  securityGroups: [privateSg],
  removalPolicy: cdk.RemovalPolicy.DESTROY,
})

This database also comes with the credentials created in an AWS service named SecretsManger. We will be needing to fetch the database credentials from this service when we interact with the database from the AWS Console as well as from the Lambda function that we will be creating next.

Finally, let's create the Lambda function that will interact with this database that we created.

// lib/appsync-cdk-rds-stack.ts

const postFn = new lambda.Function(this, 'MyFunction', {
  vpc,
  vpcSubnets: { subnetType: ec2.SubnetType.ISOLATED },
  securityGroups: [privateSg],
  runtime: lambda.Runtime.NODEJS_12_X,
  code: new lambda.AssetCode('lambda-fns'),
  handler: 'index.handler',
  memorySize: 1024,
  timeout: cdk.Duration.seconds(10),
  environment: {
    SECRET_ID: cluster.secret?.secretArn || '',
    AWS_NODEJS_CONNECTION_REUSE_ENABLED: '1',
  },
})

// Grant access to Secrets manager to fetch the secret
postFn.addToRolePolicy(
  new iam.PolicyStatement({
    effect: iam.Effect.ALLOW,
    actions: ['secretsmanager:GetSecretValue'],
    resources: [cluster.secret?.secretArn || ''],
  })
)

// An endpoint to fetch the secret from Secrets Manager
new ec2.InterfaceVpcEndpoint(this, 'secrets-manager', {
  service: ec2.InterfaceVpcEndpointAwsService.SECRETS_MANAGER,
  vpc,
  privateDnsEnabled: true,
  subnets: { subnetType: ec2.SubnetType.ISOLATED },
  securityGroups: [privateSg],
})

Let's look at the three snippets of code that we created above.

The first snippet is a basic Node.js 12 Lambda function that takes a directory for code named lambda-fns which we will discuss further in this post.

The second snippet provides permissions via IAM for Secrets Manager to allow Lambda to read the credentials i.e. host, port, username, password from the secret.

The final snippet is an Interface Endpoint. By default, our Lambda function doesn't have access to the internet as it's deployed in the VPC we created, which is why we need this endpoint to fetch database credentials from Secrets Manager.

Lambda function setup with Prisma

The last thing left before deploying our app is setting up our Lambda function code with Prisma.

The setup that we have in the folder lambda-fns is a basic Prisma setup, but we need to add a couple of things to make it work.

Firstly, we need is to setup our schema.prisma in the following manner.

// prisma/schema.prisma

generator client {
  provider      = "prisma-client-js"
  binaryTargets = ["native", "rhel-openssl-1.0.x"]
}

datasource db {
  provider = "postgresql"
  url      = env("DB_URL")
}

model Post {
  id      String  @id
  title   String?
  content String?

  @@map("posts")
}

We created the schema for the table here manually and added the rhel-openssl-1.0.x target which is required to make Prisma run on Lambda.

The second thing is fetching the database credentials from Secrets Manager and providing it to Prisma.

// lambda-fns/db.ts

const sm = new SecretsManager()
let db: PrismaClient
export const getDB = async () => {
  if (db) return db

  const dbURL = await sm
    .getSecretValue({
      SecretId: process.env.SECRET_ID || '',
    })
    .promise()

  const secretString = JSON.parse(dbURL.SecretString || '{}')
  const url = `postgresql://${secretString.username}:${secretString.password}@${secretString.host}:${secretString.port}/${secretString.dbname}?connection_limit=1`

  db = new PrismaClient({ datasources: { db: { url } } })
  return db
}

Here, we've created a function named getDB that fetches the DB credentials, constructs the URL and passes that to Prisma.

Another thing to note here is that the PrismaClient instance is declared outside the function to reuse the execution environment. There are two reasons for this.

The first being that we do not create multiple connections to the DB on each call to the API. Secondly, we limit the API calls made to Secrets Manager to fetch secrets.

Here Lambda will reuse the same variable db created outside the function for multiple invocations of our GraphQL API.

Finally lets deploy the application using the following command:

yarn cdk deploy

This will deploy all the resources to the specified account and region specified in cdk.context.json and we will be able to see the stack in Cloudformation.

One step that's remaining is to create the table in our database.

On opening RDS from the Services menu and navigating to the Query Editor, we should be greeted with a popup like this:

We can get fetch secret ARN from the Secrets Manager service here:

On entering the ARN, we will get a query window where we can run the SQL to create the table. This SQL can be obtained from the commands.sql file in the repo.

The posts table will be created when we click on the Run button and see the success message that pops up below.

Testing our app

Finally, we are all set to test our application! We can select AppSync from the Services menu and select our created API from the list. On selecting the API and navigating to Queries, we will see a Query Editor.

Let's create a post by firing a mutation.

mutation MyMutation {
  createPost(
    post: { title: "Title", content: "Some content" }
  ) {
    id
    title
    content
  }
}

And voila! It works!

To test if this is actually persisted in the DB, let's fire a query to fetch all the posts.

query MyQuery {
  listPosts {
    id
    title
    content
  }
}

And it does return the created post!

Conclusion

In this entire stack, we deployed a GraphQL API with AppSync and created Lambda resolvers for our Queries/Mutations in which we queried our Aurora Serverless Postgres database using Prisma.

Here's the link to the repo again for those who want to dive in.

ryands17 / graphql-api-cdk-serverless-postgres

A basic example of AppSync + Lambda resolvers with AWS Aurora Serverless and Prisma

Currently we need to create our tables and edit our schema.prisma manually which is not ideal. We will be adding another post on this soon where we will see how we can use a Bastion Host with SSH port forwarding to interact with our serverless database locally and create our tables using the Prisma CLI!

The reason we cannot use Prisma CLI to generate tables currently is because Aurora Serverless is not publicly accessible, which is why we needed to create a bridge (Bastion Host) with SSH port forwarding to access our database locally.

Thanks a lot for reading this post, and if you have implemented this stack, please DO NOT FORGET TO DELETE THIS STACK via

yarn cdk destroy

Caching SSM Parameter Store values in Lambda

Ryan Dsouza — Sun, 20 Dec 2020 07:34:24 +0000

Intro

This article is inspired by a blog post written by Yan Cui where he shows how we can cache AWS SSM Parameter Store values.

This approach is recommended so that we can re-use the same values during multiple invocations of our Lambda function and save costs of API calls to SSM Parameter Store.

The following is an example using AWS CDK where we define an SSM Parameter and use the aws-sdk to fetch the parameter and cache it for a given amount of time which we can configure.

Here's the repository for those who want to dive right in :)

ryands17 / lambda-ssm-cache

Cache SSM Paramter Store values in Lambda via CDK

Let's start by defining the parameter that we are going to fetch in our Lambda:

// lib/lambda-ssm-stack.ts
import * as ssm from '@aws-cdk/aws-ssm'

new ssm.StringParameter(this, 'dev-key1', {
  parameterName: '/dev/key1',
  stringValue: 'value1',
})

This is a snippet that creates a StringParameter i.e. an un-encrypted value to store with the parameter path /dev/key1.

In a project, it's recommended to store keys in a path based format like ${projectName}/${environment}/${keyName} so that we can easily fetch all the values under the same path. For e.g. myProject/dev/DB_HOST and myProject/prod/DB_HOST for different environments.

Next, let's define our Lambda function:

// lib/lambda-ssm-stack.ts
import { join } from 'path'
import * as ln from '@aws-cdk/aws-lambda-nodejs'

const lambdaDir = join(__dirname, '..', 'lambda-fns')

const handler = new ln.NodejsFunction(this, 'fetchParams', {
  runtime: lambda.Runtime.NODEJS_12_X,
  memorySize: 512,
  handler: 'handler',
  entry: join(lambdaDir, 'src', 'index.ts'),
  depsLockFilePath: join(lambdaDir, 'yarn.lock'),
  nodeModules: ['ms'],
  sourceMap: true,
})

In the above snippet, we have created a Lambda function using the Node.js 12 runtime allocating it a memory of 512 MB with some specific values. Let's have a look at these.

The entry option accepts a file in which we will define our code to fetch the above defined parameter. In this case, it's an index.ts file in a directory named lambda-fns which we will look at further.
The handler option is the name of the function which is expored in our index.ts file and that will be used by Lambda.
The depsLockFilePath, nodeModules, and sourceMap options are for bundling our Node.js function before deploying. This is done using the aws-lambda-nodejs package that transpiles our TypeScript code to JavaScript with node_modules so that it can run it on Lambda.

We're done creating the function. Let's look at the final snippet of code required to deploy our stack:

// lib/lambda-ssm-stack.ts
handler.addToRolePolicy(
  new iam.PolicyStatement({
    effect: iam.Effect.ALLOW,
    actions: ['ssm:GetParametersByPath'],
    resources: [`arn:aws:ssm:${this.region}:*:parameter/dev*`],
  })
)

This final snippet allows the Lambda function access to fetch the created parameter from the Parameter Store, and not just any parameter but the one starting with /dev. This makes sure that we do not fetch any unwanted/unneeded secret values especially for other projects.

Now let's look at our Lambda function in the lambda-fns folder:

// lambda-fns/index.ts
import { Context } from 'aws-lambda'
import { config, loadParameters } from './config'

export const handler = async (event: any, context: Context) => {
  await loadParameters()
  return {
    value: config.values['/dev/key1'],
    success: true,
  }
}

This is our handler function that calls the loadParameters function to fetch our parameter and returns the value.

Let's have a look at config.ts where the main code for fetching the parameter and caching is included.

We'll start by defining the variables required:

// lambda-fns/config.ts

import { SSM } from 'aws-sdk'
import ms from 'ms'

type Config = {
  values: Record<string, string | undefined>
  expiryDate?: Date
}

const ssm = new SSM()
export let config: Config = { values: {} }

Here we have initialized an instance of SSM which we will use to fetch the parameter and the config variable that will be used to store our parameter. We can have multiple values fetched as well that can be stored here.

Notice the type of the config variable. It has two keys, the values which will store our parameters and the expiryDate that will invalidate our cache and fetch all the parameters again.

The expiry time is configurable and we will see how this is used in the loadParameters function next.

// lambda-fns/config.ts
export const loadParameters = async ({
  expiryTime: cacheDuration = '1h',
}: {
  expiryTime?: string
} = {}) => {
  if (!config.expiryDate) {
    config.expiryDate = new Date(Date.now() + ms(cacheDuration))
  }
  if (isConfigNotEmpty() && !hasCacheExpired()) return

  console.log('[Cost]: API called')
  config.values = {}
  const { Parameters = [] } = await ssm
    .getParametersByPath({
      Path: '/dev',
    })
    .promise()

  for (let param of Parameters) {
    if (param.Name) config.values[param.Name] = param.Value
  }
}

const hasCacheExpired = () =>
  config.expiryDate && new Date() > config.expiryDate

const isConfigNotEmpty = () => Object.keys(config.values).length

The final and most important function of our Lambda, the loadParameters function accepts a single value named expiryTime that is by default set to 1 hour and can be overridden. I have used the ms library to set human readable time periods which will be automatically converted to milliseconds.

We first check if the expiry date exists. If not, we set it to the value: current time + expiryTime.

Then we have this snippet:

if (isConfigNotEmpty() && !hasCacheExpired()) return

The following indicates that if we have values present in our config and if the cache hasn't expired then bail out of the function as we already have the values and we wouldn't want to call the API to fetch our parameters.

The above functions were just added to make the condition readable and they are defined as follows:

const hasCacheExpired = () =>
  config.expiryDate && new Date() > config.expiryDate

const isConfigNotEmpty = () => Object.keys(config.values).length

The next snippet is the entire fetching and setting of the parameters as we have already checked for the above conditions and we know that either the cache has expired or we do not have any values in our config.

console.log('[Cost]: API called')
config.values = {}
const { Parameters = [] } = await ssm
  .getParametersByPath({
    Path: '/dev',
  })
  .promise()

for (let param of Parameters) {
  if (param.Name) config.values[param.Name] = param.Value
}

In this, we simply added a log to know whether we're fetching the parameters and we have set the values to an empty object ({}). Then, we call the getParametersByPath method and pass the path dev we created in our resources file. Lastly, we loop over the parameters and add those to the values property in our config.

And we're done! To deploy this stack, run yarn cdk deploy or npm run cdk deploy.

Note: A prerequisite for this would be installing aws-cli and configuring the profile (I have configured the default profile in this case) with the Access and Secret keys.

Now, let's run this lambda by creating a test event from the console:

On clicking Test, we can see the logs and on the first run, we would see something like this:

We successfully get the result of the parameter we stored and if we check the logs, we can see API CALLED because the value wasn't present as it's the very first invocation and the call to Parameter Store was made to fetch the value.

Let's run the test event again and see the result.

We get the value, but notice that the log API CALLED isn't present, which means that the value was obtained from the cache :)

This values remains due the Lambda reusing the execution environment on subsequent invocations. This is although upto Lambda and we cannot configure this externally. Lambda can start on a clean state and we would need to add a condition to fetch the values again which we have done in the loadParameters function as seen above.

So that was it for caching values from the SSM Parameter Store in a Lambda function. Do not forget to delete this stack after you've done with it using yarn cdk destroy.

Thank you all for reading! Do like and share this post if you've enjoyed it :)

Generating thumbnails via S3 events using aws-cdk (TypeScript)

Ryan Dsouza — Wed, 16 Sep 2020 18:48:12 +0000

This is another post on CDK after a while that's a bit different. You can check out the previous two posts that I had written here:

Deploying a SPA using aws-cdk (TypeScript)

Ryan Dsouza ・ Apr 4 '20

#aws #cdk #spa #development

Deploying a Node app to Beanstalk using aws-cdk (TypeScript)

Ryan Dsouza ・ Apr 18 '20

#cdk #aws #node #typescript

Let's see something today that we have all done manually in the past, creating thumbnails for images for profile pic uploads in S3.

Here we shall walk through how we can react events when we upload a file to S3 and send that to SQS. We shall also attach a Lambda handler that processes the message from the SQS queue and generates our thumbnail in the same bucket.

Note: The code for the entire repository is here if you want to jump right in :)

ryands17 / s3-thumbnail-generator

Generate thumbnails with S3 via SQS and Lambda created using AWS CDK

S3 image resizer

This is an aws-cdk project where you can generate a thumbnail based on S3 event notifications using an SQS Queue with Lambda.

Steps

Change the region in the cdk.context.json to the one where your want to deploy. Default is us-east-2.
Run yarn (recommended) or npm install
Go to the resources folder cd resources.
Run yarn add --arch=x64 --platform=linux sharp or npm install --arch=x64 --platform=linux sharp to build sharp for Lambda.
Go back to the root and run yarn cdk deploy --profile profileName to deploy the stack to your specified region. You can skip providing the profile name if it is default. You can learn about creating profiles using the aws-cli here.
Now you can add image/s in the photos folder in S3 and check after a couple of seconds that images in a folder named thumbnails is generated by Lambda via SQS.

The…

View on GitHub

In this repo, all the code that we write to create resources will be in the s3_thumbnail-stack.ts file situated in the lib folder.

Let's start by initializing a couple of constants:

const bucketName = 'all-images-bucket'
const prefix = 'photos/'

These constants are the name of the bucket that we will be creating and the path where we will be storing our photos respectively.

Then, in the constructor of our S3ThumbnailStack class, we shall start by creating the Queue that will handle all the object creation events coming from S3.

const queue = new sqs.Queue(this, 'thumbnailQueue', {
  queueName: 'thumbnailPayload',
  receiveMessageWaitTime: cdk.Duration.seconds(20),
  visibilityTimeout: cdk.Duration.seconds(60),
  retentionPeriod: cdk.Duration.days(2),
})

This code creates an SQS Queue named thumbnailPayload and sets some parameters:

receiveMessageWaitTime: This is the time at what interval should one poll for messages. We have kept it at 20 seconds as that is the longest we can have. This method is also known as long polling.
visibilityTimeout: This is the time for a message to be visible again for other handlers to process it. Here we are assuming that the Lambda will complete the thumbnail generation within 60 seconds. This time should be set accordingly so that the same messages are not consumed twice.
retentionPeriod: The amount of days the message will remain in the queue if unprocessed.

Moving on, lets create the lambda handler.

const handler = new lambda.Function(this, 'resizeImage', {
  runtime: lambda.Runtime.NODEJS_12_X,
  code: lambda.Code.fromAsset('resources'),
  handler: 'index.handler',
  timeout: cdk.Duration.seconds(20),
  memorySize: 256,
  logRetention: logs.RetentionDays.ONE_WEEK,
  environment: {
    QUEUE_URL: queue.queueUrl,
    PREFIX: prefix,
  },
})

Most of these options are familiar so lets discuss about the code and environment parameters respectively.

code: This is where our Lambda code will be taken from. We have specified a fromAsset('resources'), meaning CDK will take this from a folder named resources in our code. In this folder, there's a simple NodeJS Lambda function that resizes images using the sharp library.
environment: These contain two values. One for the queue URL which we will be needing to delete the message after we have processed it and the other is the S3 bucket prefix from where we need to fetch the image uploaded.

The Lambda handler looks something like this that is in the resources folder.

const AWS = require('aws-sdk')
const sharp = require('sharp')

const prefix = process.env.PREFIX
const s3 = new AWS.S3()
const sqs = new AWS.SQS({
  region: process.env.AWS_REGION,
})

exports.handler = async event => {
  try {
    for (let message of event.Records) {
      const body = JSON.parse(message.body)

      if (Array.isArray(body.Records) && body.Records[0].s3) {
        const objectName = decodeURIComponent(
          body.Records[0].s3.object.key.replace(/\+/g, ' ')
        )
        const bucketName = body.Records[0].s3.bucket.name
        console.log({ objectName, bucketName })
        try {
          await resizeAndSaveImage({ objectName, bucketName })
          await deleteMessageFromQueue(message.receiptHandle)
        } catch (e) {
          console.log('an error occured!')
          console.error(e)
        }
      } else {
        await deleteMessageFromQueue(message.receiptHandle)
      }
    }

    return {
      statusCode: 200,
      body: JSON.stringify('success!'),
    }
  } catch (e) {
    console.error(e)
    return {
      statusCode: 500,
      error: JSON.stringify('Server error!'),
    }
  }
}

async function resizeAndSaveImage({ objectName, bucketName }) {
  const objectWithoutPrefix = objectName.replace(prefix, '')
  const typeMatch = objectWithoutPrefix.match(/\.([^.]*)$/)
  if (!typeMatch) {
    console.log('Could not determine the image type.')
    return
  }

  // Check that the image type is supported
  const imageType = typeMatch[1].toLowerCase()
  if (imageType !== 'jpg' && imageType !== 'png') {
    console.log(`Unsupported image type: ${imageType}`)
    return
  }

  // Download the image
  const image = await s3
    .getObject({
      Bucket: bucketName,
      Key: objectName,
    })
    .promise()

  // Transform the image via sharp
  const width = 300
  const buffer = await sharp(image.Body).resize(width).toBuffer()

  const key = `thumbnails/${objectWithoutPrefix}`
  return s3
    .putObject({
      Bucket: bucketName,
      Key: key,
      Body: buffer,
      ContentType: 'image',
    })
    .promise()
}

function deleteMessageFromQueue(receiptHandle) {
  if (receiptHandle) {
    return sqs
      .deleteMessage({
        QueueUrl: process.env.QUEUE_URL,
        ReceiptHandle: receiptHandle,
      })
      .promise()
  }
}

This handler performs two functions, resize and save the image to S3, and delete the message from the queue once processed successfully.

And the third and final resource that we will be creating is our S3 bucket as follows:

const imagesBucket = new s3.Bucket(this, 'allImagesBucket', {
  bucketName,
  removalPolicy: cdk.RemovalPolicy.DESTROY,
})

This will just create our bucket and destroy it once we delete our Cloudformation stack.

There are three things needed to make this entire workflow running. Lets go through them and perform the final touches.

Send object creation events from S3 to SQS: This is so that S3 sends an event to SQS whenever an object, in our case an profile picture is uploaded in the photos folder. This is how we can add that:

imagesBucket.addObjectCreatedNotification(new s3Notif.SqsDestination(queue), {
  prefix,
})

Here, we are adding an ObjectCreatedNotification in which we are setting the destination to be an SQS Queue. Note the prefix paramter. This means only events for objects added in the photos folder or prefix in the case of S3 will be sent to SQS.

Allow Lambda access to S3: This is important as we need to fetch the image, generate the thumbnail, and put the thumbnail back in the bucket. Lets attach a policy to the handler:

handler.addToRolePolicy(
  new iam.PolicyStatement({
    effect: iam.Effect.ALLOW,
    resources: [`${imagesBucket.bucketArn}/*`],
    actions: ['s3:PutObject', 's3:GetObject'],
  })
)

This tells Lambda "You have access just to GetObject and PutObject methods on S3 on the created bucket and nothing else".

Allow Lambda to process messages from SQS: Finally, we tell Lambda to execute when a mesasge is added in the SQS Queue from the S3 object creation event.

handler.addEventSource(new eventSource.SqsEventSource(queue))

This basically says, "Lambda, consider this SQS Queue as a source and consume any messages that you get from this queue".

And we're done! Let's deploy this stack by firing yarn cdk deploy or npm run cdk deploy if you're using NPM.

Note: I'm assuming you have setup the AWS CLI and set your access and secret keys in the default profile. You can read more about that here.

After successful stack creation, you will get a bucket like this:

As you can see I have already created a folder named photos. Let's upload a profile picture for our user that's a cat, and see if the event fires to SQS and the Lambda function is triggered.

After uploading, we can see two folders here after some time. You can refresh it after a while if you don't see it.

Awesome! Our Lambda has successfully run and we can check the difference between the two pictures as well to see whether it has worked!

Original profile pic:

Resized thumbnail:

This is how we successfully deployed a stack with S3, Lambda and SQS to generate thumbnails automatically on upload.

Thank you all for reading! Do like and share this post if you've enjoyed it :)

Exploring the new MobX API

Ryan Dsouza — Sat, 11 Jul 2020 18:54:07 +0000

I have always been an avid admirer of MobX and its simple API. Do read my previous posts regarding those in my profile.

Recently Michel Weststrate tweeted about a new API for MobX without the need of decorators or the decorate API.

// Detect dark theme var iframe = document.getElementById('tweet-1280987713341132803-528'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1280987713341132803&theme=dark" }

I have expanded on this example to show the simple API introduced by MobX named makeAutoObservable.

This API removes the need of using decorators which are still experimental and their API might change. This also removes the need of using the decorate method as we used to do before.

export class User {
  name = 'anonymous'

  updateName(name: string) {
    this.name = name
  }
}

decorate(User, {
  name: observable,
  updateName: action
})

Most of you folks would be familiar with the above syntax that we used to use MobX without decorators. One issue that I had with the above was I could easily miss any propery and that wouldn't be obsevable which could lead to unwanted behaviour.

makeAutoObservable solves the above problem and all the class properties are observable by default. The getters are computed and the methods are actions as well.

There's also another method makeObservable that lets us specify the properties that are required to be made as observable if we do not want all properties of the class to be observable by default.

I personally like this API as given the uncertain future of decorators and the pitfalls that I faced with the decorate API, this is a great addition to the library for cleaner code and not needing decorators as a primary API in MobX.

I have created a sandbox with React that you can explore. In this I have shown how we can combine multiple stores with a Root store, create our RootStore context, and fetch those values we require via the useContext hook.

As usual, the best part about MobX being components re-render only when changes are made to the values they're using and not otherwise.

Do let me know your thoughts on this and thanks for reading :)