Tag: aws-lambda

How to initiate a Media Conversion with Ruby in AWS Lambda?

I’m trying to start a MediaConvert job with Ruby in AWS Lambda after a file has been uploaded to a bucket. The event kicks off fine, but I’m having trouble initiating the job.

I was trying to follow instructions from here to initiate the client: https://docs.aws.amazon.com/sdkforruby/api/Aws/MediaConvert/Client.html

And from here to kick off the job:
https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/MediaConvert/Client.html#create_job-instance_method

# Event to kick off the media conversion: file uploaded to a bucket
sourceS3Bucket = event['Records'][0]['s3']['bucket']['name']
sourceS3Key = event['Records'][0]['s3']['object']['key']
sourceS3 = 's3://'+ sourceS3Bucket + '/' + sourceS3Key
jobMetadata['input'] = sourceS3

# Loading the MediaConvert settings
json_from_file = File.read('job.json')
jobSettings = JSON.parse(json_from_file)

# Initiating a client
client = Aws::MediaConvert::Client.new(
  access_key_id: ENV['ACCESS_KEY_ID'],
  secret_access_key: ENV['SECRET_ACCESS_KEY']
)

# Kicking off a job
response = client.create_job({
  Role: ENV['MediaConvertRole'],
  UserMetadata: jobMetadata,
  Settings: jobSettings
})

I’m getting this error:

Critical exception from handler
{
  "errorMessage": "uninitialized constant Aws::MediaConvert",
  "errorType": "Function<NameError>",
  "stackTrace": [
    "/var/task/convert.rb:38:in `rescue in call'",
    "/var/task/convert.rb:6:in `call'"
  ]
}

I’m unsure why the function is failing, do you have any clues please?

Solution:

You need to install aws-sdk-mediaconvert gem. Add this to Gemfile:

gem 'aws-sdk-mediaconvert'

and run 

bundle

If you still getting this error require it at the top of your MediaConvert job:

require 'aws-sdk-mediaconvert'

What are the pros and cons of having one lambda function with "smaller functions" vs multiple functions

I have a lambda function with 10 “smaller functions” inside that single lambda which gets accessed by 10 api gateway endpoints.

What are the pros and cons of designing the backend this way?
I know it’s more difficult to manage all those functions when testing, but how would it compare speedwise to 10 seperate lambda functions?

exports.handler =  (event) => {

    const path = event.path;
    const method = event.httpMethod;

    if(path === '/getmail' && method === 'GET'){
        return mailQuery(event);
    }

   if(path === '/getmessagethread' && method === 'GET'){
        return getMessageThread(event);
    }

   if(path === '/replytomessage' && method === 'POST'){
        return replyToMessage(event);
    }

    if(path === '/sendmessage' && method === 'POST'){
        return sendMessage(event);
    }

    . . .
};

Solution:

imo do whatever you feel comfortable with and that works best for the use case. Here is just my 2 cents on the subject though:

Pros:

  • Only need to manage one function and it’s code
  • This might be easier to test then using SAM Local as we could have multiple event objects to test with

Cons:

  • If multiple requests come in that function’s reserved concurrency(if it has that) could be exhausted since it is only that one function serving all those requests
  • API Gateway’s max timeout is 30 seconds, so if this function executes longer due to this extra code then the request will fail
  • Could also increase the deployment package size which Lambda has the following limit
    https://docs.aws.amazon.com/lambda/latest/dg/limits.html

    50 MB (zipped, for direct upload)
    250 MB (unzipped, including layers)
    3 MB (console editor)

Speedwise I don’t think it would make much difference. I don’t believe there would be much difference in the time to execute if a function was called as opposed to having the code directly in the handler function.

HTH

-James

Unable to retrieve the attribute/field value of DynamoDB item in Lambda function

I am developing an Android application using AWS services as back-end. Now I am doing something in the Lambda function when an item is added to a DynamoDB table. But I cannot retrieve the attribute value inside the lambda function. This is what I have done so far.

I created a DynamoDB table from the Mobile Hub application.

enter image description here

enter image description here

Table name is item.

In the lambda function I tried to retrieve the name value of the item table like this.

if (record.eventName == 'INSERT') {
     var name = record.dynamodb.item.Name.S
}

But it is saying that name is undefined. How can I fix it?

I tried this as well.

record.dynamodb['thegoodyardandroid-mobilehub-330286608-item'].Name.S

not working

Solution:

The event you get from a dynamo db stream is documented here: https://docs.aws.amazon.com/lambda/latest/dg/eventsources.html#eventsources-ddb-update

Critically there is no Item key. For an INSERT event there’s only a NewImage key.

So in your case you’ll want:

var name = record.dynamodb.NewImage.Name.S

How to get current cognito user from within go lambda

I’m having a hard time to get the current Cognito user attributes from within my lambda function, that is written in Go. I’m currently doing:

userAttributes = request.RequestContext.Authorizer["claims"]

And if I want to get the email:

userEmail = request.RequestContext.Authorizer["claims"].(map[string]interface{})["email"].(string)

I don’t think this is a good way or even an acceptable way – it must have a better way to do it.

Solution:

You can use 3rd party library to convert map[string]interface{} to a concrete type. Check the mitchellh/mapstructure library, it will help you to implement in a better way.

So, you could improve your code with this code : 

import "github.com/mitchellh/mapstructure"

type Claims struct {
    Email string
    // other fields
    ID int
}

func claims(r request.Request) (Claims, error) {
    input := r.RequestContext.Authorizer["claims"]
    output := Claims{}
    err := mapstructure.Decode(input, &output)

    if err != nil {
        return nil, err
    }

    return output, nil
}

And somewhere in your handlers, you could get your claims by calling this method

func someWhere(){

    userClaims, err := claims(request)

    if err != nil {
        // handle
    }

    // you can now use : userClaims.Email, userClaims.ID
}

Don’t forget to change func claims request parameter type according to yours (r parameter).

DynamoDB BatchGetItem dynamic TableName in Lambda function

I’m building a serverless backend for my current application using dynamoDb as my database. I use aws sam to upload my lambda functions to aws. In addition, I pass all my table names as global variables to lambda (nodejs8.10 runtime) to access them on the process.env object within my lambda function. The problem that I’m facing with this is the following: Whenever I run the batchGetItem method on dynamoDB I have to pass a string as my table name, I cannot dynamically change the table name depending on the global variable: 

const AWS = require('aws-sdk');
const dynamodb = new AWS.DynamoDB({region: 'ap-south-1'}, {apiVersion:'2012-08-10'});

const params = {
    RequestItems: {
        //needs to be a string, cannot be a variable containing a string
        'tableName': {
            Keys: [] //array of keys
         }
    }
}
dynamodb.batchGetItem(params, (err, result) => {
// some logic
})

I need to pass the table name as a string, essentially hardcoding the table name into my function. Other DynamoDB operations, like for example the getItem method, accept a key value pair for the table name in the parameter object: 

const tableName = process.env.TableName;
const getItemParams = {
   Key: {
        "Key": {
             S: 'some key'
         }
   },
   // table name can be changed according to the value past to lambda's environment variable
   TableName: tableName
}
dynamodb.getItem(getItemParams, (err, result) => {
// some logic
}

Hence my question, is there any way to avoid hardcoding the table name in the batchGetItem method and, instead, allocate it dynamically like in the getItem method?

Solution:

You can use the tableName from environment variables. Build your params in 2 steps :

const { tableName } = process.env;

const params = {
  RequestItems: {},
};

// `tableName` is your environment variable, it may have any value
params.RequestItems[tableName] = {
  Keys: [], //array of keys
};

dynamodb.batchGetItem(params, (err, result) => {
  // some logic
})

aws lambda function timed out when querying data from MySQL database

I have an index.js file which contains code like the following:

const dbConfig = require('./config/dbConfig')
const mysql = require('mysql')

var con = mysql.createConnection({
  host: dbConfig.host,
  user: dbConfig.username,
  password: dbConfig.password,
  database: dbConfig.database
})

function readMessages (event, context, callback) {
  console.log('function triggered')
  con.connect((err) => {
    if (err) {
      console.error(err)
      callback(err)
    } else {
      console.log('Connected!')
      con.query('SELECT * FROM Messages WHERE isDeleted = 0;', (err, result, fields) => {
        if (err) {
          console.error(err)
          callback(err)
        } else {
          console.log(result)
          con.end()
          callback(null, result)
        }
      })
    }
  })
}

exports.handler = readMessages

The code correctly gets data from the mysql database and displays them on the screen when I run it on my local machine.

However, I got Task timed out after 7.01 seconds error when it is run on aws-lambda.

The code and its dependencies are packaged in a file named app.zip, then uploaded to aws-lambda.

app.zip
├── config
│   └── dbConfig.js
├── index.js
└── node_modules

The only log message being printed by my function is function triggered. I cannot find other log messages generated by my function in the cloud watch log.

Why does the function timed out on aws-lambda?

Solution:

If I had to guess it is a permissions issue, when you run locally it is going to grab credentials from your local machine/environment – when you run this in lambda you need to assign a role to the lambda that has the permissions it needs to access the mysql database.

Also, make sure that the mysql database is accessible to the lamba – i.e. your not trying to access a mysql database that is local to your machine from the lambda function (I was assuming you were using rds).

How do I get Cloud Formation to create a changeset to update my Lambda functions?

I have a Lambda function which I’ve verified to work correctly. I’m able to update the function by hand on the command line using “update-function-code” but I’ve been trying to get it working with Code Pipeline and Cloud Formation.
Here are the steps I have so far:

  1. Source – fetch the code from github. This works correctly.
  2. Build – test the code in Solano (3rd party CI). This works too and on the last stage it zips up the repo and uploads it to my S3 bucket.
  3. Deploy – This is the “deploy” action category with the action mode “create or replace a change set”. This doesn’t work if the Lambda function already exists.
  4. Beta – Execute the changeset. This works if the change set was generated correctly.

My samTemplate.yml looks like this:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: My Lambda function
Resources:
LambdaFunction:
Type: AWS::Serverless::Function
Properties:
FunctionName: MyLambdaExecute
Description: My Lambda function
Handler: myhandler.handler
Runtime: nodejs6.10
CodeUri: s3://mybucket/mydirectory/mylambdacode.zip
AutoPublishAlias: Staging
Timeout: 30
DeploymentPreference:
Type: AllAtOnce

If the lambda function with the name “MyLambdaExecute” doesn’t exist and I push up code to github, it works perfectly. But if I modify some code and push again it runs the first two steps, but then generates an empty change set with the status:

FAILED – No updates are to be performed.

I’m not sure what I have to do to get it to publish a new version. How do I get it to realize it needs to create a new changeset?

Solution:

I believe you are receiving the “No updates” message because technically nothing is changing in your CloudFormation template. When attempting to build the changeset, the contents of the S3 file are not examined. It just sees that none of the CloudFormation resource properties have changed since the previous deployment.

Instead, you may use a local relative path for the CodeUri, and aws cloudformation package can upload the file to a unique S3 path for you. This ensures that the template changes each time and the new Lambda code is recognized/deployed. For example:

aws cloudformation package --template-file samTemplate.yml --output-template-file sam-output-dev.yml --s3-bucket "$CodePipelineS3Bucket" --s3-prefix "$CloudFormationPackageS3Prefix"

This command can be put into the build step before your create/execute changeset steps.

To see an demonstration of this entire flow in practice, you can look at this repository

Although I will warn that it’s a bit outdated thanks to the new features released at the end of 2017. (For example, I was publishing Lambda aliases manually via extra steps because it was written pre-AutoPublishAlias.)

How To Put DynamoDB Put Requests In A Queue

I have seen a lot of people mentioning, that a way to deal with limited WCUs in DynamoDB is to send your requests in a Queue and let the queue insert the data into dynamodb in a way that will not go over your allocated WCUs.

Does anyone have any examples of this? I am currently working with aws lambda in python and nodeJS.

What I understand:

If lambda wants to put 2,000 items into dynamodb and we only have 100 WCUs, instead of having lambda retry and or waiting between each requests.

We can send the items to a SQS queue which will then input the items at a rate of 100WCUs per second.

Is this the right workflow?

Solution:

You might use Redis. Especially if you are inserting into the same partition over and over again (called ‘hot partition’) Redis provides random selection from a set. Simply you queue (insert) them into Redis and read from another process constantly and insert into db. On Nodejs and Python redis has thousands of example and pretty handy to use 🙂

There is an AWS Blog entry on caching. Have a look at it. Also recommends this pattern.

Although the number of use cases for caching is growing, especially
for Redis, the predominant workload is to support read-heavy workloads
for repeat reads. Additionally, developers also use Redis to better
absorb spikes in writes. One of the more popular patterns is to write
directly to Redis and then asynchronously invoke a separate workflow
to de-stage the data to a separate data store (for example, DynamoDB).

There are probably more ways of achieving what you want maybe even in a simpler way but Redis goes with literally everything maybe you are even using it already 😉

How to specfiy existing FunctionName on SAM Template

I’m trying to deploy AWS Lambda function by using SAM.
What I want to do is to update exsiting lambda function by deploying local source code.

In order to do that, I specified the existing lambda function name as ‘FunctionName’ in template.yaml like below.
However, ‘FunctionName’ does only support for creating new function, not updating to existing function.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-function.html#cfn-lambda-function-functionname

Are there any ways to specify Function Name in SAM in order to update lambda function?

[template.yaml]

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  HelloWorld:
    Type: 'AWS::Serverless::Function'
    Properties:
      FunctionName: 'hello_world'
      Description: ''
      Handler: index.handler
      MemorySize: 256
      Role: 'arn:aws:iam::368834739507:role/lambda_basic_execution'
      Runtime: nodejs6.10
      Timeout: 120

Solution:

Using SAM (and/or CloudFormation), you cannot update existing resources.

SAM (and CloudFormation) create and manage their own resources. All resources specified in the template are going to be created when the stack is created. They cannot be “taken over”.

Instead, you should allow SAM (or CloudFormation) to create the Lambda function for you, then update users to reference the new function. After that, you can update your code using SAM.

How get Environment Variables from lambda (nodejs aws-sdk)

We can set up Environment Variables in aws-lambda for example via AWS SAM:

Environment:
    Variables:
      TABLE_NAME: !Ref Table

How can I get this variables from current lambda via Node JS AWS-SDK?

Solution:

Just as you would any environment variable from node

const process = require('process');
const tableName = process.env.TABLE_NAME;