Skip to content

Conversation

@eahefnawy
Copy link
Contributor

@eahefnawy eahefnawy commented Aug 24, 2016

This PR redesigns all of our CF logical IDs as described here. It also fixes lots of bugs related to logical IDs.

This PR resolves the following Issues:

Progress:

  • Update all logical IDs
  • Fix S3 bug
  • Fix SNS bug
  • Fix CORS bug
  • Update tests
  • Add tests for S3 & SNS bugs
  • Test SNS & S3 bugs on AWS
  • Fix conflicts with master

Generated Logical IDs

For this serverless.yml:

service: serverless-logical-id

provider:
  name: aws
  runtime: nodejs4.3

functions:
  hello:
    handler: handler.hello
    events:
      - s3: serverless-logical-id
      - schedule: rate(10 minutes)
      - http:
          path: users
          method: get
      - http:
          path: users
          method: post
      - sns: topic-name
  world:
    handler: handler.world
    events:
      - http:
          path: users/{number}
          method: post
      - sns: topic-name
      - s3: serverless-logical-id

These logical IDs are generated:
screen shot 2016-08-25 at 3 26 08 pm

Validating S3/SNS Deployment

  1. Create a service with the following config (don't forget to use different bucket name)
service: serverless-logical-id

provider:
  name: aws
  runtime: nodejs4.3

functions:
  hello:
    handler: handler.hello
    events:
      - s3: serverless-logical-id
      - sns: topic-name
  world:
    handler: handler.world
    events:
      - s3:
          bucket: serverless-logical-id
          event: s3:ObjectRemoved:*
      - sns: topic-name
  1. Deploy
  2. You can see the notification config in the bucket for both functions:
    screen shot 2016-08-25 at 3 51 27 pm
  3. You can see the subscription config in the topic for both functions:
    screen shot 2016-08-25 at 3 54 57 pm
  4. serverless invoke -f hello -> invoke the function to make sure it's deployed
  5. serverless logs -f hello -> you'll see that the function was invoked
  6. Publish a topic using AWS SNS UI
  7. serverless logs -f hello -> you'll see that the function was invoked again
  8. upload a file to the S3 bucket
  9. serverless logs -f hello -> you'll see that the function was invoked again
  10. Here's the total invocations
    screen shot 2016-08-25 at 4 10 45 pm

@serverless/core Feel free to review

@eahefnawy eahefnawy added this to the v1.0.0-beta.3 milestone Aug 24, 2016
@eahefnawy eahefnawy self-assigned this Aug 24, 2016
@eahefnawy eahefnawy changed the title New Logical IDs New Logical ID Names Aug 24, 2016
@eahefnawy eahefnawy changed the title New Logical ID Names New CF Logical IDs Aug 24, 2016
@pmuens
Copy link
Contributor

pmuens commented Aug 25, 2016

Wow @eahefnawy great job! 🎉 👍

Also love the guideline on how to test it! 💥 ❤️ it.

I had some problems with the outputs of the info plugin. This might be broken as it tries to access invalid CloudFormation Outputs properties.

Edit: Looked through the source code and added some minor comments.

normalizedAuthorizerName}LambdaPermissionApiGateway`;

const newAuthPermissionObject = {
const newAuthrizerPermissionObject = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

authOrizer

@pmuens
Copy link
Contributor

pmuens commented Aug 25, 2016

One update:

Did a recent change to an already deployed service.

This one was what I've deployed at first:

functions:
  hello:
    handler: handler.hello
    events:
      - s3: serverless-logical-id-test
      - sns: topic-name
  world:
    handler: handler.world
    events:
      - s3:
          bucket: serverless-logical-id-test
          event: s3:ObjectRemoved:*
      - sns: topic-name

after that I changed it to (basically the service file you've mentioned in the PR description):

functions:
  hello:
    handler: handler.hello
    events:
      - s3: serverless-logical-id-test
      - schedule: rate(10 minutes)
      - http:
          path: users
          method: get
      - http:
          path: users
          method: post
      - sns: topic-name
  world:
    handler: handler.world
    events:
      - http:
          path: users/{number}
          method: post
      - sns: topic-name
      - s3: serverless-logical-id-test

which failed with the following error messages:

An error occurred while provisioning your cloudformation:
     The following resource(s) failed to create: [HelloEventsRule1,
     WorldLambdaPermissionApiGateway, HelloLambdaPermissionApiGateway].
     The following resource(s) failed to update: [S3BucketServerlesslogicalidtest].

Here's a screenshot of a part from the CloudFormation stack output on AWS:
bildschirmfoto 2016-08-25 um 15 49 57

@flomotlik
Copy link
Contributor

@pmuens that error is intended and is because you can't add ObjectCreated twice through S3 in Cloudformation. If you want two functions to react to the same S3 event you need to push the event into SNS and then read it from SNS (I had the same issue and looked it up :D )

@flomotlik
Copy link
Contributor

flomotlik commented Aug 25, 2016

@eahefnawy Can you check out the CORS stuff as well, the naming there isn't correct yet:

ApiGatewayMethodPreflight0 Should imho be ApiGatewayMethodUsersCreateOptions as its simply creating an OPTIONS endpoint for a specific path. There is also a bug at the moment if you have multiple endpoints for the same path (e.g. GET and DELETE requests for users/create) the OPTIONS request will be replaced and only one of the HTTP Methods will be allowed)

Here's a link to the issue with the bug: #1960

@pmuens
Copy link
Contributor

pmuens commented Aug 25, 2016

Damn! Good catch @flomotlik!
Works like a charm. Here's the functions definition of the service I've used to test it:

functions:
  hello:
    handler: handler.hello
    events:
      - s3: serverless-logical-id-test
      - schedule: rate(10 minutes)
      - http:
          path: users
          method: get
      - http:
          path: users
          method: post
      - sns: topic-name
  world:
    handler: handler.world
    events:
      - http:
          path: users/{number}
          method: post
      - sns: topic-name
      - s3:
          bucket: serverless-logical-id-test
          event: s3:ObjectRemoved:*

@flomotlik
Copy link
Contributor

HelloEventsRule1 should have Schedule as the name imho so HelloEventsRuleSchedule1 so if we add other EventsRule implementations we can distinguish them from schedule

@flomotlik
Copy link
Contributor

flomotlik commented Aug 25, 2016

IamRoleLambda -> IamRoleLambdaExecution to make it clear that this is the Lambda Execution Role and follow their naming conventions. Same with IamPolicyLambda

@pmuens
Copy link
Contributor

pmuens commented Aug 25, 2016

When doing that could we also update the corresponding template file names (e.g. here: https://github.com/serverless/serverless/tree/master/lib/plugins/aws/deploy/compile/functions).

So e.g.

iam-policy-lambda-template.json --> iam-policy-lambda-execution-template.json
iam-role-lambda-template.json --> iam-role-lambda-execution-template.json

@eahefnawy
Copy link
Contributor Author

@pmuens Really good catch! I was worried I'd miss something like that :)

I just made all the updates mentioned above, and I'm now looking into the CORS bug. Will keep you posted.

const unzippedFileData = unzippedData.files;

// binary file is set with chmod of 777
console.log(unzippedFileData['bin/some-binary'])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is probably on accident still here?

Copy link
Contributor

@pmuens pmuens Aug 26, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Faced the same today while reviewing it. Should be resolved with 8464954 as otherwise the linter will bark loud 🐶 .

@rowanu
Copy link
Contributor

rowanu commented Aug 30, 2016

Will the base names be documented as part of this PR? I can see them in #1931 but it would be helpful to have them in the docs so that they can be referenced in custom resources.

@flomotlik
Copy link
Contributor

@rowanu yup definitely!

@eahefnawy eahefnawy force-pushed the logical-id branch 2 times, most recently from a8a3a4d to e037724 Compare August 31, 2016 09:37
@flomotlik
Copy link
Contributor

flomotlik commented Aug 31, 2016

Used the following configuration and the deployment and update failed on it. Please test with it and make sure it works before going further.

service: serverless-logical-id

custom:
  s3-bucket-name: serverless-logical-id-${opt:stage, self:provider.stage}

provider:
  name: aws
  runtime: nodejs4.3
  apiKeys:
    - myFirstKey
functions:
  hello1:
    handler: handler.hello
    events:
      - s3: ${self:custom.s3-bucket-name}
      - schedule: rate(10 minutes)
      - http:
          path: users
          method: get
      - http:
          path: users
          method: post
          cors: true
      - sns: topic-name
  world1:
    handler: handler.hello
    events:
      - http:
          path: users/{number}
          method: post
          authorizer: hello1
      - sns: topic-name
      - s3: 
          bucket: ${self:custom.s3-bucket-name}
          event: s3:ObjectRemoved:*

@flomotlik
Copy link
Contributor

There is also a problem with updating an existing service as the the Code is only looking for the new deployment bucket and ignoring the old one. Adding the following to lib/plugins/aws/index.js makes it possible to detect the correct bucket, but still fails in other places. Please take a look so its possible to deploy on top of an existing service/bucket as well.

     ).then((result) => {
       const found = _.find(result.StackResources,
-        { LogicalResourceId: 'S3BucketServerlessDeploymentBucket' });
+        { LogicalResourceId: 'S3BucketServerlessDeploymentBucket' }) ||
+      _.find(result.StackResources,
+          { LogicalResourceId: 'ServerlessDeploymentBucket' });
       return found.PhysicalResourceId;
     });
   }

@ghost
Copy link

ghost commented Jan 18, 2019

how can create an event to already exist Bucket with CF ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants