{"id":342,"date":"2024-12-21T17:59:04","date_gmt":"2024-12-21T17:59:04","guid":{"rendered":"https:\/\/livingdevops.com\/?p=342"},"modified":"2025-07-05T10:35:49","modified_gmt":"2025-07-05T09:35:49","slug":"terraform-to-deploy-aws-lambda-function-with-s3-trigger","status":"publish","type":"post","link":"https:\/\/livingdevops.com\/devops\/terraform-to-deploy-aws-lambda-function-with-s3-trigger\/","title":{"rendered":"Terraform To Deploy AWS Lambda Function With S3\u00a0Trigger"},"content":{"rendered":"\n<p>The purpose of this blog post is to build a real-world DevOps automation project using Python, AWS Lambda, and Terraform. If you are someone interested in learning\/building a cool DevOps project this is for you.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">DevOps real-world AWS Lambda automation project that you can add to your&nbsp;resume<\/h3>\n\n\n\n<h3 class=\"wp-block-heading\">Scenario<\/h3>\n\n\n\n<p>Our team receives numerous files from a third-party vendor, who uploads them to an S3 bucket. These files are suffixed with date stamps. Over time, we accumulated over a thousand files, which presented a challenge since S3 doesn\u2019t allow sorting objects by date when there are over 1,000 objects.<\/p>\n\n\n\n<p>Our team performs daily checks, downloading the current day\u2019s file to process the information. However, they struggled to sort and locate the latest files efficiently. To address this issue, we developed a Lambda function that organizes files in a specific path into folders structured by year\/month\/day.<\/p>\n\n\n\n<p>Good read: <strong><mark><a href=\"https:\/\/livingdevops.com\/devops\/aws-lambda-tutorial-python-terraform\/\">Building a production Lambda function that monitors IAM access keys and sends automated email alerts using boto3 and AWS\u00a0SES.<\/a><\/mark><\/strong><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Implementation<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>I will use Terraform to provision the Lambda function.<\/li>\n\n\n\n<li>I will use Python as Lambda runtime.<\/li>\n\n\n\n<li>Python script will pick the files uploaded to a path and move them to their respective folder with year, month, and date.<\/li>\n\n\n\n<li>S3 notification will trigger the Lambda (When any new files get uploaded to the bucket on a path)<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Prerequisite:<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Basic understanding of AWS services such as Lambda, S3, IAM, etc.<\/li>\n\n\n\n<li>Basic understanding of Python and boto3 SDK<\/li>\n\n\n\n<li>Basic knowledge of Terraform.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">Project setup<\/h3>\n\n\n\n<p>Creating the file structure as below.&nbsp;<code>.tf<\/code> files will store the Terraform code, and the Python code will be in <code>lambda_functions\/main.py<\/code><\/p>\n\n\n\n<p><code>pre-setup-script.sh <\/code>will be used to test the Lambda function. This script will create random files in the bucket and this event will trigger the Lambda function.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*BLAzSNv0PH1aYO0cJiRgTg.png\" alt=\"\"\/><\/figure>\n\n\n\n<p><strong>Before we write Terraform, let me create a bucket with the name <\/strong><code><strong>inbound-bucket-custome<\/strong><\/code><strong> and a folder <\/strong><code><strong>incoming<\/strong><\/code><strong>&nbsp;. I will also create a bucket to store Terraform state\u200a\u2014\u200a<\/strong><code><strong>my-backend-devops101-terraform<\/strong><\/code><\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p><strong>Note:<\/strong> <a href=\"https:\/\/github.com\/akhileshmishrabiz\/Devops-zero-to-hero\/tree\/main\/AWS-Projects\/aws-lambda-with-s3-trigger-terraform\" rel=\"noreferrer noopener\" target=\"_blank\"><strong>I have placed all the code used in this blog post in my public GitHub repository. You can find it with this link.<\/strong><\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Python script for Lambda&nbsp;function<\/h3>\n\n\n\n<p><code>lambda_functions\/main.py<\/code><\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*7oKwUCc2inH5z1mx0GXKyw.png\" alt=\"\"\/><\/figure>\n\n\n\n<p>This code will perform the below functions.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Retrieves the S3 bucket and file path from an environment variable (<code>BUCKET_PATH<\/code>).<\/li>\n\n\n\n<li>Lists all files in a specific directory (<code>prefix<\/code>) within the S3 bucket.<\/li>\n<\/ul>\n\n\n\n<p>For each file:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Extracts the <code>year<\/code>, <code>month<\/code>, and <code>date<\/code> from the file name.<\/li>\n\n\n\n<li>Constructs a new path for the file based on this date information.<\/li>\n\n\n\n<li>Copies the file to the new path.<\/li>\n\n\n\n<li>Deletes the original file.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Terraform to deploy an AWS Lambda function with Python&nbsp;runtime<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Setting up Terraform providers and s3&nbsp;backend<\/strong><\/h4>\n\n\n\n<p><code>versions.tf<\/code><\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*CGXDyuKTxVeCve_8XwFmnA.png\" alt=\"\"\/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>Package the Python code as a zip&nbsp;file<\/strong><\/h4>\n\n\n\n<p>The lambda function will require the zip file with Python code. I have used the Terraform data source <code>archive_file<\/code> to zip the Python function code. We have provided the path to the Lambda function and the destination where to store the zip file.<\/p>\n\n\n\n<p><code>lambda.tf<\/code><\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*tdYk14qgSMrXuYAS_K6Owg.png\" alt=\"\"\/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\">Create Lambda function Terraform resources<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>runtime -&gt; python3.11<\/code><\/li>\n\n\n\n<li><code>filename -&gt; location of the zip file with Python code<\/code><\/li>\n\n\n\n<li><code>source_code_hash -&gt; This allows the Terraform to update the Lambda with the code changes<\/code><\/li>\n\n\n\n<li>We will pass the bucket name as an environment variable, Python code will access it using <code>os.getenv(\u201cENV_NAME\u201d)<\/code><\/li>\n<\/ul>\n\n\n\n<p><code>lambda.tf<\/code><\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*sFem5jr3vloVgkc5yTH8ww.png\" alt=\"\"\/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\">Create a Lambda execution role.<\/h4>\n\n\n\n<p><code>lambda.tf<\/code><\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*oYsq_ZIX-fryufg5vG_1aA.png\" alt=\"\"\/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\">Iam policy that allows Lambda to access the files from the&nbsp;bucket<\/h4>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*CgNdDpa_-CZAT7cySCebkw.png\" alt=\"\"\/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\">Attach the policy with the required permissions to the Lambda execution role<\/h4>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*IL3G6bsxKnpGVBk7DJ9Jxg.png\" alt=\"\"\/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\">Create an S3 trigger for the Lambda&nbsp;function<\/h4>\n\n\n\n<p>Lambda needs explicit permission to be triggered by S3 hence we will create lambda permissions before creating the S3 trigger<\/p>\n\n\n\n<p><code>lambda.tf<\/code><\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*PZDrGy6W31MFJxTcQF-ZeA.png\" alt=\"\"\/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\">Create A bucket policy that will allow Lambda to get the bucket objects for the S3 notification trigger<\/h4>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*uxEi7lRfYL7qId8TZe8paw.png\" alt=\"\"\/><\/figure>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h4 class=\"wp-block-heading\">Configure the AWS credentials<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Create an IAM user with only CLI access<\/li>\n\n\n\n<li>Create Access and Security key<\/li>\n\n\n\n<li>Install AWS CLI<\/li>\n\n\n\n<li>Run <code>aws configure<\/code><\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*6uzuc0w9xZ-vwju9X4sXBQ.png\" alt=\"\"\/><\/figure>\n\n\n\n<p><strong><em>Note<\/em><\/strong><em>: We should never expose sensitive such as access keys, or security keys in the repo, or any public place. I will remove these credentials before publishing the blog post.<\/em><\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Deploy the Terraform<\/h4>\n\n\n\n<pre class=\"wp-block-code has-palette-color-3-color has-pale-cyan-blue-background-color has-text-color has-background has-link-color wp-elements-e876bbb50dde8f54f49e51da7afc9972\"><code># Initialize the Terraform\nterraform init\n\n# Plan the Terraform\nterraform plan\n\n# Apply terraform\nterraform apply<\/code><\/pre>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*6WFjSWrSXgS4HurarsTXeQ.png\" alt=\"\"\/><\/figure>\n\n\n\n<p>This will deploy all the resources.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p>Let\u2019s test it. As you can see screenshots below, there are no files in the bucket path.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*2xnOQ46ab-dvEo5_h0bcmg.png\" alt=\"\"\/><\/figure>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*SbBTK9QsuIZ45dMi7oxqiA.png\" alt=\"\"\/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\">Let me create some objects in the bucket, which should trigger the lambda function.<\/h4>\n\n\n\n<pre class=\"wp-block-code has-palette-color-3-color has-pale-cyan-blue-background-color has-text-color has-background has-link-color wp-elements-23801a38a82965a6284cf2347278bcef\"><code>I have created a short script that does it for us\u200a\u2014\u200a<code>pre-setup-script.sh<\/code>## pre-setup-script.sh ##\n#!\/bin\/bash\n\n# Define the S3 bucket name\nS3_BUCKET=\"inbound-bucket-custome\"\n\n# Create 10 files with the format filename-randomnumber-yyyy-mm-dd\nfor i in {1..10}; do\nRANDOM_NUMBER=$((1 + RANDOM % 1000))\nFILENAME=\"filename-$RANDOM_NUMBER-$(date +%Y-%m-%d).txt\"\necho \"This is file number $i\" &gt; $FILENAME\naws s3 cp $FILENAME s3:\/\/$S3_BUCKET\/incoming\/\ndone<\/code><\/pre>\n\n\n\n<p>We will run this script to upload files to the bucket.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*PLR5oGVhXTC3fv0dwHHh6Q.png\" alt=\"\"\/><\/figure>\n\n\n\n<p>This will trigger the lambda function, which will move these files into the new folder structure.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*mDhzdBtwXbKTqlK_7uRr0Q.png\" alt=\"\"\/><\/figure>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/cdn-images-1.medium.com\/max\/1600\/1*vdMcs9vqi7CB821s33zrPQ.png\" alt=\"\"\/><\/figure>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p>That is all for this blog post, I hope you found it useful.<\/p>\n\n\n\n<p><a href=\"https:\/\/github.com\/akhileshmishrabiz\/Devops-zero-to-hero\/tree\/main\/AWS-Projects\/aws-lambda-with-s3-trigger-terraform\" rel=\"noreferrer noopener\" target=\"_blank\"><strong>I have placed all the code used in this blog post in my public GitHub repository. You can find it with this link.<\/strong><\/a><\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p>Connect with me on Linkedin: <a href=\"https:\/\/www.linkedin.com\/in\/akhilesh-mishra-0ab886124\/\" rel=\"noreferrer noopener\" target=\"_blank\">https:\/\/www.linkedin.com\/in\/akhilesh-mishra-0ab886124\/<\/a><\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>The purpose of this blog post is to build a real-world DevOps automation project using Python, AWS Lambda, and Terraform. If you are someone interested in learning\/building a cool DevOps project this is for you. DevOps real-world AWS Lambda automation project that you can add to your&nbsp;resume Scenario Our team receives numerous files from a [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":344,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,10,12],"tags":[],"class_list":["post-342","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops","category-aws","category-tutorials"],"blocksy_meta":[],"_links":{"self":[{"href":"https:\/\/livingdevops.com\/wp-json\/wp\/v2\/posts\/342","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/livingdevops.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/livingdevops.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/livingdevops.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/livingdevops.com\/wp-json\/wp\/v2\/comments?post=342"}],"version-history":[{"count":2,"href":"https:\/\/livingdevops.com\/wp-json\/wp\/v2\/posts\/342\/revisions"}],"predecessor-version":[{"id":880,"href":"https:\/\/livingdevops.com\/wp-json\/wp\/v2\/posts\/342\/revisions\/880"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/livingdevops.com\/wp-json\/wp\/v2\/media\/344"}],"wp:attachment":[{"href":"https:\/\/livingdevops.com\/wp-json\/wp\/v2\/media?parent=342"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/livingdevops.com\/wp-json\/wp\/v2\/categories?post=342"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/livingdevops.com\/wp-json\/wp\/v2\/tags?post=342"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}