Understanding Ephemeral Storage in AWS Lambda

AWS Lambda provides serverless compute service that runs code in response to events and automatically manages the compute resources. One of the key components provided by Lambda is ephemeral storage, which allows temporary storage for function code and runtime.

What is Ephemeral Storage?

Ephemeral storage refers to temporary disk storage that exists during the lifetime of a Lambda function invocation. When the function execution completes, AWS releases the ephemeral storage and removes any temporary data.

Some key characteristics of ephemeral storage:

• Provides high-performance temporary scratch space for Lambda functions
• Storage size ranges from 512 MB to 10,240 MB
• Contents are discarded after the execution completes
• Well suited for temporary files, caches, buffers, etc

Ephemeral storage differs from other storage options like Amazon S3 in the following ways:

• Ephemeral: Temporary, exists during function execution only
• S3: Persistent object storage, retains data after execution

Ephemeral Storage Size

The available ephemeral storage size depends on the amount of memory allocated to the Lambda function. Originally limited to 512 MB, AWS increased the size to sync with memory in late 2022.

Here is the mapping of memory to ephemeral storage:

• 128 MB – 512 MB
• 192 MB – 512 MB
• 256 MB – 512 MB
• 3008 MB – 5040 MB
• 4096 MB – 8192 MB
• 6144 MB – 10,240 MB

To customize the ephemeral storage, configure the memory allocation for your function up to 10,240 MB.

Use Cases

Ephemeral storage enables several useful scenarios for Lambda functions:

Temporary Files

Write temporary files like CSV exports, PDF reports, and other artifacts instead of slower external services. The /tmp directory can hold output files that exceed the response size limits.

Caches & Buffers

Keep frequently used data in memory for low latency access compared to external calls. For example, cache a machine learning model, database connections etc.

Sort & Transform

Good option for ETL jobs, image processing etc that require fast I/O and prefer tmp storage over provisioned disks.

ML Inference

Perform real-time machine learning inference with models loaded in ephemeral storage for cost savings and performance.

Managing Ephemeral Storage

When working with ephemeral volumes, be aware of the storage capacity and lifecycle:

Monitoring Usage

Watch the storage consumption using tools like AWS X-Ray to catch errors before capacity is exhausted.

Handle Full Storage Gracefully

Code defensively for scenarios when temporary storage fills up unexpectedly due to spikes in data. React by dropping older files.

Adjust Capacity

Tune the memory allocation and ephemeral storage higher or lower based on usage data. Balance performance vs. excess capacity.

Best Practices

Follow these guidelines to use ephemeral storage effectively:

• Use Appropriate File System: Choose tmpfs filesystem optimized for speed instead of ext4 in some cases

• Timeout Functions If Needed: Configure timeout to avoid open connections/files leading to resource leaks

• Delete Files Explicitly: Don’t rely on auto-delete, remove temporary files in cleanup handlers

• Handle Errors Cleanly: Catch errors from I/O issues or capacity limits and handle gracefully

• Monitor For Optimization: Track usage metrics with X-Ray/CloudWatch Logs and optimize sizing

Code Example

Here is a Python example to demonstrate using ephemeral storage:

import boto3
import tempfile

s3 = boto3.client(‘s3‘)

def lambda_handler(event, context):

  # Use temporary file
  temp_file = tempfile.TemporaryFile()

  # Write output data to file
  process_data(temp_file) 

  # Upload result to S3  
  s3.upload_fileobj(Fileobj=temp_file, Bucket="results", Key="output.csv”)

  # Close file to remove from ephemeral storage
  temp_file.close()  

This uses the tempfile module to create a temporary file that processes data and uploads results to S3. The file is closed afterwards so Lambda can delete it automatically.

Ephemeral Storage vs EFS

AWS also provides Elastic File System (EFS) that offers persistent shared storage for Lambda functions. Key tradeoffs:

Feature	Ephemeral	EFS
Persistence	Temporary	Persistent
Availability	Function lifetime	Highly-available
Sharing	Private per function	Shared storage
Management	Automatic	Admin controls
Use Cases	Caching, temp files	Shared files & data
Performance	Very fast SSD	Slower network fs

Pricing

A key benefit of ephemeral storage is there is no additional charge for using it. Only the usual Lambda compute charges apply based on function duration, requests and memory allocated.

EFS has a separate hourly charge for storage used similar to other AWS storage services.

Alternatives

Other options if ephemeral storage does not meet your requirements:

• Instance Store Volumes: Temporary disks attached to EC2 virtual machines
• EBS Volumes: Persistent block storage volumes for EC2 instances
• S3: Durable object storage, requires uploading data
• EFS: Persistent shared file system accessed over network

Conclusion

Ephemeral storage brings high-performance temporary storage to Lambda functions, opening up new use cases like caching, batch operations, and machine learning. By understanding the ephemeral capabilities and best practices, developers can build serverless applications faster with reduced operational overhead.

The auto-scaled scratch space pairs perfectly with the ease of use and automation provided by Lambda. Just be sure to plan storage usage carefully and handle errors gracefully within the temporary constraints of ephemeral storage.