Here is a 2500+ word comprehensive guide on uploading files to Amazon S3 using the AWS Command Line Interface (CLI):
Amazon Simple Storage Service (S3) is a popular cloud object storage service widely used by developers to store and retrieve any amount of data, at any time, from anywhere on the web. With its scalability, durability, security, and low costs, S3 has become the go-to solution for everything from static websites to big data lakes.
The AWS Command Line Interface (CLI) provides a unified tool to manage your AWS services directly from the command line. In this step-by-step guide, you‘ll learn how to leverage the flexibility of the AWS CLI to efficiently upload files to S3 buckets.
Benefits of Using the AWS CLI with S3
Here are some of the main reasons why the AWS CLI is a great choice for programmatically interacting with S3 buckets and objects:
- Automation – Script file uploads, downloads, copies, deletions, and more
- Speed – Quickly transfer thousands of files without manually uploading
- Repeatability – Codify workflows to ensure consistency
- Portability – Run on Linux, Windows, macOS environments
- Control – Fine-grained control over S3 operations
Whether you need to occasionally upload some files or automate complex S3 workflows, the AWS CLI fits the bill. Next, let‘s go over how to install and configure it.
Installing the AWS CLI
The AWS CLI is a Python-based tool that can be installed through Python‘s pip package manager.
On Linux and macOS, you typically already have Python available system-wide. Just run:
pip3 install awscli --upgrade --user
For Windows, download and run the MSI installer available on Amazon‘s site:
https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
This will handle installing Python and pip as dependencies on Windows machines before setting up the awscli package.
Make sure you also have Git installed on your system if you want to leverage some helper utilities like aws-vault.
Confirm the AWS CLI version with:
aws --version aws-cli/2.9.19 Python/3.7.4 Linux/5.4.0-1084-aws exe/x86_64.ubuntu.22
Configuring the AWS CLI
The next step is to configure your AWS access credentials and settings. The aws configure command will prompt you for four key pieces of information:
- AWS Access Key ID
- AWS Secret Access Key
- Default region name (e.g. us-east-1)
- Default output format (json, yaml, etc)
Run aws configure and input your keys and settings when promoted. These can be found in your IAM dashboard.
An alternative is to keep your AWS keys in a separate credentials file and reference it in the CLI. This adds some extra security without exposing keys directly on the command line.
Here‘s an example ~/.aws/credentials file:
[default] aws_access_key_id = AYYYYYZZZZZZ3BLXXXXX aws_secret_access_key = bp+8aLLLLLLlleWYQQQQXXXX+chTcYIiyyyyyyr
You can also configure named profiles for different accounts:
[personalaccount] aws_access_key_id = AAAAAAAAAA3BLXXXXX aws_secret_access_key = bppp8aLLLLLLlleWYQQQQXXXX+chTcYIiiyyyyyr [workaccount] aws_access_key_id = AAAAAAAZZZZZZZZZZZZ4 aws_secret_access_key = bBZZZZZZZZZZZZZZZZZZZZQ
And reference them directly using the –profile option:
aws s3 ls --profile personalaccount
Now that the AWS CLI is installed and configured, let‘s go over how to use it to manage S3 buckets and upload files.
Working with Buckets using the AWS CLI
Amazon S3 uses the concept of buckets to organize and store objects (your files). Buckets have a globally unique name across AWS and contain related objects you wish to share access to.
Here are common S3 bucket operations accessible from the AWS CLI:
- Create bucket – aws s3 mb s3://myuniquebucket
- List buckets – aws s3 ls
- Delete bucket – aws s3 rb s3://myoldbucket
- Show bucket permissions – aws s3api get-bucket-acl –bucket myprivatebucket
- Set bucket permissions – aws s3api put-bucket-acl –bucket myprivatebucket –acl private
Buckets can have access permissions set to private, public-read, public-read-write, etc. Private buckets cannot be accessed without authentication.
Some key notes around S3 bucket naming:
- Globally unique name across all S3 accounts
- Lowercase letters, numbers, periods, and dashes
- Must start and end with lowercase letter or number
- 3 to 63 characters long
Now let‘s go over how to upload files!
Uploading Files and Folders to S3
The AWS CLI makes it trivially easy to upload files and folders into your S3 buckets. No need for a GUI–just use aws s3 commands:
Uploading Single Files
aws s3 cp test.txt s3://mybucket/atest.txt
This uploads test.txt from the current working directory and copies it to the mybucket S3 bucket with the key path /atest.txt.
Any existing files will be overwritten without warning, so be careful!
Uploading All Files in Folder
aws s3 cp testfolder s3://mybucket/atestfolder --recursive
The –recursive flag uploads an entire directory‘s contents instead of a single file. All sub-folders and files will be copied over while maintaining the original tree structure.
Very useful for migrating whole static websites!
Uploading Files Matching Patterns
Use wildcards like * and ? to control what gets uploaded:
aws s3 cp testfolder s3://mybucket/atestfolder --recursive --exclude "*.tmp" --include "*.js"
This example would upload only the .js files from testfolder to the S3 bucket excluding any files ending in .tmp.
The include/exclude pattern filters give you precision over exactly which files match.
Setting Metadata on Uploads
Object metadata allows you to set custom attributes on files uploaded to S3:
aws s3 cp test.txt s3://mybucket/atest.txt --metadata createdDate=20230205 </preLater, you could search or filter S3 objects by this custom metadata!
Other supported file options include:
- --acl public-read (permissions policy)
- --content-type image/jpeg
- --cache-control max-age=10000
- --content-disposition (control download file name)
Refer to Amazon‘s S3 developer guide for additional file upload options.
Downloading Files from S3 Buckets
Just as uploading has aws s3 cp, downloading follows similar syntax:
aws s3 cp s3://mybucket/key.txt key.txt
This copies key.txt from inside the S3 bucket down to your current working directory under just key.txt.
You can also copy directly to or from compressed archives for convenience:
aws s3 cp uploads.zip s3://mybucket/archives/uploads.zipaws s3 cp s3://mybucket/archives/backups.tar.gz backups.tar.gz
All the filtering and options we covered for uploads also apply for downloads:
aws s3 cp s3://mybucket/data/ logs/ --recursive --exclude "*.log"
This would grab all files except logs from the S3 data path into the local logs folder.
Copying Files Between Buckets
A handy AWS CLI feature for S3 is quickly copying objects between buckets or accounts, including cross-region.
For example:
aws s3 cp s3://mybucket/log.txt s3://archivebucket/logs/log_backup.txt
This command copies log.txt from mybucket and places the copy into archivebucket under a new key of logs/log_backup.txt.
You can leverage this for:
- Aggregating shared logs, metrics, or data into central repositories
- Maintaining redundant backups in different regions
- Transferring static assets across accounts
Note the AWS CLI copy does not delete the original by default. You need to use aws s3 rm after to delete it if desired.
Generating Pre-Signed URLs for Temporary Access
When uploading files to private S3 buckets, you normally want to tightly restrict access. However, there are cases where you need to share something temporarily outside your account without making the bucket fully public.
Pre-signed URLs solve this issue by granting time-limited access tokens.
Here is an example to generate a pre-signed URL valid for 1 hour:
aws s3 presign s3://myprivatebucket/shares/sample.jpg --expires-in 3600
Anyone in possession of the generated URL can access the exact sample.jpg file without any authentication for up to 1 hour. After the expiry, the same URL will be denied access.
Pre-signed URLs are fantastic for selectively handing out temporary credentials on uploads you wish to share in a structured way.
The AWS CLI documentation covers advanced options like binding IP ranges, allowed HTTP verbs beyond GET, and auto-correcting object paths.
Using AWS CLI Sync for Large Data Sets
When dealing with tons of files, it‘s inefficient to continually cp or mv them. The aws s3 sync operation makes this simpler:
aws s3 sync data/ s3://bucketname/data
This recursively transfers new and updated files from your data/ folder into S3.
Any files deleted locally will also get cleaned up remotely in the destination S3 path /data automatically.
The sync works similar to rsync and gives you changset synchronization from endpoint to endpoint.
You can run this on a schedule with cron jobs to maintain an always up-to-date mirror of your critical data.
Some helpful sync flags:
--size-only (only check filesize without checking hash) --exclude "*.temp" --content-type image/jpeg
The sync operation is tremendously helpful for maintaining live states between your local workstations, build servers, S3 buckets, and other data sources you connect with the AWS CLI.
Automating File Transfers with the AWS CLI
A major benefit of using the CLI for your S3 workflows is scripting repetitive tasks and infrastructure management.
Here are some examples that demonstrate ways to incorporate aws s3 commands into deployment automation:
Schedule large uploads during off hours:
0 1 * * * aws s3 sync /var/log s3://logstorage --exclude "*.gz"
This cron job would fire a sync at 1AM every day to copy over server logs, excluding any you‘ve already compressed, into S3 cheap storage.
Centralize build artifacts from CI/CD pipeline runs:
aws s3 cp ./dist s3://releases/1.2.34/artifact.jar
Stitching the AWS CLI into Docker containers opens possibilities like directly uploading assets from ephemeral builder images. No need to persist build servers.
Take regular data snapshots from containers:
docker run --rm awscli aws s3 cp /var/myapp/data s3://backups
Fire one-off CLI uploads from your local workstation:
aws s3 mv important.doc s3://personalfiles/taxdocs/previousyears/
The AWS CLI allows you to incorporate S3 usage directly alongside other standard CLI tools for writing infrastructure code. Treat it as just another Unix-style building block.
Over time, all these little automations will pay dividends!
Security Best Practices with S3
When working with sensitive data, make sure you follow security best practices:
- Use IAM policies with least privilege permissions
- Enforce encryption in transit (SSL) and at rest (server-side encryption)
- Turn on access logging for audit trails
- Require MFA delete to recover or destroy files
- Enable object versioning to recover from mistakes
The AWS CLI complements builtin S3 safeguards by letting you script complex file management workflows.
Some other measures like hosting a static website directly from S3 can improve resilience to attacks compared to managing your own file servers.
Evaluate your risk scenarios, follow IAM separation principles, and take advantage of S3 protections for robust and safe file storage and serving.
Pricing Considerations for Data Storage and Transfers
One last thing to note when moving significant amounts of data into Amazon S3 is to understand the pricing model for requests, storage, and data transfers:
- Storage - Base price per GB stored per month. GS tiers for discounts.
- Requests - Charged per 1,000 GET/PUT requests.
- Data transfer in - Free, no charge for inbound data.
- Data transfer out - A few pennies per GB of data transferred out.
Monitor your AWS usage dashboard as you ramp up application usage and cloud data gravity. Placing data near compute such as EC2 instances can help minimize excessive transfer fees.
Compare against running your own physical storage hardware and backup solutions. S3 tends to offer extreme economies of scale for safely storing vast amounts of data cost effectively.
Wrapping Up
This guide covered a ton of ground around using the AWS Command Line Interface for efficient S3 bucket file management.
We installed the AWS CLI, configured access credentials, created buckets, uploaded and downloaded files with various options, automated transfers, generated shareable links, handled security, and lots more!
The AWS CLI removes the need to manually click around the S3 console and also unlocks sophisticated scripting potential. It‘s a must-have tool for developers working in the cloud.
Hopefully these examples provide a broad picture of everything you can achieve directly from your terminal when working with Amazon S3 storage by leveraging the AWS CLI.
Let me know in the comments if you have any other S3 or CLI questions!
