How to Use Linux SCP Command to Securely Transfer Files

The SCP (Secure Copy) command allows you to securely transfer files between a local and remote host or between two remote hosts. It uses SSH (Secure Shell) for data transfer, providing the same authentication and same level of security as SSH.

In this comprehensive guide, I cover everything a systems administrator, DevOps engineer, or full-stack developer needs to know about using SCP including:

• Deep dive on SSH and SCP protocols
• Installation & configuration steps
• SCP syntax and usage examples
• Automating transfers safely
• Common issues and troubleshooting

So let‘s get started.

An In-Depth Look at SSH and How SCP Leverages SSH

To understand how SCP provides secure transfers, you need to understand SSH.

SSH (Secure Shell) is a cryptographic network protocol that provides administrators and developers shell access over an encrypted connection.

Some key capabilities SSH enables:

• Strong authentication – Via passwords or public-key cryptography
• Encrypted data transfer – Using algorithms like AES, Blowfish, etc.
• Secure channel for network services – Like remote command execution, file transfers, VPN tunneling

SSH has become the de-facto standard for securely accessing Linux and UNIX-based systems. According to Statista, 98% of surveyed developers use SSH.

To enable this secure channel, SSH uses a client-server model. The SSH server daemon (sshd) listens on the host for incoming connections. Users then connect via the SSH client program (ssh) to access the server.

Once connected, all data flowing between the client and server is encrypted. By default this uses 128-bit AES, but stronger ciphers can be configured.

Understanding the SSH Encryption Handshake

The way SSH enables secure encrypted sessions comes down to its encryption handshake:

1. The SSH client contacts the server
2. The client and server agree on encryption algorithms to use
3. They generate session keys to encrypt the SSH tunnel
4. The tunnel enables secure communication after the handshake

This handshake uses public-key cryptography for several stages:

• Server authentication – The server sends its public host key which the client validates against its known hosts
• Key exchange – A shared secret is derived securely by the client and server to generate symmetric keys
• User authentication – Users can provide their public key as identification instead of a password

Public-key cryptography provides the foundation for strong 2-way authentication and secure encrypted data transfer via SSH.

Why SCP Relies on SSH for Security

SCP is completely dependent on SSH for its underlying encryption, data transfer, and authentication.

It does not have any capability for encryption or authentication on its own.

Specifically, SCP is nothing more than an SSH data transfer utility:

1. It invokes the SSH client with specific data transfer instructions
2. An SSH encrypted tunnel is established to target server by the client
3. Data is transferred over the tunnel through SSH‘s encryption
4. The SSH connection terminates after data transfer

By piggybacking on SSH in this way, SCP conveniently and securely transfers files without reinventing the wheel. Administrators and developers can leverage their existing SSH infrastructure and policies.

This makes SCP and SSH power combinations for secure automated data movement.

Next let‘s install SCP and its SSH dependencies.

Installing & Configuring Secure Shell (SSH)

As discussed above, to use SCP you need to have SSH installed and configured correctly.

On Linux, this entails setting up OpenSSH which consists of two components:

• sshd – The SSH daemon that runs on servers
• ssh – The SSH client used to connect to servers

Here are the commands to install OpenSSH on common Linux environments:

Ubuntu/Debian

sudo apt update
sudo apt install openssh-client openssh-server

RHEL/CentOS

sudo yum update 
sudo yum install openssh-clients openssh-server

Arch Linux

sudo pacman -Syu 
sudo pacman -S openssh

With OpenSSH installed, you need to properly configure sshd_config to ensure connections are secure.

Here are key settings to verify or update:

• Protocol 2 – Only permit protocol 2 connections
• PermitRootLogin no – Disable direct root login
• PasswordAuthentication no – Only enable public key authentication
• Use strong ciphers like AES256-GCM
• Disable old insecure ciphers like AES128-CBC

After updating sshd_config with best practice hardening, restart the SSH service to apply the changes:

sudo systemctl restart sshd

You should also configure SSH public key authentication for yourself (and developers/administrators).

Here is a quick example for how to generate and authorize keys for passwordless logins:

# As user who needs login access   

ssh-keygen -t rsa -b 4096

# Just press enter to accept default location and no passphrase 

ssh-copy-id -i ~/.ssh/id_rsa.pub user@server  

This will securely copy your new public key to authorized_keys on the server so you can SSH in without passwords going forward.

With SSH installed and properly hardened on your systems, you are ready to start securely transferring files with scp!

Using SCP for Secure File Transfers

Now that you understand ssh and have it configured correctly, using scp for encrypted data transfer is very straightforward.

The syntax of scp is simply:

scp [options] source destination

Where:

• source – Path of local file/directory to copy from
• destination – Remote directory to copy source to in user@host:path format

Let‘s go through some common examples.

Copy Local File to Remote Server

To copy a file from your local system up to a remote server, provide the source file path and destination location in user@host:path format:

scp local-file.txt user@192.168.1.10:/remote/destination/path  

After authenticating, this will securely copy local-file.txt to /remote/destination/path on remote host 192.168.1.10

Copy Remote File to Local Host

To download a file from a remote server, simply flip around the source and destination:

scp user@192.168.1.10:/remote/file.txt /local/download/path

The file file.txt will be securely downloaded from the remote host to the local directory /local/download/path

Transfer Entire Directories

You can recursively copy entire directories hierarchies using the -r option:

scp -r /local/website user@host:/var/www/new-site 

Here we recursively copied /local/website and all its contents to /var/www/new-site/ on the remote host.

This enables easy mirroring of entire directory structures from local to remote or between multiple remotes.

Automating SCP Transfers

You can easily script SCP commands to enable automated secure data transfer. For example:

#!/bin/bash

LOG_DIR="/var/log/app"  
DEST_HOST="backupsrv"
DEST_PATH="/backups"

scp -r $LOG_DIR user@$DEST_HOST:$DEST_PATH  

Here we have a simple script that runs daily to copy over the application logs from our web server to a central backup location.

Built-in utilities like cron combined with scp make building automated, secure data pipelines straightforward.

Some other advanced automation tips:

• Use sshpass if needing to script password authentication
• Look into rsync over ssh for advanced sync/mirroring workflows
• Script pre/post scp commands to integrate into pipelines with checks

Next let‘s discuss some common errors and scp troubleshooting.

Troubleshooting: Common scp issues & solutions

Like any utility, you may run into issues using scp such as connectivity troubles, authentication problems, or transferring files efficiently.

Here are some common errors and fixes:

Lost connection errors

If you lose connectivity during an scp transfer, increase SSH keepalive settings on the server with ClientAliveInterval and ClientAliveCountMax in sshd_config.

Permission denied

Verify your SSH user has correct read/write permissions setup on the target paths you are copying to.

Authentication failure

Double check your password or SSH keys are set up correctly. Try to SSH directly into the host to validate before scp.

No space left on device

Check disk usage on targets with df -h. Remove or move data if needed before scp.

scp is too slow

Use -C to enable compression for faster transfer speeds. Also limit bandwidth with --limit if needed.

For full details on debugging connection issues, see my Comprehensive SSH Troubleshooting Guide.

Using the above examples and tips combined with targeted troubleshooting advice, you can resolve most common challenges with using scp.

scp vs sftp: What‘s the Difference?

Since SCP relies on SSH, some engineers prefer using SFTP instead for encrypted file transfers.

What‘s the difference between scp and sftp?

Both scp and sftp offer encrypted file transfer capability. But:

• scp – Simple file copying utility using SSH as transportation layer
• sftp – Full interactive file access and transfer via SSH, offers feature-rich client and server components

In other words, scp is great for simple automated command line file transfers while sftp provides a full file server system over ssh.

Here is a comparison table highlighting some key differences:

Feature	scp	sftp
Full interactive shell	No	Yes
Easy scripting	Yes	Complex
Transfer directories	Manual recursively	Built-in support
Symbolic links handling	No	Yes
Resume transfers	No	Yes
Remote file system access	No	Yes

So in summary:

• Use scp for simple automated file transfers
• Use sftp for advanced interactive file tasks

Now that you understand their differences, you pick the right secure transfer tool for your needs.

Best Practices for Secure Automated Data Movement

When working with transferring sensitive files like customer data, authentication keys, logs, etc. security is a top priority.

Here are some best practices to follow:

Have Dedicated Transfer User Accounts

Here are some best practices to follow:

Validate Server Host Keys

Add new hosts to ~/.ssh/known_hosts to protect against MiTM attacks when first connecting.

Use SSH Bastion Hosts

Proxy SCP traffic through bastion/jump hosts to isolate data pathways.

Enable SCP Logging

Monitor user activity by enabling verbose sftp logging on servers.

Tunnel SCP Within VPNs

For ultimate data isolation, tunnel scp inside an encrypted VPN tunnel.

By taking steps to harden your SCP transfers following security best practices, you minimize risk of leaks and data exfiltration.

For further reading, see my articles on Creating a Secure Data Pipeline and Hardening SSH Security.

Summary

SCP powered by SSH offers a simple yet secure way for automated data transfer between systems.

Key points to remember:

• SCP relies on SSH encryption and authentication for security
• Install SSH server on any hosts you want to access with scp
• Use either password or public key based authentication
• Script scp with cron and other tools to build data pipelines
• Handle common errors and bottlenecks for smooth transfers

Whether you are a developer deploying code, a sysadmin backing up logs, or data engineer moving datasets, SCP is a versatile tool for encrypted data transport.

Combine it with related protocols like sftp and rsync over SSH for robust data management workflows.

Thanks for reading and let me know if you have any other questions!