Fortifying SSH Security on Ubuntu with ssh-keyscan

SSH is the lifeblood that connects and secures servers, enabling vital administration, automation and data flows. As a prolific full-stack developer and cybersecurity-focused Linux engineer, constantly upholding SSH integrity is rule number one in my book.

A faultlessly effective tool I routinely leverage across all my Ubuntu server fleets is ssh-keyscan – a utility built specifically for collecting and verifying SSH public host keys at scale. In this comprehensive technical deep-dive, I’ll unpack everything ssh-keyscan has to offer, from bolstering key-based trust in SSH connections to automation techniques that take stability to the next level.

Anatomy of an SSH Handshake

To understand ssh-keyscan’s value, it helps to briefly explain what public keys mean for SSH security. The underpinning of any secure SSH link is this initial sequence called the asymmetric cryptographic handshake:

1. Client connects to server’s port 22 requesting SSH access
2. Server presents its public key as unique identification
3. Client verifies this public key matches previous one seen for that server
4. If key is recognized, encrypted SSH session is established using associated private key

This public-private keypair mechanism is what ultimately allows secure tunnels without needing passwords sent over the wire.

But here’s the catch – what prevents malicious imposters from simply swapping out the real public keys? Out-of-band public key verification before SSH connections are made.

Methods like ssh-keyscan provide key continuity checking that confirms legitimacy of keys presented during the handshake. Keeping your fleet’s keys catalogued and vetted is vital for fortifying SSH against various intrusion vectors.

Scanning Server Keys with ssh-keyscan

ssh-keyscan offers a simple yet powerful capability: non-intrusively collecting public keys from UNIX-based servers, without needing account credentials. Under the hood, it efficiently connects to TCP port 22 and requests identity public keys one or more servers present.

For example, grabbing public keys off three of my production database hosts:

ssh-keyscan db01.xyzco.com db02.xyzco.com db03.xyzco.com

Which would display all SSH public keys installed on those hosts, identifying their cryptographic types and fingerprints:

db01.xyzco.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2g73OLg8Y8T9hHxIwA3m0cEzuf2V1FmoxHHrrsQ3uP52OdcjQ0C9hTac+7erUD+q+Z6Y4PhFkCL2s9dgfO2bZqmhQ8lywrJxswcIOzRivcGCgpA1aD7FK7bPzuA3/A2t7viOv1OX4ngTMdEAa8RituwwBZxU72eBe2KRo+UrFU6gUKyktolx+Vr9CxnjIzxakUxsRbhu3nQChhZvdv1X3eOqwLLZ9S0gARfXa9FVdmiYN50KqpMGH287xR8BY839GmZUizoMERf3nmBD0iKVN1VvJ2NUHHJeiwbJW10QR10yl1rwwvkrhYGu0OjLvcZCh4VajeoWMB2AF/hQ3Z3AzF1vF
db02.xyzco.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2g73OLg8Y8T9hHxIwA3m0cEzuf2V1FmoxHHrrsQ3uP52OdcjQ0C9hTac+7erUD+q+Z6Y4PhFkCL2s9dgfO2bZqmhQ8lywrJxswcIOzRivcGCgpA1aD7FK7bPzuA3/A2t7viOv1OX4ngTMdEAa8RituwwBZxU72eBe2KRo+UrFU6gUKyktolx+Vr9CxnjIzxakUxsRbhu3nQChhZvdv1X3eOqwLLZ9S0gARfXa9FVdmiYN50KqpMGH287xR8BY839GmZUizoMERf3nmBD0iKVN1VvJ2NUHHJeiwbJW10QR10yl1rwwvkrhYGu0OjLvcZCh4VajeoWMB2AF/hQ3Z3AzF1vF 
db03.xyzco.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2g73OLg8Y8T9hHxIwA3m0cEzuf2V1FmoxHHrrsQ3uP52OdcjQ0C9hTac+7erUD+q+Z6Y4PhFkCL2s9dgfO2bZqmhQ8lywrJxswcIOzRivcGCgpA1aD7FK7bPzuA3/A2t7viOv1OX4ngTMdEAa8RituwwBZxU72eBe2KRo+UrFU6gUKyktolx+Vr9CxnjIzxakUxsRbhu3nQChhZvdv1X3eOqwLLZ9S0gARfXa9FVdmiYN50KqpMGH287xR8BY839GmZUizoMERf3nmBD0iKVN1VvJ2NUHHJeiwbJW10QR10yl1rwwvkrhYGu0OjLvcZCh4VajeoWMB2AF/hQ3Z3AzF1vF

As you can see, ssh-keyscan fetched all 3 database server’s public keys for identity verification and auditing purposes. The unique fingerprints can be checked against known-good values to confirm no unauthorized key changes occurred.

Automating Scans for SSH Trust Continuity

Intermittently running ssh-keyscan for one-off checks is useful. But where it really excels is ongoing automation. This aligns with security best practices like continuous monitoring and infrastructure as code.

Here are some ways I incorporate recurring ssh-keyscan operations across my Ubuntu environments:

Scheduled Cron Scans

Crontab jobs give simple scheduled automation:

# Scan production web heads nightly  
0 1 * * * ssh-keyscan -H 10.1.2.{1..20} >> /var/log/ssh_keys.log

Dedicated Script

For advanced workflows, I built a custom bash script that:

• Fetches keys from ~200 Ubuntu servers
• Compares latest against previous day
• Diffs new keys against central known_hosts
• Alerts on mismatches indicating tampering
• Uploads daily key audit log to external FTP

Triggers via cron twice a day.

Configuration Management Integration

Within my Ansible playbooks, I have ssh-keyscan stages that frequently update dynamic server group variables with latest host fingerprints. These then populate the global known_hosts inventory for streamlined SSH trust.

Adding ssh-keyscan scans to your existing config management framework allows slain key integrity alongside deployment pipelines.

Publishing Keys in SSHFP DNS

To seamlessly distribute and authenticate server keys company-wide, I integrated ssh-keyscan with SSHFP DNS resource records. My Ansible automation configures our BIND9 nameservers, leveraging ssh-keyscan results piped into domain SSHFP records.

Once populated, SSH clients automatically resolve and cross-check SSHFP records served by DNS against keys servers present. This provides built-in continuity checking transparently for all engineers SSH’ing across our domains.

Here are some best practices I follow when automating ssh-keyscan for comprehensive SSH trust:

• Use cron or scheduled jobs for consistent background scanning – Routinely collect keys to your target audit schedule, at least daily.
• Implement secondary transfer methods like FTP/S3 – Offload finalized scans to isolated storage so findings are centralized and versioned.
• Integrate scans into CI/CD, configuration management, and DNS – Maximize distribution and security checks by propagating keys systemwide.
• Incorporate key mismatch alerting via Slack, SMS or email notifications to qualifying staff.
• Minimize and isolate ssh-keyscan scope using specific hosts lists to only gather required keys.
• Secure storage and access of scanned keys by minimum permissions on key files and encryption via SSH tunneling.

ssh-keyscan Compared to Other Key Scanning Methods

While ssh-keyscan is purpose-built for easily gathering keys at scale, there are a couple other common techniques that achieve similar public key collection:

Manual Copy

Simply SSH in and cat the host keys from /etc/ssh/*.pub can work for small fleets. But this becomes extremely tedious beyond 5-10 servers. Lack of automation causes configuration drift and security assurance gaps over time.

parse_ssh_key utility

Part of my org‘s compliance regime scripts extracting installed SSH public keys via parse_ssh_key output collected by our monitoring agent. This works but only provides point-in-time accuracy and lacks native automation.

Comparatively, ssh-keyscan excels through flexible automation, efficiency at scale, not requiring login or agent installs, and specialization for SSH environment security monitoring.

Integrating with ssh_known_hosts

A core capability unlocked by ssh-keyscan is automatically maintaining the ssh_known_hosts file across large Ubuntu server groups. This contains public key fingerprints of trusted servers as a baseline for authorization checks during SSH handshakes.

Here is how streamlined ssh_known_hosts integration works in practice:

Centralized known_hosts Seed

ssh-keyscan -H -f ssh_seeds.txt > /etc/ssh/ssh_known_hosts

On brand new Ubuntu machines, I pre-seed /etc/ssh/ssh_known_hosts using a verified database I control containing target server public keys. This allows engineers using the workstations to immediately SSH trusts.

Daily Automated Updates

ssh-keyscan -H $(groups.txt) >> /etc/ssh/ssh_known_hosts

In my daily cron job, it appends to ssh_known_hosts any new public keys collected from production server groups. This allows any user on the host to SSH seamlessly as keys naturally update.

Maintaining known_hosts definitely takes some ongoing upkeep. Here are a few key tips:

🔑 Seed new client machines from validated sources like my ssh_seeds repository signed by our SSH CA. This adds pre-approved keys.

🔑 Have automation like ssh-keyscan routinely integrate latest server keys for engineers.

🔑 Follow a change management process for intentionally added keys.

🔑 Filter ssh_known_hosts externally published to minimize exposure.

🔑 Occasionally purge outdated stale entries not seen recently during scans.

Adhering to known_hosts best practices prevents unauthorized keys being implicitly trusted on clients.

Potential Risks and Alternatives

While undoubtedly useful, relying solely on ssh-keyscan does entail some latent threats to consider:

Race Conditions

Since scanning is periodic, a mismatch during the window between checks could go undetected. Using supplementary protections like SSHFP DNS records provides redundancy here.

Manipulated Scanning

If scan session traffic is intercepted before reaching real servers, forged keys could be presented instead. Critical thing is scanning from isolated controlled clients using trusted trajectories.

Deprecated Algorithms

You must ensure the allowed key types match latest guidance, disabling outdated ones per NIST. Periodically check what ssh-keyscan gathers.

Non-Route Cause

Keep in mind public key mismatches don’t definitively indicate foul play. Maintenance, upgrades, etc can legitimately invalidate them. Know your environments and investigate wisely.

Balancing ssh-keyscan with real-time authorization checks via SSHFP records, ongoing penetration testing and input validation, plus multifactor authentication provides comprehensive assurance.

Adding Additional Hosts and Troubleshooting

Expanding ssh-keyscan usage to additional server groups is straightforward:

1. Enable SSH daemon on target hosts if not already running
2. Create simple newline delimited list of IPs/hostnames in file
3. Run ssh-keyscan -H <hostslist.txt> >> /etc/ssh/ssh_known_hosts

Some common troubleshooting points:

• Adjust connection timeouts if some hosts lag using -T 30 (30 second timeout)
• Specify alternative SSH port with -p 22222 if not standard port 22
• Use -v argument for verbose diagnostic output
• Try scanning from separate client subnet in case of network issues
• Validate SSH service actively listening on targets using ss -tan | grep sshd
• Check network firewalls allowing outbound 22 access from ssh-keyscan source

With debug output enabled, you can pick apart where in the protocol handshake failures occur when troublestrikes.

Conclusion

Routine scanning and distribution of SSH public host keys enables transparency and continuity trust checking in Ubuntu environments. As both a security-focused Linux engineer and steadfast defender of SSH stability, ssh-keyscan offers unmatched flexibility securing server fleets at scale.

Integrating recurring ssh-keyscan scans significantly uplifts exposure detection, swiftly alerting on any unauthorized infrastructure alterations. Auditing keys likewise supports seamless maintenance, troubleshooting and SSH hardening.

From automated integration like SSHFP DNS and configuration management, to purpose-built features catering to ops efficiency, ssh-keyscan tackles the ongoing insecurity of ephemeral keys head-on. Reliably answering that pivotal question – who are my servers today? – is imperative considering SSH‘s pivotal role, and no tool answers better than ssh-keyscan.