Do It Better: SSH

Let's talk about SSH. It is the quintessential Swiss army knife for system administrators. It can browse and transfer remote files, forward ports, proxy and tunnel traffic, but it's most famous for its eponymous job of giving you a remote shell, securely (hence its name). As processors have gotten faster and cheaper, and encryption has become almost frictionless, it's long since replaced Telnet for configuring embedded devices. In fact, even Microsoft acquiesced and made SSH a first-class citizen starting with Windows Server 2019.

You'd think with SSH everywhere, it would be easy to configure, right? Well, you are right. In fact, the default configuration of OpenSSH is pretty secure, but I guarantee you're still not implementing it as well as you could.

I'm going to assume that your operating system either includes or is supported by a recent version of OpenSSH. We're going to configure things to support only the most modern features, so if you're trying to connect with an ancient version of macOS or PuTTY for Windows Vista, you might get shown the cold shoulder. We're dealing with security. Upgrade your software.

Let me state the obvious. Just because SSH is secure, why risk it? You are enabling and exposing a remote access method. That's the whole point, but it bears repeating. Consider how much risk you're willing to take, and don't push a half-assed solution into production.

The first thing to consider is where you will be accessing this remote server from. Just because we can open port 22 to the whole world doesn't mean that we should. Ask yourself:

Some people will tell you that you should never run SSH on the default port of TCP 22. They'll advise you to pick another number. While that may defer script kiddies and automated scans, it's trivial to find an SSH service. For example, I could just scan all of your ports in the span of a few minutes:

Or, I can just type your IPv4 address into Shodan and get a report instantly. In either case, I found that your SSH server is running on port 42069, let's say. I can also see what version of OpenSSH it uses, and begin to see how it might be configured. Don't waste your time with security theater and just leave it on port 22.

SSH relies on public key cryptography to authenticate servers and users. There are four major key types still in use today:

• RSA is the tried and true key generation algorithm. It's the oldest, but widely supported. It does require large keys, which results in slower cryptography.
• DSA was another popular algorithm, but its time has passed. It's been removed from OpenSSH 7.0 and newer, finally.
• ECDSA was the first major challenger to the RSA hegemony. It uses smaller keys without compromising security. However, it's imperfect, and some people believe that the United States government may have backdoored it. (If so, Edward Snowden hasn't told us about it.)
• EdDSA is the newest algorithm that fixes the weaknesses of ECDSA and matches or exceeds the security of RSA. It's also much faster. One specific curve, Curve25519, has taken the cryptography world by storm and is gaining wide support in the industry.

My recommendation is to use EdDSA for everything, falling back to ECDSA or RSA if needed. However, since we're only targeting the newest versions of OpenSSH, let's forget that anything besides Curve25519 exists.

Now, only clients that support Ed25519 host keys can even attempt to log in. This step by itself will break many old IP scanners, botnets, and script kiddies' tools. (I look through my logs sometimes and see a lot of connections failing because ssh-rsa is unsupported.)

Now that we have only the best host key type remaining, let's take a look at the next stage of the SSH connection, when the client and server exchange keys and prepare to set up an encrypted session. Just like with host keys, there are a lot of key exchanges supported. We should only support the most secure ones.

And, just like with host keys, older versions of OpenSSH may not be able to connect. But if all our computers are using, let's say, OpenSSH 8, why bother supporting older versions?

You can run the ssh -Q kex command to see all key exchanges supported by your version of OpenSSH, but there are three that we'll be using:

• sntrup761x25519-sha512@openssh.com provides excellent post-quantum security that doesn't affect performance too badly. We'll use that first.
• curve25519-sha256@libssh.com and curve25519-sha256 provide excellent security and performance. Enable them both. (I'm not sure what the difference is between them — toot at me if you do.)

We're going to leave these off the list for varying reasons:

• The ecdh curves use ECDSA, and we prefer EdDSA. They're fine to enable if needed.
• diffie-hellman-group-exchange-sha256 is older and slower, but still secure. Still, I won't be using this.
• Anything like diffie-hellman-groupN should be avoided.

Go ahead and put the algorithms you want into the sshd_config file, like so:

With all of these multiple-choice directives, an SSH client and server will go down the list, from left to right, and use the first thing that they mutually agree on. Always arrange them from most secure to least secure, and omit ones you don't want to support.

Now, any DH key exchanges will use 3072-bit RSA at a minimum.

Now that both sides can exchange keys securely, we need to look at ciphers. We have far fewer options here, but that's fine because what's offered is quite secure. You can use ssh -Q cipher to see what's supported.

• The aes256-gcm@openssh.com and aes128-gcm@openssh.com ciphers are the most secure, and AES in Galois Counter Mode is hardware-accelerated on most processors. These are what most everyone should use.
• chacha20-poly1305@openssh.com is another secure 256-bit algorithm that may be faster on some embedded devices and older ARM processors.
• aes256-ctr, aes192-ctr, aes128-ctr are older but still secure implementations of AES. They might not be as fast as the AES-GCM ones.

Do not use any of these:

• CBC cipher modes are subject to padding oracle attacks.
• CAST-128, 3DES, and Blowfish use 64-bit blocks which are considered too small these days.
• SHA-1 collisions can be manufactured.
• Rijndael is a pre-standards version of AES.

Let's add this line to our sshd_config file:

Finally, there are MACs — Message Authentication Codes — used to verify that encrypted data has not been modified, intentionally or otherwise, in transit. Again, you can use ssh -Q MACs to view what's supported, and we'll promptly cull this long list.

The SHA-2 family is the most secure, and there isn't much of a performance impact on modern hardware. Sort by number of bits, in descending order.

Avoid or disable anything with the following characteristics:

1. Omit anything using 64-bit or 96-bit digests, as those are too small for modern use.
2. Omit anything using broken algorithms like MD5 or SHA-1.
3. Avoid anything using weak algorithms like RIPEMD-160
4. Avoid anything not using ETM (Encrypt-then-MAC mode), as the alternative of hashing before encrypting can lead to weaknesses.

That doesn't leave us with much, but it leaves us with good options. Add this line to your sshd_config.

Now, we're going to explore the other lines in the sshd_config file. Many of these need little explanation, so I'm going to go through this next section quickly.

When you connect to a new SSH server for the first time, you will be asked to verify the SSH fingerprint, so that you can avoid a future man-in-the-middle attack. Most people will blindly accept it. However, this process can be automated by putting the server's host key fingerprint(s) into DNS.

Pick a public name for your server, and then feed it to the ssh-keygen -r command, like so:

If you get more than two records, then you must have forgotten to delete your unwanted SSH host keys in the previous steps. That's fine. We only want the Ed25519 host key fingerprints, which all begin with the number 4. Ignore the others.

Now, go to wherever you go to edit your DNS zone. This might be Cloudflare, a local BIND server, or your registrar. If you're unlucky enough to be using Microsoft DNS Server, I have you covered, believe it or not.

Next, add two new SSHFP records, one for each line. If you're unluckier still and your DNS host doesn't support SSHFP records (looking at you, GoDaddy and Network Solutions), then you're out of luck.

Note that this is all for naught if you don't use DNSSEC. If your zones aren't signed, please look at a calendar, see what year it is, then ask yourself why you haven't secured your infrastructure yet. Even GoDaddy realized how important secure DNS is to the Internet ecosystem and stopped charging extra for DNSSEC. (Even those scummy capitalists can do the right thing once in a while.)

Now, edit your local machine's /etc/ssh/ssh_config file and add the following two lines:

Now, when you connect to an SSH server for the first time, SSHFP records will be retrieved from DNS, and your computer will help you decide if you're connecting to the right server.

Believe it or not, this is now supported! If using SSH as a native transport for PowerShell remoting sounds like something you want, add this line, substituting the correct path for your particular system:

Now that we've set up OpenSSH, we're ready to expose it to the Internet. However, there is one problem we haven't addressed: we can't log in. Remember, we disabled password-based logins earlier! To make SSH usable and secure, we're going to log in with a keypair.

By this point, we've set up a pretty secure SSH server, but I'm sure my astute and security-conscious readers are getting ready to send me angry toots about me teaching them to expose a remote access method to the Internet with only single-factor authentication enabled. If you're using Kerberos or only allowing certain IP addresses to connect, this may not be necessary. Plus, keys are far less likely to be stolen than passwords. Still, let me show you how to implement MFA via the classic TOTP method.

These steps only apply to Linux systems that use PAM for user authentication, which should be almost all of them these days. On your server, find and install the Google Authenticator PAM module. On Debian-like systems, a simple apt install libpam-google-authenticator will do the trick.

Since we're about to mess with the thing that authenticates users, open up another terminal or SSH session on the server, and keep it open until we're done and we've tested that it works. Otherwise, a typo could lock you out and give you a very hard problem to fix.

Edit the file /etc/pam.d/sshd and add the following line to the top, above the one that ends in pam_nologin.so:

If you've made other changes to your PAM configuration files, then you may need to do some debugging. Add debug to the end of that pam_google_authenticator.so line to generate more debugging output in your logs.

Next, edit your /etc/ssh/sshd_config file, adding this new stuff and making sure these other lines are present, too:

Now, on the server, run the google-authenticator command to generate your secret (as a QR code) and some backup codes. Scan the QR code with your authenticator app (despite the name, it doesn't need to be Google Authenticator) and be sure to save the backup MFA codes somewhere safe.

Moment of truth, again! In a new window, try to SSH into your server again. If all goes well, you'll be prompted for an MFA code before your SSH login completes.

Note that all SSH users will be required to use MFA. If they haven't set up MFA, then PAM will deny their SSH login every time. To work around this, remove MFA enforcement by adding nullok to that PAM.d line we added earlier.

Now that we've configured our servers, we can create profiles for our connections so that we don't need to remember and type in every single detail every time. OpenSSH gives us two files: the system-level /etc/ssh/ssh_config and the user-specific ~/.ssh/config. Both files have the same syntax. Whichever one you choose to edit depends on your personal preferences.

Note that, when you're writing this file, you may be connecting to SSH servers that you do not control. You may want to leave all secure options in here, and let your SSH servers be more restrictive.

SSH isn't hard. It's actually quite simple to configure and use, certainly more so than a modern email server. However, we've taken the time to adjust some default settings, break some compatibilty, and we've hardened things significantly. You can now SSH safely, and that's all that matters.