Sat Apr 7 2018 #ssh

SSH Tips

Table of Contents

Most people I work with have used OpenSSH but aren’t too familiar with its features. It’s got a lot of creature comforts! It also has powerful security features some people misuse by mistake, and that’s dangerous!

There’s no real TL;DR here, but if you need the highlights:

  • use ssh keys to avoid typing passwords into servers all day
  • use ssh-agent to avoid typing your passphrase all day
  • use ProxyJump or ForwardAgent to avoid copying your secret keys onto jump servers
  • use HashKnownHosts so your ~/.ssh/known_hosts file is less useful to attackers if compromised
  • use other ssh options to make your life easier

Using SSH Keys

SSH keys are a huge convenience. If you’ve never had one before you’re going to want to try this right away; it will change your life!

Generate a Key-Pair

Generate a 4096-bit RSA key by running ssh-keygen -t rsa -b 4096. Remember the passphrase (a password that protects your secret key). Don’t use an empty passphrase, because I’m going to show you a related feature to make reentering the passphrase a rare necessity instead of a constant annoyance.

jsw@athena:~$ ssh-keygen -t rsa -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/jsw/.ssh/id_rsa):     
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/jsw/.ssh/id_rsa.
Your public key has been saved in /Users/jsw/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:tQs9uD6qRk6n6cBCsYEPnjkKk73GnGcE9OQ/1v7DWBM jsw@athena.local
The key's randomart image is:
+---[RSA 4096]----+
|  . .            |
|.. +             |
|oo. o     .      |
|.=*. . . +E.     |
|+Bo . + S +.     |
|+=.+o..o ooo     |
|o O+o+  o+..     |
| o +=  .o.o      |
|   oo.......     |
+----[SHA256]-----+

You’ll find two files in your ~/.ssh/ directory, named (by default) id_rsa and id_rsa.pub. The first file, id_rsa, contains your ssh secret key. Don’t ever share the secret key file. Don’t copy it onto servers. Notice the secret key file is only readable by you:

jsw@athena:~$ ls -l ~/.ssh/id_rsa*
-rw-------  1 jsw  staff  3326 Apr  7 08:59 /Users/jsw/.ssh/id_rsa
-rw-r--r--  1 jsw  staff   742 Apr  7 08:59 /Users/jsw/.ssh/id_rsa.pub

The public key file id_rsa.pub is readable by everyone. That’s what we want! You can share the contents of the public key file with anyone. You will copy the public key onto servers to allow yourself to login using your private key.

The secret key and public key are collectively a key-pair.

Copy Public Key to Server

Let’s try that. Run the command ssh-copy-id -i ~/.ssh/id_rsa.pub <your test server name> and the tool will copy the contents of your public key file into a file on your server called $HOME/.ssh/authorized_keys. If there are already some keys there, it appends the new one at the end. If you omit the -i option it will heuristically search for keys; see the man page for details.

jsw@athena:~$ ssh-copy-id -i ~/.ssh/id_rsa.pub hera
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/Users/jsw/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

Number of key(s) added:        1

Now try logging into the machine, with:   "ssh 'hera'"
and check to make sure that only the key(s) you wanted were added.

If your computer is missing ssh-copy-id you can manually copy the contents of the .pub file to your $HOME/.ssh/authorized_keys file on the server.

Make sure the authorized_keys file is not writable by anyone except you! If the permissions are wrong, key authentication won’t work. Run the command chmod 0644 ~/.ssh/authorized_keys if you need to fix it.

jsw@hera:~$ ls -l .ssh/authorized_keys
-rw-r--r-- 1 jsw jsw 4105 Apr  7 13:19 .ssh/authorized_keys

Try it!

Try to login to your test server using the key:

jsw@athena:~$ ssh hera
Enter passphrase for key '/Users/jsw/.ssh/id_rsa':
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.14.14-x86_64-linode94 x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
jsw@hera:~$

It worked! But we had to type a passphrase; how is that better than typing a password? I’ll cover this in more detail throughout this post, but the highlights are:

  • you can use ssh-agent to reduce the frequency of passphrase-typing
  • your passphrase is never transmitted outside your computer
  • your secret key is also never transmitted outside your computer
  • it’s considered safe to use just one secret key (and therefore, passphrase) for every server you login to; no need to store tons of unique passwords in a password manager
  • you could remove the passphrase if you really wanted to (common for automation)
  • usually no idiotic I.T. department policies forcing you to have a passphrase with two digits, a special character, three case-transitions, and at least one lunar phase emoticon
  • usually no idiotic I.T. policies forcing you to change the passphrase monthly

Setup ssh-agent to Reduce Passphrase Reentry

ssh-agent is a small program that stores the decrypted version of your secret key in RAM. You can run it once when you boot up your PC and load key(s) into it for as long as you want. ssh will communicate with ssh-agent so you don’t need to reenter your passphrase as long as you have your secret key loaded in the agent!

I’m assuming you use the bash shell. Add the following to your $HOME/.bash_profile file; then, start a new terminal session.

### SSH Agent
export SSH_AUTH_SOCK=$HOME/.ssh/agent.sock
alias start-ssh-agent="rm $SSH_AUTH_SOCK ; ssh-agent -a $SSH_AUTH_SOCK"

In the new terminal (and after each time you reboot your PC), just run start-ssh-agent, then load your secret key using ssh-add:

jsw@athena:~$ start-ssh-agent
rm: /Users/jsw/.ssh/agent.sock: No such file or directory
SSH_AUTH_SOCK=/Users/jsw/.ssh/agent.sock; export SSH_AUTH_SOCK;
SSH_AGENT_PID=23606; export SSH_AGENT_PID;
echo Agent pid 23606;
jsw@athena:~$ ssh-add
Enter passphrase for /Users/jsw/.ssh/id_rsa:
Identity added: /Users/jsw/.ssh/id_rsa (/Users/jsw/.ssh/id_rsa)

Now, connect to your server. You will not need to supply the passphrase!

jsw@athena:~$ ssh hera
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.14.14-x86_64-linode94 x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
jsw@hera:~$

We set this up in ~/.bash_profile so the ssh program knows where to look for the agent socket. You won’t have to paste any environment variables or jump through other hoops to use the agent, even in many terminals.

Unload Keys From ssh-agent

To unload all your keys without stopping the ssh-agent program, run ssh-add -D:

jsw@athena:~$ ssh-add -t 20h
Enter passphrase for /Users/jsw/.ssh/id_rsa:
Identity added: /Users/jsw/.ssh/id_rsa (/Users/jsw/.ssh/id_rsa)
Lifetime set to 72000 seconds

To unload a key automatically after a time interval, use ssh-add -t <how long> when loading it. For example:

jsw@athena:~$ ssh-add -t 20h
Enter passphrase for /Users/jsw/.ssh/id_rsa:
Identity added: /Users/jsw/.ssh/id_rsa (/Users/jsw/.ssh/id_rsa)
Lifetime set to 72000 seconds

List Loaded Keys

To see what keys have been loaded, just run ssh-add -l.

jsw@athena:~$ ssh-add -l
2048 SHA256:icSaAtKP2Kg5t562Y7bDfLDdiMNoO1BBMxyLrp576uA /Users/jsw/.ssh/id_boomer_rsa (RSA)=

Your SSH (client!) Configuration File

You should configure ssh to be more convenient for your use. Let’s imagine your workplace has a server with a verbose name and you frequently have to login, like ssh su.per.ca.li.fra.gil.is.tic.ex.pi.a.li.do.cious. Typing that is horrible! You can add a stanza to your ~/.ssh/config file to give it an alias super and refer to the real Hostname:

Host Alias

Host super
    Hostname su.per.ca.li.fra.gil.is.tic.ex.pi.a.li.do.cious

Alternate Username

Maybe you’ve got a different login name for some of your company’s legacy servers, all named *.example.com. You can use a wildcard and the User parameter to make this easier:

Host *.example.com
    User jeffw

Alternate SSH Port

Alternate ssh server ports (and serial console via ssh) are a breeze. You can combine many options in a single stanza, too:

Host console.router1-ord1.example.com
    Hostname serialconsole1-ord1.example.com
    Port 2005

ProxyJump

Your org might have one or more jump hosts allowing you access to resources protected from the public Internet. This may seem inconvenient, but the ProxyJump option makes it mostly transparent. You can even chain multiple jump hosts together if your company has two or more layers of jump hosts to navigate:

Host jump1.example.com
    Hostname jump1.example.com

Host *.example.com
    ProxyJump jump1.example.com

ForwardAgent

If you need to login to a jump host and then ssh/scp to others routinely, but without configuring the other hosts with ProxyJump in your ~/.ssh/config file, you can use ForwardAgent to trust your jump host.

Host jump1.example.com
    ForwardAgent yes

This is much safer than copying your secret keys to the jump host, but be aware that you are still trusting the jump host while connected to it.

HashKnownHosts

One more security option, and then we’re done for today. Your ssh client maintains a known_hosts file of every server you’ve connected to. You might have one on various jump boxes, too.

If an attacker gains access to a PC where you use the ssh client, they may look for all the known_hosts files and then try to break into those systems. If they manage to access your ssh secret keys, you can be sure they will try this, because known_hosts would basically be a directory of everything they can probably gain entry to using your identity!

OpenSSH will now allow you to hash the known_hosts file. This makes it less useful to attackers, but the ssh client can still use it to check host keys. For example, outside of any stanza in ~/.ssh/config, add:

HashKnownHosts yes

If you want to hash all of your existing known_hosts entries, you can use ssh-keygen to do that. Run ssh-keygen -H.

Special-Purpose Key Files

Your department might have a shared emergency key for certain functions, like serial console access. In that case, you want to use that secret key file (IdentityFile <filename>) and avoid using keys in your ssh-agent (IdentityAgent /dev/null). Let’s combine a few options common to using serial console servers in emergency situations:

Host console.router1-ord1.example.com
    Hostname serialconsole1-ord1.example.com
    Port 2005
    User emergency
    IdentityAgent /dev/null
    IdentityFile ~/.ssh/secrets/id_console_example_rsa

Multiplexing Sessions

Your org might have two-factor authentication setup on its servers, but perhaps you need to login repeatedly during the day (or use scp, etc.)

You can configure a control socket to allow several ssh processes to share one underlying connection. As a side-effect, you’ll only have to complete two-factor auth when you initiate the first connection.

Host jump1.example.com
    ControlMaster auto                      # auto-create "master" client-instances when needed
    ControlPath ~/.ssh/controlsocks/%h      # %h expands to the hostname
    ControlPersist 600m                     # remain open for 600 minutes

Keep-Alive Through Firewalls

Your org might have a firewall that drops TCP sessions after some inactivity, but perhaps you need to remain logged into a host for much longer. Use ServerAliveInterval:

Host jump1.example.com
    ServerAliveInterval 120

Why You Shouldn’t Copy SSH Keys to Jump Hosts

The most important thing to know about SSH is what not to do. Don’t copy your secret key onto jump hosts. EVER. It’s unnecessary; read the ProxyJump section if you need to know how to avoid it.

Why shouldn’t you do this? I’ll illustrate. If the JumpBox pictured, below, is compromised by an attacker, they might steal any ssh secret keys on the box. The keys could allow them to break into other parts of your company’s network, download all your code from GitHub or BitBucket, etc. and do it all while impersonating you!

SSH Connection Diagram Showing JumpBox

One hacked Jump Box full of secret keys can snowball fast. Avoid that problem by never copying your secret keys onto a jump box.

If you notice a coworker has secret keys on a jump box, please, link them to this post. If your company’s op-sec people have a clue, point out the problem to them so they’ll establish a policy and scan for secret keys to identify colleagues who don’t know they’re doing something hazardous (and unnecessary!)

Change Your Key’s Passphrase

If you need to change the passphrase of an ssh secret key file, use ssh-keygen -p -f <secret key filename>, for example:

jsw@athena:~$ ssh-keygen -p -f .ssh/id_rsa
Enter old passphrase:
Enter new passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved with the new passphrase.

Generate a Public Key From a Private Key

If you’ve misplaced your public key file, that’s no problem. You can generate a new one from your secret key file. Just run ssh-keygen -y -f <secret key filename>, like:

jsw@athena:~$ ssh-keygen -y -f ~/.ssh/id_rsa
Enter passphrase:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9guHPiSoZDwOeppeaIsKCW+gGAgVr27TqJXNVKOW4yFVd7HFPlJBRFAhurTsSzbasXlM667TesfB1jTkJYOuPtWUYCkY6vd82mO0ZQK/mVUg76A//F8+uf7i5tJMnnNu+flxwtQ/VUdyhsVBizsMrM2oNnCE8cZCwT1ggmh6Hw9p+knoHLby0Ac/zIgiBQDY9X+WFUsZmemcfIvww3npcAR/eHWxx28tkeSVVUJMDzd4Y5kEa1M5tLBuxZoAkba/DPRdZFT2KqgxaJz2iM6rB5evbpqNbFiH/inbLpfcFOsC+TmIc9tAjZCYNBdTeQPo8NRKLKNpvz2EKGemhcEqHQsubwPt7XiKn7AMPxnq9F8p58T1y9QxWo3T2ZSVP/38sz/OGTE7X1eMUUM5UMmzJAfdTTSTAPellps7Zu7zkqQBkNq8AOZ2k3/CWtTJrlLgNSMWfdfUuqbaR4DH05pyhkm3wPWNiWXH9u0dwJeNv0TxUH17KG0svLxJWRxmoJ7q2JEU5su97ZLWxl+lVrgb1N4bJhO3vt0AzQd2obIsVG0yTM8sfOfQ4TPQgK7kHXvisEJn02QrOBk+r6bGxgUOK/EwN7uTHKIoO0bXflIa1pVAJdUay/4R6oBwJnzNGJGlQHH4Nds2WN2gjDlEroyy5X16r3gpgqrQaV0c/axH8SQ==

You can simply redirect the output to a new .pub file if you want, such as ssh-keygen -y -f ~/id_rsa > ~/id_rsa.pub.