Testing new ssh keys
You created a new ssh key, but are you actually testing it?
I routinely have to create new ssh keys for system users and also verify that they are working correctly. One possible area of confusion is how ssh keys are found by the ssh client.
While most of this information is not specific to osx, since I typically use a macbook running osx, I'll cover a couple of things somewhat specific to osx.
SSH clients will typically default to searching for a private key that exists in the user's home directory in a subdirectory named ".ssh"
So let's assume that you create an ssh key for a new user, utilizing ssh-keygen
ssh-keygen -t ed25519 -N some-new-passphrase -f id_newuser -C "new user"
Should you omit the -N parameter above you will be prompted for a passphrase. In either case, you should use passphrases in almost all situations to protect your system from lost or stolen private keys!!!
So if all goes well, you will now have a new public/private ssh key pair.
On the remote system, you will typically add the public key (id_newuser.pub in this example) to the .ssh/authorized_keys file.
While not the point of this article, you will also need to understand the permission requirements of the user's home .ssh directory and files. Make sure the directory and files are private to the user, or ssh connections will fail.
Time to test
At the point you actually want to test the connection you have a few different options.
• You an use ssh -T user@host
ssh -i /path/to/key -T user@host
You might see this advice for testing authentication to your github account, however, there are cases where without the allocation of a pseudo-tty, this command will appear to hang. Github allows the use of ssh keys for git authentication, but they are not providing a tty, so this makes sense for github. In general, you are trying to test an actual system login, so in most cases you want to use:
• ssh -i /path/to/key user@host
ssh -i /path/to/key user@host
So what is the problem here?
If you look carefully at the manual for the -i flag, you might find that your assumptions in regards to what it is doing are unexpected. By itself, using -i does not guarantee that the key you are testing is the one actually being utilized. In particular, you might notice that in certain circumstances you aren't prompted for the passphrase you set.
This could lead you to assume that the passphrase for the key wasn't set.
One way to see the issue, is to add a -v to the command, to see debug information for the connection:
ssh -v -i id_newuser user@host
Despite specifying the key we wanted to test, the system has instead used a pre-existing key to authenticate us. The ssh client will try other keys in its default search path locations before it actually attempts to use the key you specified with -i.
The is where the -o option "IdentitiesOnly=yes" comes to the rescue.
ssh -o "IdentitiesOnly=yes" -i id_newuser user@host
Enter passphrase for key 'id_newuser':
Immediately we can see that we are actually using the id_newuser key, because we are prompted for the passphrase we set. When we authenticate, we can now be sure that we only used the -i ("identity") key we passed to ssh. For your own edification you might also use the -v flag to see how the ssh client has attempted to authenticate and make the connection.
So let's assume for a moment you have a gateway/bastion/jump box you utilize for security purposes. In order to get to servers on your internal/private networks, you first need to ssh into the gateway box, and from there ssh into other servers.
With multiple keys, you might have a public key deployed to internal servers, and you want to make sure that is the key being used, and not another key that ssh found in its search path. This is where using the prior command with the -o "IdentitiesOnly=yes" -i path/to/key can save you a lot of pain and suffering, in trying to debug why you are not able to ssh from the gateway to an internal server that has been setup with your public key.
Frustrated sysadmins confronted with this issue may be tempted to try and copy a private key to the gateway server, but you should never do that, as that is not a workaround for a problem you don't actually understand. You should never propagate private keys from your workstation!
The issue is likely that you don't have the proper configuration for your ssh-agent, or you are unknowingly using the wrong key to authenticate to the gateway.
This is where the ssh agent comes into play. The typical way to handle key forwarding is to create (if it doesn't exist) or edit the .ssh/config file on your workstation.
In this example, we assume that the gateway server is "gateway". It's not uncommon for systems to exclude gateways from DNS so as not to attract unwanted attention to their existence and purpose, but for this example, just assume "gateway" is equivalent to an IP address, or perhaps something you added to your local /etc/hosts.
#contents of /home/user/.ssh/config
Host gateway
ForwardAgent yes
This setting tells ssh to make the authenticated ssh key available to the active connection you have on the gateway box. From that point, you should be able to connect/jump to an internal box, which has the authorized public key installed in the .ssh/authorized_keys file. Again the "IdentitiesOnly=yes" will insure that the right key is being forwarded.
Most people who make a lot of use of ssh for administration will likely at some point make a .ssh/config file to setup configuration flags for certain hosts, or alias hosts.
Additionally, many users want the ssh agent to store the passphrase so you aren't continually prompted to enter it (and this also works on windows and linux, although you may have to start the ssh agent manually). Sites like github have some good documentation covering setup and debugging of agent forwarding on osx, windows and linux, in case this is new to you.
One common way to automatically store passphrases is to have a setting like this in your workstation .ssh/config file:
Host *
UseKeychain yes
AddKeysToAgent yes
ServerAliveInterval 30
ServerAliveCountMax 2
The first two entries are options to store passphrases in the ssh agent, so once you've supplied one, you won't be prompted again for that session. UseKeychain yes is a mac specific option, so you'll get an error if you try to use that on windows or linux. Most mac users should recognize the reference to the mac "KeyChain".
For testing purposes, this configuration might be a problem. Once you've supplied authentication, osx is going to store the key in the the ssh agent. For example, you'll notice that you won't be prompted for the passphrase again.
The ssh-add command, despite the implication in the name, will not only add keys to the agent, but will also allow you to list stored keys as well as delete some or all the keys it is keeping in memory.
ssh-add -l
You should see a list of any keys that have been used to authenticate during the session, or in some cases previously, depending on how you've set things up. See the github article linked previously for more details.
256 SHA256:vP.............4 new user (ED25519)
Here we can see the new user key we used is being stored in the agent.
To remove the key:
ssh-add -d id_newuser
Identity removed: id_newuser ED25519 (new user)
Running ssh-add -l again should show that the id_newuser key is no longer being stored in the agent. If you authenticate again, as per the prior instructions you will once again be prompted for the passphrase.
Another option, is to remove all keys from the agent:
ssh-add -D
You should see the message "All identities removed."
—
• You an use ssh -T user@host
ssh -i /path/to/key -T user@host
You might see this advice for testing authentication to your github account, however, there are cases where without the allocation of a pseudo-tty, this command will appear to hang. Github allows the use of ssh keys for git authentication, but they are not providing a tty, so this makes sense for github. In general, you are trying to test an actual system login, so in most cases you want to use:
• ssh -i /path/to/key user@host
ssh -i /path/to/key user@host
So what is the problem here?
If you look carefully at the manual for the -i flag, you might find that your assumptions in regards to what it is doing are unexpected. By itself, using -i does not guarantee that the key you are testing is the one actually being utilized. In particular, you might notice that in certain circumstances you aren't prompted for the passphrase you set.
This could lead you to assume that the passphrase for the key wasn't set.
One way to see the issue, is to add a -v to the command, to see debug information for the connection:
ssh -v -i id_newuser user@host
debug1: Offering public key: /Users/me/.ssh/id_rsa RSA SHA256:.....DM agent
debug1: Server accepts key: /Users/me/.ssh/id_rsa RSA SHA256:....DM agent
Despite specifying the key we wanted to test, the system has instead used a pre-existing key to authenticate us. The ssh client will try other keys in its default search path locations before it actually attempts to use the key you specified with -i.
How can we actually test the new key?
The is where the -o option "IdentitiesOnly=yes" comes to the rescue.
ssh -o "IdentitiesOnly=yes" -i id_newuser user@host
Enter passphrase for key 'id_newuser':
Immediately we can see that we are actually using the id_newuser key, because we are prompted for the passphrase we set. When we authenticate, we can now be sure that we only used the -i ("identity") key we passed to ssh. For your own edification you might also use the -v flag to see how the ssh client has attempted to authenticate and make the connection.
Other uses for "IdentitiesOnly=yes"
So let's assume for a moment you have a gateway/bastion/jump box you utilize for security purposes. In order to get to servers on your internal/private networks, you first need to ssh into the gateway box, and from there ssh into other servers.
With multiple keys, you might have a public key deployed to internal servers, and you want to make sure that is the key being used, and not another key that ssh found in its search path. This is where using the prior command with the -o "IdentitiesOnly=yes" -i path/to/key can save you a lot of pain and suffering, in trying to debug why you are not able to ssh from the gateway to an internal server that has been setup with your public key.
Frustrated sysadmins confronted with this issue may be tempted to try and copy a private key to the gateway server, but you should never do that, as that is not a workaround for a problem you don't actually understand. You should never propagate private keys from your workstation!
The issue is likely that you don't have the proper configuration for your ssh-agent, or you are unknowingly using the wrong key to authenticate to the gateway.
This is where the ssh agent comes into play. The typical way to handle key forwarding is to create (if it doesn't exist) or edit the .ssh/config file on your workstation.
In this example, we assume that the gateway server is "gateway". It's not uncommon for systems to exclude gateways from DNS so as not to attract unwanted attention to their existence and purpose, but for this example, just assume "gateway" is equivalent to an IP address, or perhaps something you added to your local /etc/hosts.
#contents of /home/user/.ssh/config
Host gateway
ForwardAgent yes
This setting tells ssh to make the authenticated ssh key available to the active connection you have on the gateway box. From that point, you should be able to connect/jump to an internal box, which has the authorized public key installed in the .ssh/authorized_keys file. Again the "IdentitiesOnly=yes" will insure that the right key is being forwarded.
Using OSX, and the passphrase isn't being prompted anymore?
Most people who make a lot of use of ssh for administration will likely at some point make a .ssh/config file to setup configuration flags for certain hosts, or alias hosts.
Additionally, many users want the ssh agent to store the passphrase so you aren't continually prompted to enter it (and this also works on windows and linux, although you may have to start the ssh agent manually). Sites like github have some good documentation covering setup and debugging of agent forwarding on osx, windows and linux, in case this is new to you.
One common way to automatically store passphrases is to have a setting like this in your workstation .ssh/config file:
Host *
UseKeychain yes
AddKeysToAgent yes
ServerAliveInterval 30
ServerAliveCountMax 2
The first two entries are options to store passphrases in the ssh agent, so once you've supplied one, you won't be prompted again for that session. UseKeychain yes is a mac specific option, so you'll get an error if you try to use that on windows or linux. Most mac users should recognize the reference to the mac "KeyChain".
For testing purposes, this configuration might be a problem. Once you've supplied authentication, osx is going to store the key in the the ssh agent. For example, you'll notice that you won't be prompted for the passphrase again.
Seeing the ssh agent keys
The ssh-add command, despite the implication in the name, will not only add keys to the agent, but will also allow you to list stored keys as well as delete some or all the keys it is keeping in memory.
ssh-add -l
You should see a list of any keys that have been used to authenticate during the session, or in some cases previously, depending on how you've set things up. See the github article linked previously for more details.
256 SHA256:vP.............4 new user (ED25519)
Here we can see the new user key we used is being stored in the agent.
To remove the key:
ssh-add -d id_newuser
Identity removed: id_newuser ED25519 (new user)
Running ssh-add -l again should show that the id_newuser key is no longer being stored in the agent. If you authenticate again, as per the prior instructions you will once again be prompted for the passphrase.
Another option, is to remove all keys from the agent:
ssh-add -D
You should see the message "All identities removed."
—
Comments
Display comments as Linear | Threaded