Resetting SSH Hosts

Infrequently the HPCC will update the identifier for its development or gateway nodes called an SSH host key. These host keys are identified by their "fingerprint". If you try to connect to a gateway or development node after one of these nodes has changed its host key, you'll see a warning like the following:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:bWaEpc+YCgTvppCmDyIbnFTCnjhsGCQOsGGGHjMFqes.
Please contact your system administrator.
Add correct host key in /mnt/home/kopenhaf/.ssh/known_hosts to get rid of this message.
Offending RSA key in /etc/ssh/ssh_known_hosts:950
RSA host key for gateway-04 has changed and you have requested strict checking.
Host key verification failed.

Alternatively, in VS Code you'll see an error like the following pop up:

Warning

If using proxy jumping, VS Code will only inform you that the final host in the chain (i.e., the development node) has changed its key regardless of whether it was the gateway or development node that actually changed it's key. See the instructions below for how to remove both keys.

If you see this error when connecting to a gateway node, check the host key fingerprint against our published list of known hosts. If you see a discrepancy, please file a help ticket immediately as you may be experiencing a genuine security threat. ICER does not currently publish a list of host key fingerprints for the HPCC's development nodes.

If the key fingerprint from the warning message matches our published keys, you are not experiencing a security threat. Rather, you need to update the list of known host keys saved on your local computer. The steps for doing this depend on your SSH Client.

TerminalMobaXTermVisual Studio Code

If you connect to the HPCC via a Mac or Linux terminal, run the following command to clear the host key for the gateway nodes:

ssh-keygen -R hpcc.msu.edu -f ~/.ssh/known_hosts

If you received a warning about the host key when connecting to a development node via proxy jump, replace hpcc.msu.edu with the name of the relevant development node; e.g., dev-amd20.

Alternatively, you can delete the file ~/.ssh/known_hosts. Every time you connect via SSH (including to sites like GitHub) you will be asked to verify the SSH host key fingerprint.

Once your host key(s) have been removed, you'll be prompted to confirm the host key fingerprint the next time you connect via SSH. If you are connecting to the HPCC's gateway (hpcc.msu.edu), check the key against our published host key fingerprints before continuing. If the host key fingerprint matches those we have published, type "yes" and hit enter to accept.

Run MobaXTerm and choose the "Start local terminal" option as shown below:

With a local terminal open, you can use the following command to clear the host key for the gateway nodes:

ssh-keygen -R hpcc.msu.edu 

If you received a warning about the host key when connecting to a development node via proxy jump, replace hpcc.msu.edu with the name of the relevant development node; e.g., dev-amd20.

Alternatively, you can delete the whole known hosts file, either with rm ~/.ssh/known_hosts or by using the Windows file browser to delete either C:\Users\<USERNAME>\.ssh\known_hosts or C:\Users\<USERNAME>\Documents\MobaXterm\home\.ssh\known_hosts, depending on where MobaXTerm stores your SSH information. Deleting the known_hosts file means that every time you connect via SSH (including to sites like GitHub) you will be asked to verify the SSH host key fingerprint.

Once your host key(s) have been removed, you should be prompted to confirm the host key fingerprint the next time you connect via SSH. If you are connecting to the HPCC's gateway (hpcc.msu.edu), check the key against our published host key fingerprints before continuing.

First, open a terminal in VS Code. You can do this from the Command Palette (Ctrl+Shift+P or Cmd+Shift+P) by searching for "Terminal: Create New Terminal" and selecting it.

With the terminal open, you can use the following command to clear the host key for the gateway nodes:

ssh-keygen -R hpcc.msu.edu 

If you received a warning about the host key when connecting to a development node via proxy jump, you may need to replace hpcc.msu.edu with the name of the relevant development node; e.g., dev-amd20. Unfortunately, the warning issued by VS Code will only show the final host name in a proxy jump chain. This means that either the gateway or development node host key could change and VS Code will still tell you that the development node's host key has changed.

Alternatively, you can delete the file ~/.ssh/known_hosts. Every time you connect via SSH (including to sites like GitHub) you will be asked to verify the SSH host key fingerprint.

Once your host key(s) have been removed, you'll be prompted to confirm the host key fingerprint the next time you connect via SSH:

If you are connecting to the HPCC's gateway (hpcc.msu.edu), check the key against our published host key fingerprints before continuing.