SSH Host Key Management

Aside from the SSH authentication keys for users and root which must be distributed to a new system, the one item that is key to providing a unified operating environment is the known_hosts file located in each user's .ssh directory. This must be maintained with the host keys for all current hosts -- both by hostname and by IP address -- in order that automated access between hosts is possible.

Updating 'known_hosts'

Currently, aslan runs a job that uses the master hosts file located at http://config/config/hosts as input to the ssh-keyscan program to retrieve the host keys for all systems and put them in http://config/config/ssh/known_hosts. This script is host_update.sh and is included here for reference:

#!/bin/zsh -f
#
# update the known_hosts file from the master hosts file when needed
BASE=/workspace/nginx/config

HOSTS_DATE=0
if [ -e $BASE/hosts.date ]; then
    HOSTS_DATE=`cat $BASE/hosts.date`
fi

if [[ $HOSTS_DATE != `stat -c '%y' $BASE/hosts` ]] || [[ $1 = '-f' ]]; then
    #
    # scrape the host keys from the known hosts and put them up for access on the config server
    #
    ssh-keyscan -f $BASE/hosts -t rsa,ecdsa | grep -v localhost  > $BASE/ssh/known_hosts      
    cat $BASE/hosts | cut -f 1 | ssh-keyscan -f - -t rsa,ecdsa  >> $BASE/ssh/known_hosts      
    cat $BASE/ssh/pfsense_host_key >> $BASE/ssh/known_hosts
    stat -c '%y' $BASE/hosts > $BASE/hosts.date
    sha1sum $BASE/ssh/known_hosts > $BASE/ssh/known_hosts.sum
fi

The script is run hourly, but the test to see if the hosts file has changed is rather minimal, so resource usage isn't a factor.

Propagating 'known_hosts'

Whenever a new system is added to the environment, its hostname and all its IP addresses must be added to the master hosts file. Each system should (either automatically or on-demand) retrieve the hosts and known_hosts file using commands such as:

curl -s http://config/config/ssh/known_hosts -o $HOME/.ssh/known_hosts
sudo curl -s http://config/config/ssh/known_hosts -o /root/.ssh/known_hosts
sudo curl -s http://config/config/hosts -o /etc/hosts

When a system is rebuilt with new keys, that system will have issues until the new host keys propagate: all systems will see that it has a new host key and SSH will not allow connections to be made. Until the update job runs, individual known_hosts files can be updated, but the automated process will overwrite any changes the next time it runs.

The script below (host_check) compares the known_hosts file used by the root account with the one on the config server using sha1sum so that the files are not excessively updated with an identical copy. If a change is detected, both the known_hosts and hosts files are downloaded and installed. This script should be copied into the cron.hourly directory so that updates occur relatively quickly when changes are made.

#!/bin/zsh -f
#
# check to see if we need to download new hosts and known_hosts files
REMOTE=`curl -s http://config/config/ssh/known_hosts.sum | cut -f 1 -d ' '`
LOCAL=`sha1sum /root/.ssh/known_hosts | cut -f 1 -d ' '`
if [[ $REMOTE != $LOCAL ]] ; then
    curl -s http://config/config/ssh/known_hosts -o /home/ewilliam/.ssh/known_hosts
    curl -s http://config/config/ssh/known_hosts -o /root/.ssh/known_hosts
    curl -s http://config/config/hosts -o /etc/hosts
fi

Usage

With the host_update script in place in a central host (aslan currently) and the host_check script installed in each host, any change to the master hosts file will get propagated within two hours (depending on the synchronization between the hourly cron jobs on each host). For the two use cases, these procedures must be followed for this propagation to occur:

New Host

Simply add a new line to the master hosts file and either wait for the updates to occur or manually regenerate the known_hosts file by issuing the command:

sudo ./host_update.sh -f

Reinstall host with new host keys

Since the automated update is triggered on the modified date of the master hosts file, you can touch the file to update the modification date and the update will be triggered. You can also force the update as above.

There is no way to manually trigger the propagation to all the other hosts; but if one particular host is needed, log into the host from another system and issue the command:

sudo /etc/cron.hourly/host_check

That will force the update for the host that is needed immediately; the other will catch up when the cron job runs.