SSH Host Key Management

Aside from the SSH authentication keys for users and root which must be distributed to a new system, the one item that is key to providing a unified operating environment is the known_hosts file located in each user's .ssh directory. This must be maintained with the host keys for all current hosts -- both by hostname and by IP address -- in order that automated access between hosts is possible.

Updating 'known_hosts'

Currently, aslan runs a job that uses the master hosts file located at http://config/config/hosts as input to the ssh-keyscan program to retrieve the host keys for all systems and put them in http://config/config/ssh/known_hosts. This script is host_update.sh and is included here for reference:

#!/bin/zsh -f
#
# update the known_hosts file from the master hosts file when needed
BASE=/workspace/nginx/config

HOSTS_DATE=0
if [ -e $BASE/hosts.date ]; then
    HOSTS_DATE=`cat $BASE/hosts.date`
fi

if [[ $HOSTS_DATE != `stat -c '%y' $BASE/hosts` ]] || [[ $1 = '-f' ]]; then
    #
    # scrape the host keys from the known hosts and put them up for access on the config server
    #
    ssh-keyscan -f $BASE/hosts -t rsa,ecdsa | grep -v localhost  > $BASE/ssh/known_hosts      
    cat $BASE/hosts | cut -f 1 | ssh-keyscan -f - -t rsa,ecdsa  >> $BASE/ssh/known_hosts      
    cat $BASE/ssh/pfsense_host_key >> $BASE/ssh/known_hosts
    stat -c '%y' $BASE/hosts > $BASE/hosts.date
    sha1sum $BASE/ssh/known_hosts > $BASE/ssh/known_hosts.sum
fi

The script is run hourly, but the test to see if the hosts file has changed is rather minimal, so resource usage isn't a factor.

Propagating 'known_hosts'

Whenever a new system is added to the environment, its hostname and all its IP addresses must be added to the master hosts file. Each system should (either automatically or on-demand) retrieve the hosts and known_hosts file using commands such as:

curl -s http://config/config/ssh/known_hosts -o $HOME/.ssh/known_hosts
sudo curl -s http://config/config/ssh/known_hosts -o /root/.ssh/known_hosts
sudo curl -s http://config/config/hosts -o /etc/hosts

When a system is rebuilt with new keys, that system will have issues until the new host keys propagate: all systems will see that it has a new host key and SSH will not allow connections to be made. Until the update job runs, individual known_hosts files can be updated, but the automated process will overwrite any changes the next time it runs.

The script below (host_check) compares the known_hosts file used by the root account with the one on the config server using sha1sum so that the files are not excessively updated with an identical copy. If a change is detected, both the known_hosts and hosts files are downloaded and installed. This script should be copied into the cron.hourly directory so that updates occur relatively quickly when changes are made.

#!/bin/zsh -f
#
# check to see if we need to download new hosts and known_hosts files
REMOTE=`curl -s http://config/config/ssh/known_hosts.sum | cut -f 1 -d ' '`
LOCAL=`sha1sum /root/.ssh/known_hosts | cut -f 1 -d ' '`
if [[ $REMOTE != $LOCAL ]] ; then
    curl -s http://config/config/ssh/known_hosts -o /home/ewilliam/.ssh/known_hosts
    curl -s http://config/config/ssh/known_hosts -o /root/.ssh/known_hosts
    curl -s http://config/config/hosts -o /etc/hosts
fi