Editing SSH Host Key Management

Aside from the SSH authentication keys for users and root which must be distributed to a new system, the one item that is key to providing a unified operating environment is the '''known_hosts''' file located in each user's .ssh directory.  This must be maintained with the host keys for all current hosts -- both by hostname and by IP address -- in order that automated access between hosts is possible.

== Updating 'known_hosts' ==
Currently, aslan runs a job that uses the master '''hosts''' file located at http://config/config/hosts as input to the '''ssh-keyscan''' program to retrieve the host keys for all systems and put them in http://config/config/ssh/known_hosts.  This script is '''host_update.sh''' and is included here for reference:

<pre>#!/bin/zsh -f
#
# update the known_hosts file from the master hosts file when needed
BASE=/workspace/nginx/config

HOSTS_DATE=0
if [ -e $BASE/hosts.date ]; then
    HOSTS_DATE=`cat $BASE/hosts.date`
fi

if [[ $HOSTS_DATE != `stat -c '%y' $BASE/hosts` ]] || [[ $1 = '-f' ]]; then
    #
    # scrape the host keys from the known hosts and put them up for access on the config server
    #
    ssh-keyscan -f $BASE/hosts -t rsa,ecdsa | grep -v localhost  > $BASE/ssh/known_hosts      
    cat $BASE/hosts | cut -f 1 | ssh-keyscan -f - -t rsa,ecdsa  >> $BASE/ssh/known_hosts      
    cat $BASE/ssh/pfsense_host_key >> $BASE/ssh/known_hosts
    stat -c '%y' $BASE/hosts > $BASE/hosts.date
    sha1sum $BASE/ssh/known_hosts > $BASE/ssh/known_hosts.sum
fi</pre>

The script is run hourly, but the test to see if the '''hosts''' file has changed is rather minimal, so resource usage isn't a factor.

== Propagating 'known_hosts' ==

Whenever a new system is added to the environment, its hostname and all its IP addresses must be added to the master '''hosts''' file.  Each system should (either automatically or on-demand) retrieve the '''hosts''' and '''known_hosts''' file using commands such as:

 curl -s http://config/config/ssh/known_hosts -o $HOME/.ssh/known_hosts
 sudo curl -s http://config/config/ssh/known_hosts -o /root/.ssh/known_hosts
 sudo curl -s http://config/config/hosts -o /etc/hosts

When a system is rebuilt with new keys, that system will have issues until the new host keys propagate:  all systems will see that it has a new host key and SSH will not allow connections to be made.  Until the update job runs, individual '''known_hosts''' files can be updated, but the automated process will overwrite any changes the next time it runs.

The script below ('''host_check''') compares the known_hosts file used by the root account with the one on the config server using '''sha1sum''' so that the files are not excessively updated with an identical copy.  If a change is detected, both the '''known_hosts''' and '''hosts''' files are downloaded and installed.  This script should be copied into the '''cron.hourly''' directory so that updates occur relatively quickly when changes are made.

<pre>#!/bin/zsh -f
#
# check to see if we need to download new hosts and known_hosts files
REMOTE=`curl -s http://config/config/ssh/known_hosts.sum | cut -f 1 -d ' '`
LOCAL=`sha1sum /root/.ssh/known_hosts | cut -f 1 -d ' '`
if [[ $REMOTE != $LOCAL ]] ; then
    curl -s http://config/config/ssh/known_hosts -o /home/ewilliam/.ssh/known_hosts
    curl -s http://config/config/ssh/known_hosts -o /root/.ssh/known_hosts
    curl -s http://config/config/hosts -o /etc/hosts
fi</pre>

== Usage ==
With the '''host_update''' script in place in a central host (aslan currently) and the '''host_check''' script installed in each host, any change to the master '''hosts''' file will get propagated within two hours (depending on the synchronization between the hourly cron jobs on each host).  For the two use cases, these procedures must be followed for this propagation to occur:

; New Host
: Simply add a new line to the master '''hosts''' file and either wait for the updates to occur or manually regenerate the '''known_hosts''' file by issuing the command:
: <pre>sudo ./host_update.sh -f</pre>
; Reinstall host with new host keys
: Since the automated update is triggered on the modified date of the master '''hosts''' file, you can '''touch''' the file to update the modification date and the update will be triggered.  You can also force the update as above.  

There is no way to manually trigger the propagation to all the other hosts; but if one particular host is needed, log into the host from ''another'' system and issue the command:
 sudo /etc/cron.hourly/host_check

That will force the update for the host that is needed immediately; the other will catch up when the cron job runs.