Production Cluster Configuration

These packages form the basic functionality of the production cluster.

Scripts & config files are checked into gitlab under the Kubernetes group project listed.

activity	gitlab	script/procedures/config	IP	hostname(s)
K8Dash Dashboard	k8s-admin		10.0.0.191
Ceph Storage Cluster	k8s-admin
Rook Storage (future)	k8s-admin			(StorageClass) rook-ceph-hdd rook-ceph-nvme
Contour Ingress Controller (optional)	k8s-admin		10.0.0.115
rsyslog	k8s/rsyslog		10.0.0.113	rsyslog.williams.localnet
mail	k8s/mail		10.0.0.114	mail.williams.localnet
wordpress (dredwilliams.com)	k8s/dredwilliams			dredwilliams.williams-net.org
mediawiki	mediawiki		10.0.0.116	wiki.williams.localnet wiki.williams-net.org
MariaDB	mariadb		10.0.0.117	database.williams.localnet
Rocket.Chat (not currently deployed)	rocketchat		(contour)	rocket.williams-net.org

Storage

The production cluster depends on the /shared filesystem for its persistent storage as provided by the production Ceph cluster. The Ceph is configured as shown here:

system	function	storage	size
telmar	master	NVMe HDD	1TB 1TB
compute4	node	NVMe HDD HDD	1TB 1TB 1TB
pro5	node	NVMe HDD	1TB 250GB

All systems mount the /shared ceph filesystem following the directions in the installation page. The relevant line for /etc/fstab is:

10.0.0.10:/ /shared ceph name=prodcluster,_netfs 0 0

The client keyring must be copied from the master node (calormen) and placed in the /etc/ceph directory on the client system prior to mounting.

The work filesystem can be mounted via NFS:

10.0.0.75:/work /work nfs4 soft 0 0

Backups

In addition to the normal backups configured in the basic OS installation steps, the databases in the production cluster must be backed up daily using the 'mysqldump' command:

mysqldump -u root -pmenagerie --all-databases -h 10.96.244.162 > /shared/mediawiki-all.dump
mysqldump -u root -pmenagerie bitnami_mediawiki -h 10.96.244.162 > /shared/mediawiki.dump
mysqldump -u root -pmenagerie --all-databases -h database.williams.localnet > /shared/database.dump

These commands should be inserted into the /etc/cron.daily/backup file on one of the cluster nodes (telmar is a good choice). The first does a complete database dump of the MediaWiki database server, the second dumps just the mediawiki database itself, and the third dumps the general purpose database server. Additional dump commands should be inserted for additional significant databases, as parsing individual databases out of a system dump can be tedious.

Dashboard Token

Obtain the token needed to log into the dashboard with this command:

kubectl -n kube-system describe secrets \
   `kubectl -n kube-system get secrets | awk '/clusterrole-aggregation-controller/ {print $1}'` \
   | awk '/token:/ {print $2}'    

The current token for the Production cluster is:

eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbHVzdGVycm9sZS1hZ2dyZWdhdGlvbi1jb250cm9sbGVyLXRva2VuLTdydDQ3Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNsdXN0ZXJyb2xlLWFnZ3JlZ2F0aW9uLWNvbnRyb2xsZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIwYjk1NmU5Yi01MmJiLTQwMWEtYTgwOC03MWI5YWVjNDZjNGQiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06Y2x1c3RlcnJvbGUtYWdncmVnYXRpb24tY29udHJvbGxlciJ9.bOL_ObIZ5vNkTlMd1Cdxsy6AHd_LRH-uf3-6g3YeKVoCtaKkGyR9C7mZlTQrpc6844l4sGMWBWW5HytCK9JTBoHpDADeJZQa0Q5S8cyQMPpNJUukatxzUtHN07FZ6iIl6j_wqLvVJq1dPcu_orD2HGUt7peb0FJ8Ut17opGjR9elLdR0AbZy91EJMoNj5tDCXn0-hdtjbNTu0mGzXfON9Mt3ZIjbXE31uJlji-5KfZjPzhqV0UI7v0R3yoEfPINZlqX7xmqeJt8lI0z-rgRdygLmepRaT6CYpP6IJvAsog06JpQpoU0mZmWKOqEYHS7K_AFGRV5z3vp7QLSPi1PKFA

Kubernetes Node Join Command

kubeadm join 10.0.0.10:6443 --token hqxg8k.bcz5utygyd2sa4yn \
   --discovery-token-ca-cert-hash sha256:ec16325aa0d701961337bc15889e8a90dd1f2d37e08f47d6211d4d7b839b4eb3 \
    --ignore-preflight-errors Swap --node-name=`hostname -s`