Production Cluster Configuration: Difference between revisions
DrEdWilliams (talk | contribs) m (updated /shared workspace configuration) |
DrEdWilliams (talk | contribs) mNo edit summary |
||
| Line 41: | Line 41: | ||
|} | |} | ||
Systems that require access to both the development filesystem ('''/workspace''') and the production filesystem ('''/shared''') require a [[BeeGFS Installation#Mounting multiple filesystems on the same client|special client configuration]]. | Systems that require access to both the development filesystem ('''/workspace''') and the production filesystem ('''/shared''') require a [[BeeGFS Installation#Mounting multiple filesystems on the same client|special client configuration]]. | ||
Systems that cannot use BeeGFS can access the production shared filesystem by adding this line to their /etc/fstab: | |||
telmar:/shared /shared nfs4 soft 0 0 | |||
and then issuing these commands: | |||
sudo mkdir /shared | |||
sudo mount /shared | |||
The filesystem will be mounted automatically on boot from that point onward. | |||
=== Backups === | === Backups === | ||
Revision as of 12:57, 12 January 2020
These packages form the basic functionality of the production cluster.
Scripts & config files are checked into gitlab under the Kubernetes group project listed.
| activity | gitlab | script/procedures/config | IP | hostname(s) |
|---|---|---|---|---|
| BeeGFS Installation | install the parallel filesystem components on controller & nodes to support the /shared filesystem | |||
| NGINX-ingress | k8s-admin | 10.0.0.111 | ||
| GitLab | gitlab | 10.0.0.112 | gitlab.williams.localnet gitlab.williams-net.org | |
| gitlab registry secrets | gitlab-registry-kube-system.yaml gitlab-registry-secret.yaml |
|||
| Quay Container Registry | k8s/quay | quay.williams.localnet (CNAME to lamppost.williams.localnet) | ||
| rsyslog | k8s/rsyslog | 10.0.0.113 | rsyslog.williams.localnet | |
| k8s/mail | 10.0.0.114 | mail.williams.localnet | ||
| wordpress (dredwilliams.com) | k8s/dredwilliams | dredwilliams.williams-net.org | ||
| mediawiki | k8s/mediawiki | 10.0.0.116 | wiki.williams.localnet wiki.williams-net.org | |
| MariaDB | k8s/mariadb | 10.0.0.117 | database.williams.localnet |
Storage
The production cluster depends on the /shared filesystem for its persistent storage. The BeeGFS components are installed as shown here:
| component | system | location | storage | size |
|---|---|---|---|---|
| Management Server | telmar | /data/beegfs-mgmtd | local NVMe | ~1TB (shared) |
| Metadata Server | telmar | /data/beegfs-meta | local NVMe | ~1TB (shared) |
| Storage Server | telmar | /data/beegfs-data | local NVMe | ~1TB (shared) |
Systems that require access to both the development filesystem (/workspace) and the production filesystem (/shared) require a special client configuration.
Systems that cannot use BeeGFS can access the production shared filesystem by adding this line to their /etc/fstab:
telmar:/shared /shared nfs4 soft 0 0
and then issuing these commands:
sudo mkdir /shared sudo mount /shared
The filesystem will be mounted automatically on boot from that point onward.
Backups
In addition to the normal backups configured in the basic OS installation steps, the databases in the production cluster must be backed up daily using the 'mysqldump' command:
mysqldump -u root -pmenagerie --all-databases -h 10.96.244.162 > /shared/mediawiki-all.dump mysqldump -u root -pmenagerie bitnami_mediawiki -h 10.96.244.162 > /shared/mediawiki.dump mysqldump -u root -pmenagerie --all-databases -h database.williams.localnet > /shared/database.dump
These commands should be inserted into the /etc/cron.daily/backup file on the BeeGFS Master (telmar). The first does a complete database dump of the MediaWiki database server, the second dumps just the mediawiki database itself, and the third dumps the general purpose database server. Additional dump commands should be inserted for additional significant databases, as parsing individual databases out of a system dump can be tedious.
Dashboard Token
Obtain the token needed to log into the dashboard with this command:
kubectl -n kube-system describe secrets \
`kubectl -n kube-system get secrets | awk '/clusterrole-aggregation-controller/ {print $1}'` \
| awk '/token:/ {print $2}'
The current token for the Production cluster is:
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbHVzdGVycm9sZS1hZ2dyZWdhdGlvbi1jb250cm9sbGVyLXRva2VuLTdydDQ3Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNsdXN0ZXJyb2xlLWFnZ3JlZ2F0aW9uLWNvbnRyb2xsZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIwYjk1NmU5Yi01MmJiLTQwMWEtYTgwOC03MWI5YWVjNDZjNGQiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06Y2x1c3RlcnJvbGUtYWdncmVnYXRpb24tY29udHJvbGxlciJ9.bOL_ObIZ5vNkTlMd1Cdxsy6AHd_LRH-uf3-6g3YeKVoCtaKkGyR9C7mZlTQrpc6844l4sGMWBWW5HytCK9JTBoHpDADeJZQa0Q5S8cyQMPpNJUukatxzUtHN07FZ6iIl6j_wqLvVJq1dPcu_orD2HGUt7peb0FJ8Ut17opGjR9elLdR0AbZy91EJMoNj5tDCXn0-hdtjbNTu0mGzXfON9Mt3ZIjbXE31uJlji-5KfZjPzhqV0UI7v0R3yoEfPINZlqX7xmqeJt8lI0z-rgRdygLmepRaT6CYpP6IJvAsog06JpQpoU0mZmWKOqEYHS7K_AFGRV5z3vp7QLSPi1PKFA
Kubernetes Node Join Command
kubeadm join 10.0.0.10:6443 --token hqxg8k.bcz5utygyd2sa4yn \
--discovery-token-ca-cert-hash sha256:ec16325aa0d701961337bc15889e8a90dd1f2d37e08f47d6211d4d7b839b4eb3 \
--ignore-preflight-errors Swap --node-name=`hostname -s`