Production Cluster Configuration: Difference between revisions

From WilliamsNet Wiki
Jump to navigation Jump to search
mNo edit summary
mNo edit summary
 
(15 intermediate revisions by the same user not shown)
Line 7: Line 7:
! activity !! gitlab !! script/procedures/config !! IP !! hostname(s)
! activity !! gitlab !! script/procedures/config !! IP !! hostname(s)
|-
|-
| [[Ceph Storage Cluster]] || install the parallel filesystem components all nodes to support the /shared filesystem || || ||
| [[Ceph Storage Cluster]] || k8s-admin || || ||
|-
|-
| [[Contour Ingress Controller]] || k8s-admin || || 10.0.0.115 ||
| [[Rook Storage for Kubernetes|Rook Storage]] || k8s-admin || || || (StorageClass)<br/>rook-ceph-hdd<br/>rook-ceph-nvme
|-
| [[GitLab]] || gitlab || || 10.0.0.112 || gitlab.williams.localnet <br />gitlab.williams-net.org
|-
|-
| gitlab registry secrets || || gitlab-registry-kube-system.yaml <br /> gitlab-registry-secret.yaml || ||
| gitlab registry secrets || || gitlab-registry-kube-system.yaml <br /> gitlab-registry-secret.yaml || ||
|-
| [[Quay Container Registry]] || k8s/quay || ||  || quay.williams.localnet <br/> (CNAME to lamppost.williams.localnet)
|-
| rsyslog|| k8s/rsyslog || || 10.0.0.113 || rsyslog.williams.localnet
|-
| mail || k8s/mail || || 10.0.0.114 || mail.williams.localnet
|-
|-
| wordpress (dredwilliams.com) || k8s/dredwilliams || || || dredwilliams.williams-net.org
| wordpress (dredwilliams.com) || k8s/dredwilliams || || || dredwilliams.williams-net.org
|-
|-
| mediawiki || k8s/mediawiki || || 10.0.0.116 || wiki.williams.localnet <br />wiki.williams-net.org
| mediawiki || mediawiki || || 10.0.0.116 || wiki.williams.localnet <br />wiki.williams-net.org
|-
|-
| [[MariaDB]] || k8s/mariadb || || 10.0.0.117 || database.williams.localnet
| [[MariaDB]] || mariadb || || 10.0.0.117 || database.williams.localnet
|}
|}


=== Storage ===
=== Storage ===
The production cluster depends on the '''/shared''' filesystem for its persistent storage.  The BeeGFS components are installed as shown here:
The production cluster depends on the '''/shared''' filesystem for its persistent storage as provided by the production Ceph cluster.  The Ceph is configured as shown here:
{| class="wikitable"
{| class="wikitable"
|-
|-
! component !! system !! location !! storage !! size
! system !! function !! storage !! size
|-
|-
| Management Server || telmar || /data/beegfs-mgmtd || local NVMe || ~1TB (shared)
| caspian || master || NVMe<br/>HDD || 1TB<br/>1TB
|-
|-
| Metadata Server || telmar || /data/beegfs-meta || local NVMe || ~1TB (shared)
| uvilas || node || NVMe<br/>HDD<br/>HDD || 1TB<br/>1TB<br/>1TB
|-
|-
| Storage Server || telmar || /data/beegfs-data || local NVMe || ~1TB (shared)
| belisar || node || NVMe<br/>HDD || 1TB<br/>250GB
|}
|}


Systems that require access to both the development filesystem ('''/workspace''') and the production filesystem ('''/shared''') require a [[BeeGFS Installation#Mounting multiple filesystems on the same client|special client configuration]]. 
The work filesystem can be mounted via NFS:


Systems that cannot use BeeGFS can access the production shared filesystem by adding this line to their /etc/fstab:
  10.0.0.75:/work /work nfs4 soft 0 0
 
  telmar:/shared /shared nfs4 soft 0 0
 
and then issuing these commands:
 
sudo mkdir /shared
sudo mount /shared
 
The filesystem will be mounted automatically on boot from that point onward.


=== Backups ===
=== Backups ===
Line 58: Line 41:
In addition to the normal backups configured in the basic OS installation steps, the databases in the production cluster must be backed up daily using the 'mysqldump' command:
In addition to the normal backups configured in the basic OS installation steps, the databases in the production cluster must be backed up daily using the 'mysqldump' command:


  mysqldump -u root -pmenagerie --all-databases -h 10.96.244.162 > /shared/mediawiki-all.dump
  TBD
mysqldump -u root -pmenagerie bitnami_mediawiki -h 10.96.244.162 > /shared/mediawiki.dump
mysqldump -u root -pmenagerie --all-databases -h database.williams.localnet > /shared/database.dump
 
These commands should be inserted into the /etc/cron.daily/backup file on the BeeGFS Master (telmar).  The first does a complete database dump of the MediaWiki database server, the second dumps just the mediawiki database itself, and the third dumps the general purpose database server.  Additional dump commands should be inserted for additional significant databases, as parsing individual databases out of a system dump can be tedious.
 
=== Dashboard Token ===
Obtain the token needed to log into the dashboard with this command:
kubectl -n kube-system describe secrets \
    `kubectl -n kube-system get secrets | awk '/clusterrole-aggregation-controller/ {print $1}'` \
    | awk '/token:/ {print $2}'   
The current token for the Production cluster is:
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjbHVzdGVycm9sZS1hZ2dyZWdhdGlvbi1jb250cm9sbGVyLXRva2VuLTdydDQ3Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImNsdXN0ZXJyb2xlLWFnZ3JlZ2F0aW9uLWNvbnRyb2xsZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIwYjk1NmU5Yi01MmJiLTQwMWEtYTgwOC03MWI5YWVjNDZjNGQiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06Y2x1c3RlcnJvbGUtYWdncmVnYXRpb24tY29udHJvbGxlciJ9.bOL_ObIZ5vNkTlMd1Cdxsy6AHd_LRH-uf3-6g3YeKVoCtaKkGyR9C7mZlTQrpc6844l4sGMWBWW5HytCK9JTBoHpDADeJZQa0Q5S8cyQMPpNJUukatxzUtHN07FZ6iIl6j_wqLvVJq1dPcu_orD2HGUt7peb0FJ8Ut17opGjR9elLdR0AbZy91EJMoNj5tDCXn0-hdtjbNTu0mGzXfON9Mt3ZIjbXE31uJlji-5KfZjPzhqV0UI7v0R3yoEfPINZlqX7xmqeJt8lI0z-rgRdygLmepRaT6CYpP6IJvAsog06JpQpoU0mZmWKOqEYHS7K_AFGRV5z3vp7QLSPi1PKFA
 


=== Kubernetes Node Join Command ===
These commands should be inserted into the /etc/cron.daily/backup file on one of the cluster nodes (telmar is a good choice). The first does a complete database dump of the MediaWiki database server, the second dumps just the mediawiki database itself, and the third dumps the general purpose database server. Additional dump commands should be inserted for additional significant databases, as parsing individual databases out of a system dump can be tedious.
kubeadm join 10.0.0.10:6443 --token hqxg8k.bcz5utygyd2sa4yn \
    --discovery-token-ca-cert-hash sha256:ec16325aa0d701961337bc15889e8a90dd1f2d37e08f47d6211d4d7b839b4eb3 \
    --ignore-preflight-errors Swap --node-name=`hostname -s`

Latest revision as of 23:09, 14 September 2024

These packages form the basic functionality of the production cluster.

Scripts & config files are checked into gitlab under the Kubernetes group project listed.

activity gitlab script/procedures/config IP hostname(s)
Ceph Storage Cluster k8s-admin
Rook Storage k8s-admin (StorageClass)
rook-ceph-hdd
rook-ceph-nvme
gitlab registry secrets gitlab-registry-kube-system.yaml
gitlab-registry-secret.yaml
wordpress (dredwilliams.com) k8s/dredwilliams dredwilliams.williams-net.org
mediawiki mediawiki 10.0.0.116 wiki.williams.localnet
wiki.williams-net.org
MariaDB mariadb 10.0.0.117 database.williams.localnet

Storage[edit]

The production cluster depends on the /shared filesystem for its persistent storage as provided by the production Ceph cluster. The Ceph is configured as shown here:

system function storage size
caspian master NVMe
HDD		 1TB
1TB
uvilas node NVMe
HDD
HDD		 1TB
1TB
1TB
belisar node NVMe
HDD		 1TB
250GB

The work filesystem can be mounted via NFS: 

10.0.0.75:/work /work nfs4 soft 0 0

Backups[edit]

In addition to the normal backups configured in the basic OS installation steps, the databases in the production cluster must be backed up daily using the 'mysqldump' command: 

TBD

These commands should be inserted into the /etc/cron.daily/backup file on one of the cluster nodes (telmar is a good choice). The first does a complete database dump of the MediaWiki database server, the second dumps just the mediawiki database itself, and the third dumps the general purpose database server. Additional dump commands should be inserted for additional significant databases, as parsing individual databases out of a system dump can be tedious.

Retrieved from "http://wiki.williams-net.org/index.php?title=Production_Cluster_Configuration&oldid=703"