Development Cluster Configuration: Difference between revisions

From WilliamsNet Wiki
Jump to navigation Jump to search
mNo edit summary
mNo edit summary
Line 1: Line 1:
These packages form the basic functionality of the development cluster.  The packages need to be installed in this order to preserve the IP address assignments.  If the order is changed (or one is left out) you may need to review and modify IP assignments in the DNS server on the firewall.
The Development cluster is deployed using [[K3s - Kubernetes Simplified]].
 
These packages form the basic functionality of the development cluster.  The packages need to preserve the IP address assignments.   


Scripts & config files are checked into gitlab under the Kubernetes group project listed.
Scripts & config files are checked into gitlab under the Kubernetes group project listed.
Line 7: Line 9:
! activity !! gitlab !! script/procedures/config !! IP !! hostname(s)
! activity !! gitlab !! script/procedures/config !! IP !! hostname(s)
|-
|-
| [[BeeGFS Installation]] || install the parallel filesystem components on controller & nodes to support the /workspace filesystem || || ||
| NVIDIA device plugin || || https://github.com/NVIDIA/k8s-device-plugin || ||
|-
|-
| NVIDIA device plugin || || https://github.com/NVIDIA/k8s-device-plugin || ||
| [[Rook Storage for Kubernetes|Rook Storage]] || k8s-admin || || || (StorageClass)<br/>rook-ceph
|-
|-
| K8Dash Dashboard || k8s-admin || || 10.0.0.200 ||
| K8Dash Dashboard || k8s-admin || || 10.0.0.200 ||
|-
|-
| [[Dynamic Provisioning]] || k8s-admin || (k8s-admin wiki) || ||
| GitLab - Helm deployment || [https://gitlab.dev.williams.localnet/admin/projects/k8s/gitlab Kubernetes/gitlab] || kubernetes/gitlab/helm || 10.0.0.203 || gitlab.dev.williams.localnet
|-
| [[Harbor Registry]] || k8s-admin || || 10.0.0.201 || harbor-dev.williams.localnet
|}
|}


=== Storage ===
=== Storage ===
The production cluster depends on the '''/workspace''' filesystem for its persistent storage.  The BeeGFS components are installed as shown here:
The production cluster uses Rook/Ceph for its persistent storage.  The storage components are arranged:
{| class="wikitable"
{| class="wikitable"
|-
|-
! component !! system !! location !! storage !! size
! component !! system !! location !! storage !! size
|-
|-
| Management Server || controller || /beegfs/beegfs-mgmtd || local SSD || ~200G (shared with OS)
| Storage Server || storage1 || /work || local 5x4TB drives || 20TB (ceph)
|-
| Metadata Server || controller || /beegfs/beegfs-meta || local SSD || ~200G (shared with OS)
|-
| Storage Server || controller || /ws_data/beegfs/beegfs-storage || mounted from Equalogic array || 7.9T
|-
| Storage Server || storage1 || /ws_data_2 || local 5x4TB RAID5 array || 15T
|}
|}
Systems that require access to both the development filesystem ('''/workspace''') and the production filesystem ('''/shared''') require a [[BeeGFS Installation#Mounting multiple filesystems on the same client|special client configuration]].
Systems that cannot use BeeGFS can access the development workspace by adding this line to their /etc/fstab:
controller:/workspace /workspace nfs4 soft 0 0
and then issuing these commands:
sudo mkdir /workspace
sudo mount /workspace
The filesystem will be mounted automatically on boot from that point onward.
=== Dashboard Token ===
Obtain the current dashboard token with these commands:
kubectl create serviceaccount k8dash-sa
kubectl create clusterrolebinding k8dash-sa --clusterrole=cluster-admin --serviceaccount=default:k8dash-sa
kubectl describe secrets `kubectl get secrets | awk '/k8dash-sa/ {print $1}'` | awk '/token:/ {print $2}'
The current Development cluster dashboard token is:
    eyJhbGciOiJSUzI1NiIsImtpZCI6Ims0NGhxMmxFZnVSUTBnYVA4VXdta29KeFJVRjRUQkdSUWs3X1NKQ1RLTU0ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Ims4ZGFzaC1zYS10b2tlbi1oZGo5cCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrOGRhc2gtc2EiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIzMjk0MzdhZS1kMmJjLTQwZTAtODY3YS1hZmZmNTE0ZTNhMmYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDprOGRhc2gtc2EifQ.BlsyyRHIeTeJWv3x7JPuxix0xGJqg5YYozxsvPcbTcHV_TXw7T6zE_oeizIuKietyr2_RnXa-523OyhLoZo448NDK0K5XLX5sRqIbGYkDFH4AnawADTkBNPVRbRZvaBr7R8BGurRAWcDLBp0sI6D4PhZSk7O_evrrtmqOTo6KRO7fApg58OC5SpDKoJq8SpFASJNOKJs1KGURP8_tFvstsLkOwJj25uErO6Qt87mNFyWQ2r2Ta-QFtS43kTexp4FeOZAShRfUlmhJO9HWPyl-ES7HrnWpVSd9H8w6AN-dXxF-P_W6xCAF8jV4GqFIYozSrimRikRitxc23mkrYnXaQ
=== Kubernetes Node Join Command ===
<pre>
kubeadm join 10.1.0.10:6443 --token k7ujoy.bsgp5bsdkl3mybhw \
    --discovery-token-ca-cert-hash sha256:76ab40add23981ca5d86396fc81c267b50084b0fbd28fc389a4f53c5d9a2cbf0 \
    --ignore-preflight-errors Swap --node-name=`hostname -s`
</pre>

Revision as of 22:18, 15 August 2021

The Development cluster is deployed using K3s - Kubernetes Simplified.

These packages form the basic functionality of the development cluster. The packages need to preserve the IP address assignments.

Scripts & config files are checked into gitlab under the Kubernetes group project listed.

activity gitlab script/procedures/config IP hostname(s)
NVIDIA device plugin https://github.com/NVIDIA/k8s-device-plugin
Rook Storage k8s-admin (StorageClass)
rook-ceph
K8Dash Dashboard k8s-admin 10.0.0.200
GitLab - Helm deployment Kubernetes/gitlab kubernetes/gitlab/helm 10.0.0.203 gitlab.dev.williams.localnet

Storage

The production cluster uses Rook/Ceph for its persistent storage. The storage components are arranged:

component system location storage size
Storage Server storage1 /work local 5x4TB drives 20TB (ceph)
Retrieved from "http://wiki.williams-net.org/index.php?title=Development_Cluster_Configuration&oldid=499"