Development Cluster Configuration - Revision history

DrEdWilliams: /* Storage */

2021-10-29T11:26:39Z

Storage

← Older revision		Revision as of 11:26, 29 October 2021
Line 10:		Line 10:
	\|-		\|-
	\| NVIDIA device plugin \|\| \|\| https://github.com/NVIDIA/k8s-device-plugin \|\| \|\|		\| NVIDIA device plugin \|\| \|\| https://github.com/NVIDIA/k8s-device-plugin \|\| \|\|
	\|-
	~~\| K8Dash Dashboard \|\| k8s-admin \|\| \|\| 10.0.0.200 \|\|~~
	\|-		\|-
	\| GitLab - Helm deployment \|\| [https://gitlab.dev.williams.localnet/admin/projects/k8s/gitlab Kubernetes/gitlab] \|\| kubernetes/gitlab/helm \|\| 10.0.0.203 \|\| gitlab.dev.williams.localnet		\| GitLab - Helm deployment \|\| [https://gitlab.dev.williams.localnet/admin/projects/k8s/gitlab Kubernetes/gitlab] \|\| kubernetes/gitlab/helm \|\| 10.0.0.203 \|\| gitlab.dev.williams.localnet
Line 20:		Line 18:
	{\| class="wikitable"		{\| class="wikitable"
	\|-		\|-
	! component !! system !! ~~location~~ !! storage !! size		! component !! system !! storageclass !! storage !! size
	\|-		\|-
	\| Storage Server \|\| storage1 \|\| ~~/work~~ \|\| local 5x4TB drives \|\| 20TB (ceph)		\| Storage Server \|\| storage1 \|\| rook-ceph \|\| local 5x4TB drives \|\| 20TB (ceph)
	\|}		\|}

DrEdWilliams at 22:27, 15 August 2021

2021-08-15T22:27:31Z

← Older revision		Revision as of 22:27, 15 August 2021
Line 10:		Line 10:
	\|-		\|-
	\| NVIDIA device plugin \|\| \|\| https://github.com/NVIDIA/k8s-device-plugin \|\| \|\|		\| NVIDIA device plugin \|\| \|\| https://github.com/NVIDIA/k8s-device-plugin \|\| \|\|
	\|-
	~~\| [[Rook Storage for Kubernetes\|Rook Storage]] \|\| k8s-admin \|\| \|\| \|\| (StorageClass)<br/>rook-ceph~~
	\|-		\|-
	\| K8Dash Dashboard \|\| k8s-admin \|\| \|\| 10.0.0.200 \|\|		\| K8Dash Dashboard \|\| k8s-admin \|\| \|\| 10.0.0.200 \|\|

DrEdWilliams at 22:18, 15 August 2021

2021-08-15T22:18:43Z

← Older revision		Revision as of 22:18, 15 August 2021
Line 1:		Line 1:
	These packages form the basic functionality of the development cluster. The packages need ~~to be installed in this order~~ to preserve the IP address assignments. ~~If the order is changed (or one is left out) you may need to review and modify IP assignments in the DNS server on the firewall.~~		The Development cluster is deployed using [[K3s - Kubernetes Simplified]].

			These packages form the basic functionality of the development cluster. The packages need to preserve the IP address assignments.

	Scripts & config files are checked into gitlab under the Kubernetes group project listed.		Scripts & config files are checked into gitlab under the Kubernetes group project listed.
Line 7:		Line 9:
	! activity !! gitlab !! script/procedures/config !! IP !! hostname(s)		! activity !! gitlab !! script/procedures/config !! IP !! hostname(s)
	\|-		\|-
	\| ~~[[BeeGFS Installation]]~~ \|\| ~~install the parallel filesystem components on controller & nodes to support the /workspace filesystem~~ \|\| \|\| \|\|		\| NVIDIA device plugin \|\| \|\| https://github.com/NVIDIA/k8s-device-plugin \|\| \|\|
	\|-		\|-
	\| ~~NVIDIA device plugin~~ \|\| \|~~\| https://github.com/NVIDIA/~~k8s-~~device-plugin~~ \|\| \|\|		\| [[Rook Storage for Kubernetes\|Rook Storage]] \|\| k8s-admin \|\| \|\| \|\| (StorageClass)<br/>rook-ceph
	\|-		\|-
	\| K8Dash Dashboard \|\| k8s-admin \|\| \|\| 10.0.0.200 \|\|		\| K8Dash Dashboard \|\| k8s-admin \|\| \|\| 10.0.0.200 \|\|
	\|-		\|-
	\| ~~[[Dynamic Provisioning]]~~ \|\| ~~k8s-~~admin ~~\|\| (~~k8s~~-admin wiki) \|\| \|\|~~		\| GitLab - Helm deployment \|\| [https://gitlab.dev.williams.localnet/admin/projects/k8s/gitlab Kubernetes/gitlab] \|\| kubernetes/gitlab/helm \|\| 10.0.0.203 \|\| gitlab.dev.williams.localnet
	\|-
	~~\| [[Harbor Registry]~~] \|\| ~~k8s-admin \|\|~~ \|\| 10.0.0.~~201~~ \|\| ~~harbor-~~dev.williams.localnet
	\|}		\|}

	=== Storage ===		=== Storage ===
	The production cluster ~~depends on the '''~~/~~workspace''' filesystem~~ for its persistent storage. The ~~BeeGFS~~ components are ~~installed as shown here~~:		The production cluster uses Rook/Ceph for its persistent storage. The storage components are arranged:
	{\| class="wikitable"		{\| class="wikitable"
	\|-		\|-
	! component !! system !! location !! storage !! size		! component !! system !! location !! storage !! size
	\|-		\|-
	~~\| Management Server \|\| controller \|\| /beegfs/beegfs-mgmtd \|\| local SSD \|\| ~200G (shared with OS)~~		\| Storage Server \|\| storage1 \|\| /work \|\| local 5x4TB drives \|\| 20TB (ceph)
	\|-
	~~\| Metadata Server \|\| controller \|\| /beegfs/beegfs-meta \|\| local SSD \|\| ~200G (shared with OS)~~
	\|-
	~~\| Storage Server \|\| controller \|\| /ws_data/beegfs/beegfs-storage \|\| mounted from Equalogic array \|\| 7.9T~~
	\|-
	\| Storage Server \|\| storage1 \|\| /~~ws_data_2~~ \|\| local 5x4TB ~~RAID5 array~~ \|\| ~~15T~~
	\|}		\|}

	Systems that require access to both the development filesystem ('''/workspace''') and the production filesystem ('''/shared''') require a [[BeeGFS Installation#Mounting multiple filesystems on the same client\|special client configuration]].

	~~Systems that cannot use BeeGFS can access the development workspace by adding this line to their /etc/fstab:~~

	~~controller:/workspace /workspace nfs4 soft 0 0~~

	~~and then issuing these commands:~~

	~~sudo mkdir /workspace~~
	~~sudo mount /workspace~~

	~~The filesystem will be mounted automatically on boot from that point onward.~~

	~~=== Dashboard Token ===~~
	~~Obtain the current dashboard token with these commands:~~
	~~kubectl create serviceaccount k8dash-sa~~
	~~kubectl create clusterrolebinding k8dash-sa --clusterrole=cluster-admin --serviceaccount=default:k8dash-sa~~
	~~kubectl describe secrets `kubectl get secrets \| awk '/k8dash-sa/ {print $1}'` \| awk '/token:/ {print $2}'~~

	~~The current Development cluster dashboard token is:~~
	eyJhbGciOiJSUzI1NiIsImtpZCI6Ims0NGhxMmxFZnVSUTBnYVA4VXdta29KeFJVRjRUQkdSUWs3X1NKQ1RLTU0ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Ims4ZGFzaC1zYS10b2tlbi1oZGo5cCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrOGRhc2gtc2EiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIzMjk0MzdhZS1kMmJjLTQwZTAtODY3YS1hZmZmNTE0ZTNhMmYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDprOGRhc2gtc2EifQ.BlsyyRHIeTeJWv3x7JPuxix0xGJqg5YYozxsvPcbTcHV_TXw7T6zE_oeizIuKietyr2_RnXa-523OyhLoZo448NDK0K5XLX5sRqIbGYkDFH4AnawADTkBNPVRbRZvaBr7R8BGurRAWcDLBp0sI6D4PhZSk7O_evrrtmqOTo6KRO7fApg58OC5SpDKoJq8SpFASJNOKJs1KGURP8_tFvstsLkOwJj25uErO6Qt87mNFyWQ2r2Ta-QFtS43kTexp4FeOZAShRfUlmhJO9HWPyl-ES7HrnWpVSd9H8w6AN-dXxF-P_W6xCAF8jV4GqFIYozSrimRikRitxc23mkrYnXaQ

	~~=== Kubernetes Node Join Command ===~~

	~~<pre>~~
	~~kubeadm join 10.1.0.10:6443 --token k7ujoy.bsgp5bsdkl3mybhw \~~
	~~--discovery-token-ca-cert-hash sha256:76ab40add23981ca5d86396fc81c267b50084b0fbd28fc389a4f53c5d9a2cbf0 \~~
	~~--ignore-preflight-errors Swap --node-name=`hostname -s`~~
	~~</pre>~~

DrEdWilliams at 12:59, 12 January 2020

2020-01-12T12:59:22Z

← Older revision		Revision as of 12:59, 12 January 2020
Line 34:		Line 34:

	Systems that require access to both the development filesystem ('''/workspace''') and the production filesystem ('''/shared''') require a [[BeeGFS Installation#Mounting multiple filesystems on the same client\|special client configuration]].		Systems that require access to both the development filesystem ('''/workspace''') and the production filesystem ('''/shared''') require a [[BeeGFS Installation#Mounting multiple filesystems on the same client\|special client configuration]].

			Systems that cannot use BeeGFS can access the development workspace by adding this line to their /etc/fstab:

			controller:/workspace /workspace nfs4 soft 0 0

			and then issuing these commands:

			sudo mkdir /workspace
			sudo mount /workspace

			The filesystem will be mounted automatically on boot from that point onward.

	=== Dashboard Token ===		=== Dashboard Token ===

DrEdWilliams at 18:22, 26 December 2019

2019-12-26T18:22:06Z

← Older revision		Revision as of 18:22, 26 December 2019
Line 42:		Line 42:

	The current Development cluster dashboard token is:		The current Development cluster dashboard token is:
	~~eyJhbGciOiJSUzI1NiIsImtpZCI6IkxxR0l4c1pkX3puTXNRNlhTWG1JcTdHSk92cEEyRjFiTE5lQWxOUURqVXMifQ~~.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Ims4ZGFzaC1zYS10b2tlbi1wNmg4ZyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrOGRhc2gtc2EiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJhOTFiZTZlZi1hOWE5LTQwMjQtYjM2Zi03MmFjYmM5ZTEzZGUiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDprOGRhc2gtc2EifQ.PTdTVtlmu8cz6jJzR0h0A08iJGIAzaslTc8x_1d9jnQEgKC3HfqKptwAIFRSV4RqvoQfvjKbDzKxAz7CTgepgayZrAXOuumX4GsU3gQMCP5gzovdVAhVR9Jn6Hj8OE2IAKFG6fz3bytabqTqtLFZPlquKcXo9QRq_09Jj9TLxIfMFixRcwxqz5_NfyjHyI55eqidofZf2l88qjVA2_Q2-~~M36S57OPYpFrzylkedS0Z9eRUCh~~-~~o1uJahwdoyo3kqmFMahYoNRxg~~-~~ZLisbsvey9CeHpsdmHhsBGvI~~-~~9miX3WklceFgSJvBOkBXC3E6KgERUrYwzjkaNJ5ZiU6XgZGeQw~~		eyJhbGciOiJSUzI1NiIsImtpZCI6Ims0NGhxMmxFZnVSUTBnYVA4VXdta29KeFJVRjRUQkdSUWs3X1NKQ1RLTU0ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Ims4ZGFzaC1zYS10b2tlbi1oZGo5cCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrOGRhc2gtc2EiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIzMjk0MzdhZS1kMmJjLTQwZTAtODY3YS1hZmZmNTE0ZTNhMmYiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDprOGRhc2gtc2EifQ.BlsyyRHIeTeJWv3x7JPuxix0xGJqg5YYozxsvPcbTcHV_TXw7T6zE_oeizIuKietyr2_RnXa-523OyhLoZo448NDK0K5XLX5sRqIbGYkDFH4AnawADTkBNPVRbRZvaBr7R8BGurRAWcDLBp0sI6D4PhZSk7O_evrrtmqOTo6KRO7fApg58OC5SpDKoJq8SpFASJNOKJs1KGURP8_tFvstsLkOwJj25uErO6Qt87mNFyWQ2r2Ta-QFtS43kTexp4FeOZAShRfUlmhJO9HWPyl-ES7HrnWpVSd9H8w6AN-dXxF-P_W6xCAF8jV4GqFIYozSrimRikRitxc23mkrYnXaQ

	=== Kubernetes Node Join Command ===		=== Kubernetes Node Join Command ===

DrEdWilliams at 17:34, 26 December 2019

2019-12-26T17:34:15Z

← Older revision		Revision as of 17:34, 26 December 2019
Line 47:		Line 47:

	<pre>		<pre>
	kubeadm join 10.1.0.10:6443 --token ~~ly6m64~~.~~yfvsgrizw1f9ip92~~ \		kubeadm join 10.1.0.10:6443 --token k7ujoy.bsgp5bsdkl3mybhw \
	--discovery-token-ca-cert-hash sha256:~~1b627da43c2f2b0df324cd156c9f918c2005ad05ca21541d787447e3b323554b~~ \		--discovery-token-ca-cert-hash sha256:76ab40add23981ca5d86396fc81c267b50084b0fbd28fc389a4f53c5d9a2cbf0 \
	--ignore-preflight-errors Swap --node-name=`hostname -s`		--ignore-preflight-errors Swap --node-name=`hostname -s`
	</pre>		</pre>

DrEdWilliams at 15:04, 26 December 2019

2019-12-26T15:04:22Z

← Older revision		Revision as of 15:04, 26 December 2019
Line 47:		Line 47:

	<pre>		<pre>
	kubeadm join 10.1.0.10:6443 --token ~~hnu981~~.~~t55u3iytaggmxo0d~~ \		kubeadm join 10.1.0.10:6443 --token ly6m64.yfvsgrizw1f9ip92 \
	--discovery-token-ca-cert-hash sha256:~~f7a7c444aff284c40276055d114e61f7f46c82685edd68bb275d9747b4a399b1~~ \		--discovery-token-ca-cert-hash sha256:1b627da43c2f2b0df324cd156c9f918c2005ad05ca21541d787447e3b323554b \
	--ignore-preflight-errors Swap --node-name=`hostname -s`		--ignore-preflight-errors Swap --node-name=`hostname -s`
	</pre>		</pre>

DrEdWilliams at 14:27, 26 December 2019

2019-12-26T14:27:59Z

← Older revision		Revision as of 14:27, 26 December 2019
Line 47:		Line 47:

	<pre>		<pre>
	kubeadm join 10.1.0.10:6443 --token ~~t01ye7~~.~~swdmsrmwj0miox8b~~ \		kubeadm join 10.1.0.10:6443 --token hnu981.t55u3iytaggmxo0d \
	--discovery-token-ca-cert-hash sha256:~~3f49c39e9a79636e549add561311062b806bb5e977f4ac60300b95294b6ad2d1~~ \		--discovery-token-ca-cert-hash sha256:f7a7c444aff284c40276055d114e61f7f46c82685edd68bb275d9747b4a399b1 \
	--ignore-preflight-errors Swap --node-name=`hostname -s`		--ignore-preflight-errors Swap --node-name=`hostname -s`
	</pre>		</pre>

DrEdWilliams at 13:41, 26 December 2019

2019-12-26T13:41:53Z

← Older revision		Revision as of 13:41, 26 December 2019
Line 47:		Line 47:

	<pre>		<pre>
	kubeadm join 10.1.0.10:6443 --token ~~zf1p56~~.~~g5s8xrcg0icwez4s~~ \		kubeadm join 10.1.0.10:6443 --token t01ye7.swdmsrmwj0miox8b \
	--discovery-token-ca-cert-hash sha256:~~46c325c7de98b7d9086cc48a88aa27a8eb82fadc85b581b7155ea02afd67e82c~~ \		--discovery-token-ca-cert-hash sha256:3f49c39e9a79636e549add561311062b806bb5e977f4ac60300b95294b6ad2d1 \
	--ignore-preflight-errors Swap --node-name=`hostname -s`		--ignore-preflight-errors Swap --node-name=`hostname -s`
	</pre>		</pre>

DrEdWilliams at 13:26, 26 December 2019

2019-12-26T13:26:35Z

← Older revision		Revision as of 13:26, 26 December 2019
Line 47:		Line 47:

	<pre>		<pre>
	kubeadm join 10.1.0.10:6443 --token ~~4ohce0~~.~~iqfit9citzgsdlci~~ \		kubeadm join 10.1.0.10:6443 --token zf1p56.g5s8xrcg0icwez4s \
	--discovery-token-ca-cert-hash sha256:~~b42373b7f52494e0071268c361b0dfa5bfcf78d12377d82e3504785aa9547123~~ \		--discovery-token-ca-cert-hash sha256:46c325c7de98b7d9086cc48a88aa27a8eb82fadc85b581b7155ea02afd67e82c \
	--ignore-preflight-errors Swap --node-name=`hostname -s`		--ignore-preflight-errors Swap --node-name=`hostname -s`
	</pre>		</pre>