Rook Storage for Kubernetes - Revision history

DrEdWilliams: added instruction to install lvm2 on hosts where the disks live

2021-10-10T15:33:45Z

added instruction to install lvm2 on hosts where the disks live

← Older revision		Revision as of 15:33, 10 October 2021
Line 38:		Line 38:
	The reliability of creating a ceph cluster within the kubernetes space has imporved considerably -- and given that Rook is obviously biased AGAINST using external clusters these days (see above), this is really the only option if you want to use Rook to provision storage for Kubernetes. That said ...		The reliability of creating a ceph cluster within the kubernetes space has imporved considerably -- and given that Rook is obviously biased AGAINST using external clusters these days (see above), this is really the only option if you want to use Rook to provision storage for Kubernetes. That said ...

	See the instructions in the [[Ceph_Storage_Cluster#Adding_Disks\|Ceph Adding Disks]] page for how to prepare disks to be included in the cluster ... it isn't easy ...		See the instructions in the [[Ceph_Storage_Cluster#Adding_Disks\|Ceph Adding Disks]] page for how to prepare disks to be included in the cluster ... it isn't easy. One thing that needs to be done on the hosts where the disks reside is to install the '''''lvm2''''' package (it may or may not be automatically installed ... but it needs to be there).

	In the same directory as the operator manifest, there is a '''''cluster.yaml''''' that will create a ceph cluster within the kubernetes cluster, but it has one major problem: it is configured to simply grab all available storage from every node in the kubernetes cluster. Not always what you want ... so you need to copy the file into a local directory and modify the 'storage' section. The storage section for the development cluster is shown here as an example:		In the same directory as the operator manifest, there is a '''''cluster.yaml''''' that will create a ceph cluster within the kubernetes cluster, but it has one major problem: it is configured to simply grab all available storage from every node in the kubernetes cluster. Not always what you want ... so you need to copy the file into a local directory and modify the 'storage' section. The storage section for the development cluster is shown here as an example:

DrEdWilliams: /* Installation Process */

2021-09-26T20:10:50Z

Installation Process

← Older revision		Revision as of 20:10, 26 September 2021
Line 9:		Line 9:

	cd <working directory>		cd <working directory>
	git clone --single-branch --branch release-1.~~6.1~~ https://github.com/rook/rook.git		git clone --single-branch --branch release-1.7 https://github.com/rook/rook.git
	cd rook/cluster/examples/kubernetes/ceph		cd rook/cluster/examples/kubernetes/ceph

DrEdWilliams: /* Using Existing Ceph Cluster */

2021-05-09T01:59:18Z

Using Existing Ceph Cluster

← Older revision		Revision as of 01:59, 9 May 2021
Line 98:		Line 98:
	Before you think that you can just pull rook v1.3 out and use it to connect to an external ceph cluster, it seems to have a rather nasty bug where it (seriously) thinks that version 15.2.11 is older than version 15.2.8 -- making it unusable for newer versions of the Ceph Octopus clusters. Moving to Ceph Pacific may (temporarily) solve that problem, but I was not able to get a Pacific cluster to operate ... so ...		Before you think that you can just pull rook v1.3 out and use it to connect to an external ceph cluster, it seems to have a rather nasty bug where it (seriously) thinks that version 15.2.11 is older than version 15.2.8 -- making it unusable for newer versions of the Ceph Octopus clusters. Moving to Ceph Pacific may (temporarily) solve that problem, but I was not able to get a Pacific cluster to operate ... so ...

	~~I would prefer~~ to ~~have the~~ ceph cluster ~~separate from the kubernetes cluster, but with the current philosophy, I don~~'~~t see that occurring soon unless I dig into~~ the ~~rook source~~ and ~~figure it out~~.		There is another package that will allow Kubernetes to use an external ceph cluster -- it's part of the ceph distribution called [[Ceph-CSI Provisioner for Kubernetes\|Ceph-CSI]]. This is a very much lighter deployment and serves the need very nicely.

	==== External cluster instructions -- DEPRECATED, but retained for historical (hysterical) reasons ====		==== External cluster instructions -- DEPRECATED, but retained for historical (hysterical) reasons in case Rook gets their act straight ====

	Though not absolutely required, it is recommended to use a separate namespace for the external cluster. The default scripts and manifests assume you will do this, so it is easier to just go with the flow ... the namespace in the provided manifests is 'rook-ceph-external'. As with the rook operator, support roles, role bindings, and such need to be created along with the actual separate namespace:		Though not absolutely required, it is recommended to use a separate namespace for the external cluster. The default scripts and manifests assume you will do this, so it is easier to just go with the flow ... the namespace in the provided manifests is 'rook-ceph-external'. As with the rook operator, support roles, role bindings, and such need to be created along with the actual separate namespace:

DrEdWilliams: /* Updating Rook */

2021-05-07T19:20:15Z

Updating Rook

← Older revision		Revision as of 19:20, 7 May 2021
Line 190:		Line 190:

	=== Updating Rook ===		=== Updating Rook ===
	~~Fortunately~~, the process of updating ~~the rook operator~~ is ~~straightforward: just download~~ the ~~new YAML manifests and apply them~~:		Unless otherwise noted due to extenuating requirements, upgrades from one patch release of Rook to another are as simple as updating the common resources and the image of the Rook operator. For example, when Rook v1.6.2 is released, the process of updating from v1.6.0 is as simple as running the following:

	git clone --single-branch --branch v1.5.1 https://github.com/rook/rook.git		First get the latest common resources manifests that contain the latest changes for Rook v1.6.

			git clone --single-branch --depth=1 --branch v1.6.2 https://github.com/rook/rook.git
	cd rook/cluster/examples/kubernetes/ceph		cd rook/cluster/examples/kubernetes/ceph
	kubectl apply -f ~~crds~~.yaml -f ~~common~~.yaml -f operator.~~yaml~~
			If you have deployed the Rook Operator or the Ceph cluster into a different namespace than rook-ceph, see the Update common resources and CRDs section for instructions on how to change the default namespaces in common.yaml.

			Then apply the latest changes from v1.6 and update the Rook Operator image.

			kubectl apply -f common.yaml -f crds.yaml
			kubectl -n rook-ceph set image deploy/rook-ceph-operator rook-ceph-operator=rook/ceph:v1.6.2

			As exemplified above, it is a good practice to update Rook-Ceph common resources from the example manifests before any update. The common resources and CRDs might not be updated with every release, but K8s will only apply updates to the ones that changed.

	Watch the logs from the operator pod to ensure that nothing is amiss, but that should be it.		Watch the logs from the operator pod to ensure that nothing is amiss, but that should be it.

DrEdWilliams: /* Creating an internal Ceph Cluster */

2021-04-30T21:58:12Z

Creating an internal Ceph Cluster

← Older revision		Revision as of 21:58, 30 April 2021
Line 38:		Line 38:
	The reliability of creating a ceph cluster within the kubernetes space has imporved considerably -- and given that Rook is obviously biased AGAINST using external clusters these days (see above), this is really the only option if you want to use Rook to provision storage for Kubernetes. That said ...		The reliability of creating a ceph cluster within the kubernetes space has imporved considerably -- and given that Rook is obviously biased AGAINST using external clusters these days (see above), this is really the only option if you want to use Rook to provision storage for Kubernetes. That said ...

	See the instructions in the [[Ceph ~~Storage Cluster~~]] page for how to prepare disks to be included in the cluster ... it isn't easy ...		See the instructions in the [[Ceph_Storage_Cluster#Adding_Disks\|Ceph Adding Disks]] page for how to prepare disks to be included in the cluster ... it isn't easy ...

	In the same directory as the operator manifest, there is a '''''cluster.yaml''''' that will create a ceph cluster within the kubernetes cluster, but it has one major problem: it is configured to simply grab all available storage from every node in the kubernetes cluster. Not always what you want ... so you need to copy the file into a local directory and modify the 'storage' section. The storage section for the development cluster is shown here as an example:		In the same directory as the operator manifest, there is a '''''cluster.yaml''''' that will create a ceph cluster within the kubernetes cluster, but it has one major problem: it is configured to simply grab all available storage from every node in the kubernetes cluster. Not always what you want ... so you need to copy the file into a local directory and modify the 'storage' section. The storage section for the development cluster is shown here as an example:

DrEdWilliams: /* Creating an internal Ceph Cluster */

2021-04-30T21:55:06Z

Creating an internal Ceph Cluster

← Older revision		Revision as of 21:55, 30 April 2021
Line 85:		Line 85:
	... and then exec into it (or access through lens/k8dash or other means):		... and then exec into it (or access through lens/k8dash or other means):
	kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash		kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash

			If you do not have storage on 3 or more hosts, you need to reset the default replication failureDomain in the cluster itself to keep ceph from complaining about it. Go into the rook toolbox and issue the same commands as for a standalone ceph cluster located [[Ceph_Storage_Cluster#Single_Host_Operation\|here]].


	Now you can skip down to creating storage classes ... and other configuration activities.		Now you can skip down to creating storage classes ... and other configuration activities.

DrEdWilliams: /* Installation Process */

2021-04-30T21:43:27Z

Installation Process

← Older revision		Revision as of 21:43, 30 April 2021
Line 9:		Line 9:

	cd <working directory>		cd <working directory>
	git clone --single-branch --branch release-1.3 https://github.com/rook/rook.git		git clone --single-branch --branch release-1.6.1 https://github.com/rook/rook.git
	cd rook/cluster/examples/kubernetes/ceph		cd rook/cluster/examples/kubernetes/ceph

	'''NOTE''' Rook ~~version~~ 1.5 ~~has~~ problems that make it unusable for interfacing with external Ceph clusters. v1.3 is the last known-good version.		'''NOTE''' Rook versions 1.5 and later have problems that make them unusable for interfacing with external Ceph clusters.
			'''NOTE''' Rook version 1.3 is the last known-good version for using external clusters, but has a bug in comparing versions of ceph clusters.

	The installation process is divided into two parts: Installing the Rook Operator and then either installing a Ceph Cluster in the kubernetes cluster or connecting with an existing Ceph Cluster that has been installed previously.		The installation process is divided into two parts: Installing the Rook Operator and then either installing a Ceph Cluster in the kubernetes cluster or connecting with an existing Ceph Cluster that has been installed previously.
Line 21:		Line 22:
	Creating the Rook Operator simply requires loading two manifests with no customization. First, set up the namespace and all the roles, bindings, and support definitions:		Creating the Rook Operator simply requires loading two manifests with no customization. First, set up the namespace and all the roles, bindings, and support definitions:

	kubectl create -f common.yaml		kubectl create -f crds.yaml -f common.yaml

	Then create the rook operator itself and wait for it to settle into a Running state.		Then create the rook operator itself and wait for it to settle into a Running state.
Line 35:		Line 36:

	=== Creating an internal Ceph Cluster ===		=== Creating an internal Ceph Cluster ===
	The reliability of creating a ceph cluster within the kubernetes space has imporved considerably -- and given that ~~Rool~~ is obviously biased AGAINST using external clusters these days (see ~~below~~), this is really the only option if you want to use Rook to provision storage for Kubernetes. That said ...		The reliability of creating a ceph cluster within the kubernetes space has imporved considerably -- and given that Rook is obviously biased AGAINST using external clusters these days (see above), this is really the only option if you want to use Rook to provision storage for Kubernetes. That said ...

	~~The process of installing~~ the ~~rook operator is straightforward and requires no customization:~~		See the instructions in the [[Ceph Storage Cluster]] page for how to prepare disks to be included in the cluster ... it isn't easy ...
	~~git clone --single-branch --branch v1~~.6.~~1 https://github~~.~~com/rook/rook.git~~
	~~cd rook/cluster/examples/kubernetes/ceph~~
	~~kubectl create -f crds~~.~~yaml -f common~~.~~yaml -f operator~~.~~yaml~~

	In ~~this~~ same directory, there is a '''''cluster.yaml''''' that will create a ceph cluster within the kubernetes cluster, but it has one major problem: it is configured to simply grab all available storage from every node in the kubernetes cluster. Not always what you want ... so you need to copy the file into a local directory and modify the 'storage' section. The storage section for the development cluster is shown here as an example:		In the same directory as the operator manifest, there is a '''''cluster.yaml''''' that will create a ceph cluster within the kubernetes cluster, but it has one major problem: it is configured to simply grab all available storage from every node in the kubernetes cluster. Not always what you want ... so you need to copy the file into a local directory and modify the 'storage' section. The storage section for the development cluster is shown here as an example:

	storage: # cluster level storage configuration and selection		storage: # cluster level storage configuration and selection
Line 77:		Line 75:
	kubectl apply -f cluster.yaml		kubectl apply -f cluster.yaml

	It will take a while, it will create a zillion pods, but you'll end up with a cluster.		It will take a while, it will create a zillion pods, but you'll end up with a cluster ... assuming that it liked your storage. If you have to try again to get storage to connect, all you need to do is slightly modify the '''''cluster.yaml''''' file and re-apply it. That will cause the operator to refresh the cluster and try again to assimilate the storage.

	To get a ceph dashboard, you need to make sure it is enabled in the '''''cluster.yaml''' file (it is by default) and install a service to make it accessible. multiple options are provided in the distribution directory, but the simplest (if you have an [[IP Controller]] installed is to use a loadBalancer:		To get a ceph dashboard, you need to make sure it is enabled in the '''''cluster.yaml''' file (it is by default) and install a service to make it accessible. multiple options are provided in the distribution directory, but the simplest (if you have an [[IP Controller]] installed is to use a loadBalancer:

DrEdWilliams: /* Updating Ceph */

2021-04-30T21:31:50Z

Updating Ceph

← Older revision		Revision as of 21:31, 30 April 2021
Line 200:		Line 200:
	Updating the Ceph installation to the latest version only requires one change to the Rook deployment; update the image version in the 'cluster-external-management.yaml' file to match the new Ceph version and apply.		Updating the Ceph installation to the latest version only requires one change to the Rook deployment; update the image version in the 'cluster-external-management.yaml' file to match the new Ceph version and apply.

	If you have to reinstall or rebuild the ceph deployment, more is required, as the identifying information and credentials for the ceph cluster will have changed.		If you have to reinstall or rebuild the ceph deployment in an external cluster, more is required, as the identifying information and credentials for the ceph cluster will have changed.

	# As above in the installation section, get the necessary information from the ceph cluster and put it into the environment variables in the 'import-xxxx-cluster.sh' script. Before you run the script, however, you must delete a bunch of things that the script will create:		# As above in the installation section, get the necessary information from the ceph cluster and put it into the environment variables in the 'import-xxxx-cluster.sh' script. Before you run the script, however, you must delete a bunch of things that the script will create:

DrEdWilliams at 21:30, 30 April 2021

2021-04-30T21:30:29Z

@@ Line 152: / Line 152: @@
 instructions TBD
-== Testing ==
+==== Default Storage Class (Optional) ====
 Not all manifests specify a storage class -- this can be especially problematic with helm charts that don't expose the opportunity to specify a storage class.  Kubernetes has the concept of a '''default''' storage class that is widely used by the cloud providers to point to their preferred storage solution.  While not required, specifying a Rook StorageClass as default can simplify 'automatic' deployments.  In reality, all that needs to be done is to set a flag on the storage class that you want to be default ... but there's some prep work ...
@@ Line 184: / Line 174: @@
 (excerpted from https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/)
 == Maintenance & Updates ==

DrEdWilliams: /* Storage Classes */

2021-04-30T21:28:55Z

Storage Classes

← Older revision		Revision as of 21:28, 30 April 2021
Line 146:		Line 146:

	At this point, the Storage Provisioner is ready.		At this point, the Storage Provisioner is ready.

			==== Ceph Filesystem Storage Class ====
			There are limitations on using Ceph Filesystems within rook ...

			instructions TBD

	== Testing ==		== Testing ==