Editing Rook Storage for Kubernetes (section)

== Installation Process ==
The scripts and manifests required to install Rook and Ceph (if needed) are located in the Rook repository that should be cloned for local access:

 cd <working directory>
 git clone --single-branch --branch release-1.7 https://github.com/rook/rook.git
 cd rook/cluster/examples/kubernetes/ceph

'''NOTE''' Rook versions 1.5 and later have problems that make them unusable for interfacing with external Ceph clusters.  
'''NOTE''' Rook version 1.3 is the last known-good version for using external clusters, but has a bug in comparing versions of ceph clusters.

The installation process is divided into two parts:  Installing the Rook Operator and then either installing a Ceph Cluster in the kubernetes cluster or connecting with an existing Ceph Cluster that has been installed previously.

Note that any scripts and manifests that are custom and/or not part of the rook distribution are included in the 'k8s-admin' repository in Gitlab.

=== Rook Operator ===
Creating the Rook Operator simply requires loading two manifests with no customization.  First, set up the namespace and all the roles, bindings, and support definitions:

 kubectl create -f crds.yaml -f common.yaml

Then create the rook operator itself and wait for it to settle into a Running state.

 kubectl create -f operator.yaml

The Rook Operator manages the entire activity of the storage enterprise, so tailing the log in a separate window will be useful should any problems arise:

 kubectl get pods -n rook-ceph
 kubectl logs -f -n rook-ceph <pod name>

The beauty of the operator concept in kubernetes is that it is capable of accomplishing practically anything that can be done from the command line -- all in response to what it sees in the cluster configuration.  In this case, the operator looks for the Custom Resource Definition (CRD) that defines a Ceph Cluster -- with the parameters set to define what that ceph cluster looks like.  If it sees that a new cluster needs to be created, it will do all the steps needed to create and provision the ceph cluster as specified.  If interfacing with an external ceph cluster is required, it takes the provided credentials and identifiers and connects to that cluster to get the storage service.

=== Creating an internal Ceph Cluster ===
The reliability of creating a ceph cluster within the kubernetes space has imporved considerably -- and given that Rook is obviously biased AGAINST using external clusters these days (see above), this is really the only option if you want to use Rook to provision storage for Kubernetes.  That said ...

See the instructions in the [[Ceph_Storage_Cluster#Adding_Disks|Ceph Adding Disks]] page for how to prepare disks to be included in the cluster ... it isn't easy.  One thing that needs to be done on the hosts where the disks reside is to install the '''''lvm2''''' package (it may or may not be automatically installed ... but it needs to be there).

In the same directory as the operator manifest, there is a '''''cluster.yaml''''' that will create a ceph cluster within the kubernetes cluster, but it has one major problem:  it is configured to simply grab all available storage from every node in the kubernetes cluster.  Not always what you want ... so you need to copy the file into a local directory and modify the 'storage' section.  The storage section for the development cluster is shown here as an example:

  storage: # cluster level storage configuration and selection
    useAllNodes: false
    useAllDevices: false
    #deviceFilter:
    config:
      # crushRoot: "custom-root" # specify a non-default root label for the CRUSH map
      # metadataDevice: "md0" # specify a non-rotational storage so ceph-volume will use it as block db device of bluestore.
      # databaseSizeMB: "1024" # uncomment if the disks are smaller than 100 GB
      # journalSizeMB: "1024"  # uncomment if the disks are 20 GB or smaller
      # osdsPerDevice: "1" # this value can be overridden at the node or device level
      # encryptedDevice: "true" # the default value for this option is "false"
    # Individual nodes and their config can be specified as well, but 'useAllNodes' above must be set to false. Then, only the named
    # nodes below will be used as storage resources.  Each node's 'name' field should match their 'kubernetes.io/hostname' label.
    nodes:
    - name: "storage1"
      devices: # specific devices to use for storage can be specified for each node
      - name: "sdc"
      - name: "sdd"
      - name: "sde"
      - name: "sdf"
      - name: "sdg"
      #- name: "nvme01" # multiple osds can be created on high performance devices
      #  config:
      #    osdsPerDevice: "5"
      #- name: "/dev/disk/by-id/ata-ST4000DM004-XXXX" # devices can be specified using full udev paths
      #config: # configuration can be specified at the node level which overrides the cluster level config
    - name: "controller"
      devices:
      - name: "sdb"

After editing a local copy of the '''''cluster.yaml''''' file, apply it like normal ...
 kubectl apply -f cluster.yaml

It will take a while, it will create a zillion pods, but you'll end up with a cluster ... assuming that it liked your storage.  If you have to try again to get storage to connect, all you need to do is slightly modify the '''''cluster.yaml''''' file and re-apply it.  That will cause the operator to refresh the cluster and try again to assimilate the storage.  

To get a ceph dashboard, you need to make sure it is enabled in the '''''cluster.yaml''' file (it is by default) and install a service to make it accessible.  multiple options are provided in the distribution directory, but the simplest (if you have an [[IP Controller]] installed is to use a loadBalancer:
 kubectl create -f dashboard-loadbalancer.yaml

Since everything is contained in the kubernetes cluster, there is no external interface to the cluster for control/management ... so they provide the 'rook-toolbox' deployment that will allow you to exec into the resulting pod to get the familiar 'ceph' and other related utilities.  This can stay alive as long as you want (forever), and comes in a 'rook-toolbox-job' variant to allow automation of activities.  Installing the toolbox is as simple as:
 kubectl create -f toolbox.yaml

... and then exec into it (or access through lens/k8dash or other means):
 kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash

If you do not have storage on 3 or more hosts, you need to reset the default replication failureDomain in the cluster itself to keep ceph from complaining about it.  Go into the rook toolbox and issue the same commands as for a standalone ceph cluster located [[Ceph_Storage_Cluster#Single_Host_Operation|here]].


Now you can skip down to creating storage classes ... and other configuration activities.

=== Using Existing Ceph Cluster ===
One option provided by the Rook Operator is to interface with an already existing Ceph Cluster.  Instructions for deploying a standalone Ceph cluster are included on a [[Ceph Storage Cluster|separate page]]; the operation of that cluster should be completely validated before attempting to connect it to Rook.

It should be noted that while earlier versions of Rook (1.3 and before) can successfully interface with external clusters, that capability seems to be lost in newer versions (1.5 and later).  The configuration options are there, but from the log messages it is clear that the instructions and the reality of the rook operator are NOT in sync, and even forcing some configuration options did not allow a successful connection with an external cluster.

Before you think that you can just pull rook v1.3 out and use it to connect to an external ceph cluster, it seems to have a rather nasty bug where it (seriously) thinks that version 15.2.11 is older than version 15.2.8 -- making it unusable for newer versions of the Ceph Octopus clusters.  Moving to Ceph Pacific may (temporarily) solve that problem, but I was not able to get a Pacific cluster to operate ... so ...

There is another package that will allow Kubernetes to use an external ceph cluster -- it's part of the ceph distribution called [[Ceph-CSI Provisioner for Kubernetes|Ceph-CSI]].  This is a very much lighter deployment and serves the need very nicely.

==== External cluster instructions -- DEPRECATED, but retained for historical (hysterical) reasons in case Rook gets their act straight ====

Though not absolutely required, it is recommended to use a separate namespace for the external cluster.  The default scripts and manifests assume you will do this, so it is easier to just go with the flow ... the namespace in the provided manifests is 'rook-ceph-external'.  As with the rook operator, support roles, role bindings, and such need to be created along with the actual separate namespace:

 kubectl create -f common-external.yaml

The Rook Cluster definition needs authentication and identification data about the external cluster; this is loaded into kubernetes secrets with standard names so that the operator and storage provisioner can access the external ceph cluster.  The data can be obtained from the ceph master node:
* Cluster FSID -- run this command and copy the results:
 ceph fsid
* Cluster Monitor address -- located in the /etc/ceph/ceph.conf file
* Cluster admin secret -- located in the /etc/ceph/ceph.client.admin.keyring

For convenience, the set of commends has been copied to a script in the k8s-admin repository; multiple copies exist for the 'prod', 'dev' and 'test' clusters.  

Put the information into environment variables at the top of the script as shown below, then run the script to create the secrets:
 export NAMESPACE=rook-ceph-external
 export ROOK_EXTERNAL_FSID=dc852252-bd6b-11ea-b7f2-503eaa02062c
 export ROOK_EXTERNAL_CEPH_MON_DATA=storage1=10.1.0.9:6789
 export ROOK_EXTERNAL_ADMIN_SECRET=AQAclf9e0ptoMBAAracpRwXomJ6LgiO6L8wqfw==
 bash ./import-external-cluster.sh

Note that the above script adds too many secrets -- the operator tries to create them again when building the cluster interface -- and errors out since they can't be changed.  We need to either edit the script to not create the excess secrets or delete the ones that aren't needed.  For now, we will delete them.  First find all the ones that are present in the new namespace:
 kubectl get secret -n rook-ceph-external

These are the ones that need to be deleted (at least for now):
 kubectl -n rook-ceph-external delete secret \
    rook-csi-cephfs-node rook-csi-cephfs-provisioner rook-csi-rbd-node rook-csi-rbd-provisioner

Watch the operator log as you create the cluster below to see if any additional secrets need to be deleted.  Now we can create the cluster definition that the operator will use to create our interface:

 kubectl create -f cluster-external-management.yaml

=== Storage Classes ===
Rook will allow us to create StorageClasses for the different APIs provided by Ceph.  The most common one is for RBD (block storage), but you can also use Ceph Filesystems.

==== RBD Storage Class ====
Next we create the StorageClass that kubernetes will use to request RBDs (block images) from the ceph cluster.  The starting point is the manifest '''''csi/rbd/storageclass.yaml''''' in the rook distribution.  Copy it to a local directory and customize it as described below.

The manifest creates two resources:  A CephPool resource that will be used to create a new pool (or use an existing pool) in the ceph cluster, and a StorageClass entry that will allow pods to request storage.  Customize this file as follows:
* change 'failureDomain' in the pool definition from 'host' to 'osd' ... this will enable replication within a host, which is necessary for a small cluster
* change the 'name' in the pool definition and the corresponding 'pool' in the storage class to something descriptive -- this will be the name of the pool that is created in the ceph cluster.
* set the namespace (if it isn't set already) to 'rook-ceph' to match whatever was used to create the cluster
When done, install the manifest:  

 kubectl create -f storageclass.yaml

At this point, the Storage Provisioner is ready.

==== Ceph Filesystem Storage Class ====
There are limitations on using Ceph Filesystems within rook ...

instructions TBD

==== Default Storage Class (Optional) ====
Not all manifests specify a storage class -- this can be especially problematic with helm charts that don't expose the opportunity to specify a storage class.  Kubernetes has the concept of a '''default''' storage class that is widely used by the cloud providers to point to their preferred storage solution.  While not required, specifying a Rook StorageClass as default can simplify 'automatic' deployments.  In reality, all that needs to be done is to set a flag on the storage class that you want to be default ... but there's some prep work ...

First, identify all the defined storage classes:

 kubectl get sc

All the storage classes will be shown -- if one of them is already defined as a 'default' class, it will have '''(default)''' after the name.  If there is a default class identified (and it isn't yours) you need to turn off the default status for that class (i.e. setting a new default does NOT reset the old default):

 kubectl patch storageclass <old-default> -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

Then you can set your storageclass as the default:

 kubectl patch storageclass <new-default> -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Validate that it took by looking at the storageclasses again:

 kubectl get sc

Your class should now be marked '(default)'.

(excerpted from https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/)