Editing Ceph-CSI Provisioner for Kubernetes

Another way to use ceph-based storage to support a kubernetes cluster is to use the native ceph-csi driver.  It is developed and maintained by the ceph team itself, and it is documented as part of the ceph documentation as well as its own github page:

 https://docs.ceph.com/en/latest/rbd/rbd-kubernetes/?highlight=ceph-csi#configure-ceph-csi-plugins
 https://github.com/ceph/ceph-csi

==== Installation ====

This is definitely a lean deployment, but it is functional -- and since Rook doesn't handle external clusters well, it is the only way to skin this cat if you want/need to maintain an independent ceph cluster -- but still use it for dynamic storage provisioning in Kubernetes.

The documentation is not bad, so I won't repeat it here.  The one change that should be made to the manifests is to put the ceph-csi provisioner into its own namespace (I chose ''ceph-csi'').  Once I got through the ''global_id'' issue discussed below, it just worked ...

==== Default Storage Class (Optional) ====
Not all manifests specify a storage class -- this can be especially problematic with helm charts that don't expose the opportunity to specify a storage class.  Kubernetes has the concept of a '''default''' storage class that is widely used by the cloud providers to point to their preferred storage solution.  While not required, specifying a Rook StorageClass as default can simplify 'automatic' deployments.  In reality, all that needs to be done is to set a flag on the storage class that you want to be default ... but there's some prep work ...

First, identify all the defined storage classes:

 kubectl get sc

All the storage classes will be shown -- if one of them is already defined as a 'default' class, it will have '''(default)''' after the name.  If there is a default class identified (and it isn't yours) you need to turn off the default status for that class (i.e. setting a new default does NOT reset the old default):

 kubectl patch storageclass <old-default> -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

Then you can set your storageclass as the default:

 kubectl patch storageclass <new-default> -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Validate that it took by looking at the storageclasses again:

 kubectl get sc

Your class should now be marked '(default)'.

(excerpted from https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/)

==== CVE-2021-20288 ====

A recent CVE on Ceph was published -- it affects Nautilus, Octopus, and Pacific:

 https://docs.ceph.com/en/latest/security/CVE-2021-20288/

I'm not sure about the risk in a closed environment, but since the alerts in the newest versions of ceph are annoying if you don't set the parameters to prevent it, I did so when I upgraded.  The problem is, ceph-csi uses the insecure way of reclaiming global_id that the parameter changes block.  So ... after 2 days of playing with it (and failing) on two different kubernetes clusters, I finally found enough in the logfiles to lead me to conclude that ceph-csi was doing it wrong.  I un-did the config changes to prevent it ... and it finally worked.  Not good, but at least I'm not crazy.

I put in an [https://github.com/ceph/ceph-csi/issues/2063 issue] on it ... let's see what they say.

Update:  it has been fixed (previously -- I just didn't find it when searching).  using v3.3.1 of their containers takes care of it.  I will say that their documentation hadn't been updated yet to reflect that the newer version had been fixed ... and since they're using a quay registry, it's not possible to see what versions of their containers are available :-(

==== Repo ====

All the files I used for this are in the k8s-admin repo in gitlab ...