Ceph-CSI Provisioner for Kubernetes

Another way to use ceph-based storage to support a kubernetes cluster is to use the native ceph-csi driver. It is developed and maintained by the ceph team itself, and it is documented as part of the ceph documentation as well as its own github page:

https://docs.ceph.com/en/latest/rbd/rbd-kubernetes/?highlight=ceph-csi#configure-ceph-csi-plugins
https://github.com/ceph/ceph-csi

This is definitely a lean deployment, but it is functional -- and since Rook doesn't handle external clusters well, it is the only way to skin this cat if you want/need to maintain an independent ceph cluster -- but still use it for dynamic storage provisioning in Kubernetes.

The documentation is not bad, so I won't repeat it here. The one change that should be made to the manifests is to put the ceph-csi provisioner into its own namespace (I chose ceph-csi). Once I got through the global_id issue discussed below, it just worked ...

A recent CVE on Ceph was published -- it affects Nautilus, Octopus, and Pacific:

https://docs.ceph.com/en/latest/security/CVE-2021-20288/

I'm not sure about the risk in a closed environment, but since the alerts in the newest versions of ceph are annoying if you don't set the parameters to prevent it, I did so when I upgraded. The problem is, ceph-csi uses the insecure way of reclaiming global_id that the parameter changes block. So ... after 2 days of playing with it (and failing) on two different kubernetes clusters, I finally found enough in the logfiles to lead me to conclude that ceph-csi was doing it wrong. I un-did the config changes to prevent it ... and it finally worked. Not good, but at least I'm not crazy.

I put in an issue on it ... let's see what they say.

All the files I used for this are in the k8s-admin repo in gitlab ...