Ceph-CSI Provisioner for Kubernetes

Step 1: Deploy Ceph Provisioner on Kubernetes Login to your Kubernetes cluster and Create a manifest file for deploying RBD provisioner which is an out-of-tree dynamic provisioner for Kubernetes 1.5+.

$ vim ceph-rbd-provisioner.yml Add the following contents to the file. Notice our deployment uses RBAC, so we’ll create cluster role and bindings before creating service account and deploying Ceph RBD provisioner. --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata:

 name: rbd-provisioner
 namespace: kube-system

rules:

 - apiGroups: [""]
   resources: ["persistentvolumes"]
   verbs: ["get", "list", "watch", "create", "delete"]
 - apiGroups: [""]
   resources: ["persistentvolumeclaims"]
   verbs: ["get", "list", "watch", "update"]
 - apiGroups: ["storage.k8s.io"]
   resources: ["storageclasses"]
   verbs: ["get", "list", "watch"]
 - apiGroups: [""]
   resources: ["events"]
   verbs: ["create", "update", "patch"]
 - apiGroups: [""]
   resources: ["services"]
   resourceNames: ["kube-dns","coredns"]
   verbs: ["list", "get"]
 - apiGroups: [""]
   resources: ["endpoints"]
   verbs: ["get", "list", "watch", "create", "update", "patch"]

--- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata:

 name: rbd-provisioner
 namespace: kube-system

subjects:

 - kind: ServiceAccount
   name: rbd-provisioner
   namespace: kube-system

roleRef:

 kind: ClusterRole
 name: rbd-provisioner
 apiGroup: rbac.authorization.k8s.io

--- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata:

 name: rbd-provisioner
 namespace: kube-system

rules: - apiGroups: [""]

 resources: ["secrets"]
 verbs: ["get"]

- apiGroups: [""]

 resources: ["endpoints"]
 verbs: ["get", "list", "watch", "create", "update", "patch"]

--- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata:

 name: rbd-provisioner
 namespace: kube-system

roleRef:

 apiGroup: rbac.authorization.k8s.io
 kind: Role
 name: rbd-provisioner

subjects: - kind: ServiceAccount

 name: rbd-provisioner
 namespace: kube-system

--- apiVersion: apps/v1 kind: Deployment metadata:

 name: rbd-provisioner
 namespace: kube-system

spec:

 replicas: 1
 selector:
   matchLabels:
     app: rbd-provisioner
 strategy:
   type: Recreate
 template:
   metadata:
     labels:
       app: rbd-provisioner
   spec:
     containers:
     - name: rbd-provisioner
       image: "quay.io/external_storage/rbd-provisioner:latest"
       env:
       - name: PROVISIONER_NAME
         value: ceph.com/rbd
     serviceAccount: rbd-provisioner

Apply the file to create the resources.

$ kubectl apply -f ceph-rbd-provisioner.yml clusterrole.rbac.authorization.k8s.io/rbd-provisioner created clusterrolebinding.rbac.authorization.k8s.io/rbd-provisioner created role.rbac.authorization.k8s.io/rbd-provisioner created rolebinding.rbac.authorization.k8s.io/rbd-provisioner created deployment.apps/rbd-provisioner created Confirm that RBD volume provisioner pod is running.

$ kubectl get pods -l app=rbd-provisioner -n kube-system NAME READY STATUS RESTARTS AGE rbd-provisioner-75b85f85bd-p9b8c 1/1 Running 0 3m45s Step 2: Get Ceph Admin Key and create Secret on Kubernetes Login to your Ceph Cluster and get the admin key for use by RBD provisioner.

$ sudo ceph auth get-key client.admin Save the Value of the admin user key printed out by the command above. We’ll add the key as a secret in Kubernetes.

$ kubectl create secret generic ceph-admin-secret \

   --type="kubernetes.io/rbd" \
   --from-literal=key='<key-value>' \
   --namespace=kube-system

Where <key-value> is your ceph admin key. You can confirm creation with the command below.

$ kubectl get secrets ceph-admin-secret -n kube-system NAME TYPE DATA AGE ceph-admin-secret kubernetes.io/rbd 1 5m Step 3: Create Ceph pool for Kubernetes & client key Next is to create a new Ceph Pool for Kubernetes.

$ sudo ceph ceph osd pool create <pool-name> <pg-number>

1. Example

$ sudo ceph ceph osd pool create k8s 100 For more details, check our guide: Create a Pool in Ceph Storage Cluster Then create a new client key with access to the pool created.

$ sudo ceph auth add client.kube mon 'allow r' osd 'allow rwx pool=<pool-name>'

1. Example

$ sudo ceph auth add client.kube mon 'allow r' osd 'allow rwx pool=k8s' Where k8s is the name of pool created in Ceph.

You can then associate the pool with an application and initialize it.

sudo ceph osd pool application enable <pool-name> rbd sudo rbd pool init <pool-name> Get the client key on Ceph.

$ sudo ceph auth get-key client.kube Create client secret on Kubernetes

kubectl create secret generic ceph-k8s-secret \

 --type="kubernetes.io/rbd" \
 --from-literal=key='<key-value>' \
 --namespace=kube-system

Where <key-value> is your Ceph client key.

Step 4: Create a RBD Storage Class A StorageClass provides a way for you to describe the “classes” of storage you offer in Kubernetes. We’ll create a storageclass called ceph-rbd.

$ vim ceph-rbd-sc.yml The contents to be added to file:

--- kind: StorageClass apiVersion: storage.k8s.io/v1 metadata:

 name: ceph-rbd

provisioner: ceph.com/rbd parameters:

 monitors: 10.10.10.11:6789, 10.10.10.12:6789, 10.10.10.13:6789
 pool: k8s-uat
 adminId: admin
 adminSecretNamespace: kube-system
 adminSecretName: ceph-admin-secret
 userId: kube
 userSecretNamespace: kube-system
 userSecretName: ceph-k8s-secret
 imageFormat: "2"
 imageFeatures: layering

Where:

ceph-rbd is the name of the StorageClass to be created. 10.10.10.11, 10.10.10.12 & 10.10.10.13 are the IP address of Ceph Monitors. You can list them with the command: $ sudo ceph -s

 cluster:
   id:     7795990b-7c8c-43f4-b648-d284ef2a0aba
   health: HEALTH_OK

 services:
   mon: 3 daemons, quorum cephmon01,cephmon02,cephmon03 (age 32h)
   mgr: cephmon01(active, since 30h), standbys: cephmon02
   mds: cephfs:1 {0=cephmon01=up:active} 1 up:standby
   osd: 9 osds: 9 up (since 32h), 9 in (since 32h)
   rgw: 3 daemons active (cephmon01, cephmon02, cephmon03)

 data:
   pools:   8 pools, 618 pgs
   objects: 250 objects, 76 KiB
   usage:   9.6 GiB used, 2.6 TiB / 2.6 TiB avail
   pgs:     618 active+clean

After modifying the file with correct values of Ceph monitors, apply config:

$ kubectl apply -f ceph-rbd-sc.yml storageclass.storage.k8s.io/ceph-rbd created List available StorageClasses:

kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE ceph-rbd ceph.com/rbd Delete Immediate false 17s cephfs ceph.com/cephfs Delete Immediate false 18d Step 5: Create a test Claim and Pod on Kubernetes To confirm everything is working, let’s create a test persistent volume claim.

$ vim ceph-rbd-claim.yml kind: PersistentVolumeClaim apiVersion: v1 metadata:

 name: ceph-rbd-claim1

spec:

 accessModes:
   - ReadWriteOnce
 storageClassName: ceph-rbd
 resources:
   requests:
     storage: 1Gi

Apply manifest file to create claim.

$ kubectl apply -f ceph-rbd-claim.yml persistentvolumeclaim/ceph-rbd-claim1 created If it was successful in binding, it should show Bound status.

$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE ceph-rbd-claim1 Bound pvc-c6f4399d-43cf-4fc1-ba14-cc22f5c85304 1Gi RWO ceph-rbd 43s Nice!.. We are able create dynamic Persistent Volume Claims on Ceph RBD backend. Notice we didn’t have to manually create a Persistent Volume before a Claim. How cool is that?..

We can then deploy a test pod using the claim we created. First create a file to hold the data:

$ vim rbd-test-pod.yaml Add:

--- kind: Pod apiVersion: v1 metadata:

 name: rbd-test-pod

spec:

 containers:
 - name: rbd-test-pod
   image: busybox
   command:
     - "/bin/sh"
   args:
     - "-c"
     - "touch /mnt/RBD-SUCCESS && exit 0 || exit 1"
   volumeMounts:
     - name: pvc
       mountPath: "/mnt"
 restartPolicy: "Never"
 volumes:
   - name: pvc
     persistentVolumeClaim:
       claimName: ceph-rbd-claim1 

Create pod:

$ kubectl apply -f rbd-test-pod.yaml pod/rbd-test-pod created If you describe the Pod, you’ll see successful attachment of the Volume.

$ kubectl describe pod rbd-test-pod ..... vents:

 Type    Reason                  Age        From                     Message
 ----    ------                  ----       ----                     -------
 Normal  Scheduled               <unknown>  default-scheduler        Successfully assigned default/rbd-test-pod to rke-worker-02
 Normal  SuccessfulAttachVolume  3s         attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-c6f4399d-43cf-4fc1-ba14-cc22f5c85304"

If you have Ceph Dashboard, you can see a new block image created.

ceph kubernetes rbd