Ceph-CSI Provisioner for Kubernetes
Step 1: Deploy Ceph Provisioner on Kubernetes Login to your Kubernetes cluster and Create a manifest file for deploying RBD provisioner which is an out-of-tree dynamic provisioner for Kubernetes 1.5+.
$ vim ceph-rbd-provisioner.yml Add the following contents to the file. Notice our deployment uses RBAC, so we’ll create cluster role and bindings before creating service account and deploying Ceph RBD provisioner. --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata:
name: rbd-provisioner namespace: kube-system
rules:
- apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["create", "update", "patch"] - apiGroups: [""] resources: ["services"] resourceNames: ["kube-dns","coredns"] verbs: ["list", "get"] - apiGroups: [""] resources: ["endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch"]
--- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata:
name: rbd-provisioner namespace: kube-system
subjects:
- kind: ServiceAccount name: rbd-provisioner namespace: kube-system
roleRef:
kind: ClusterRole name: rbd-provisioner apiGroup: rbac.authorization.k8s.io
--- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata:
name: rbd-provisioner namespace: kube-system
rules: - apiGroups: [""]
resources: ["secrets"] verbs: ["get"]
- apiGroups: [""]
resources: ["endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch"]
--- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata:
name: rbd-provisioner namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io kind: Role name: rbd-provisioner
subjects: - kind: ServiceAccount
name: rbd-provisioner namespace: kube-system
--- apiVersion: apps/v1 kind: Deployment metadata:
name: rbd-provisioner namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app: rbd-provisioner
strategy:
type: Recreate
template:
metadata:
labels:
app: rbd-provisioner
spec:
containers:
- name: rbd-provisioner
image: "quay.io/external_storage/rbd-provisioner:latest"
env:
- name: PROVISIONER_NAME
value: ceph.com/rbd
serviceAccount: rbd-provisioner
Apply the file to create the resources.
$ kubectl apply -f ceph-rbd-provisioner.yml clusterrole.rbac.authorization.k8s.io/rbd-provisioner created clusterrolebinding.rbac.authorization.k8s.io/rbd-provisioner created role.rbac.authorization.k8s.io/rbd-provisioner created rolebinding.rbac.authorization.k8s.io/rbd-provisioner created deployment.apps/rbd-provisioner created Confirm that RBD volume provisioner pod is running.
$ kubectl get pods -l app=rbd-provisioner -n kube-system NAME READY STATUS RESTARTS AGE rbd-provisioner-75b85f85bd-p9b8c 1/1 Running 0 3m45s Step 2: Get Ceph Admin Key and create Secret on Kubernetes Login to your Ceph Cluster and get the admin key for use by RBD provisioner.
$ sudo ceph auth get-key client.admin Save the Value of the admin user key printed out by the command above. We’ll add the key as a secret in Kubernetes.
$ kubectl create secret generic ceph-admin-secret \
--type="kubernetes.io/rbd" \ --from-literal=key='<key-value>' \ --namespace=kube-system
Where <key-value> is your ceph admin key. You can confirm creation with the command below.
$ kubectl get secrets ceph-admin-secret -n kube-system NAME TYPE DATA AGE ceph-admin-secret kubernetes.io/rbd 1 5m Step 3: Create Ceph pool for Kubernetes & client key Next is to create a new Ceph Pool for Kubernetes.
$ sudo ceph ceph osd pool create <pool-name> <pg-number>
- Example
$ sudo ceph ceph osd pool create k8s 100 For more details, check our guide: Create a Pool in Ceph Storage Cluster Then create a new client key with access to the pool created.
$ sudo ceph auth add client.kube mon 'allow r' osd 'allow rwx pool=<pool-name>'
- Example
$ sudo ceph auth add client.kube mon 'allow r' osd 'allow rwx pool=k8s' Where k8s is the name of pool created in Ceph.
You can then associate the pool with an application and initialize it.
sudo ceph osd pool application enable <pool-name> rbd sudo rbd pool init <pool-name> Get the client key on Ceph.
$ sudo ceph auth get-key client.kube Create client secret on Kubernetes
kubectl create secret generic ceph-k8s-secret \
--type="kubernetes.io/rbd" \ --from-literal=key='<key-value>' \ --namespace=kube-system
Where <key-value> is your Ceph client key.
Step 4: Create a RBD Storage Class A StorageClass provides a way for you to describe the “classes” of storage you offer in Kubernetes. We’ll create a storageclass called ceph-rbd.
$ vim ceph-rbd-sc.yml The contents to be added to file:
--- kind: StorageClass apiVersion: storage.k8s.io/v1 metadata:
name: ceph-rbd
provisioner: ceph.com/rbd parameters:
monitors: 10.10.10.11:6789, 10.10.10.12:6789, 10.10.10.13:6789 pool: k8s-uat adminId: admin adminSecretNamespace: kube-system adminSecretName: ceph-admin-secret userId: kube userSecretNamespace: kube-system userSecretName: ceph-k8s-secret imageFormat: "2" imageFeatures: layering
Where:
ceph-rbd is the name of the StorageClass to be created. 10.10.10.11, 10.10.10.12 & 10.10.10.13 are the IP address of Ceph Monitors. You can list them with the command: $ sudo ceph -s
cluster:
id: 7795990b-7c8c-43f4-b648-d284ef2a0aba
health: HEALTH_OK
services:
mon: 3 daemons, quorum cephmon01,cephmon02,cephmon03 (age 32h)
mgr: cephmon01(active, since 30h), standbys: cephmon02
mds: cephfs:1 {0=cephmon01=up:active} 1 up:standby
osd: 9 osds: 9 up (since 32h), 9 in (since 32h)
rgw: 3 daemons active (cephmon01, cephmon02, cephmon03)
data:
pools: 8 pools, 618 pgs
objects: 250 objects, 76 KiB
usage: 9.6 GiB used, 2.6 TiB / 2.6 TiB avail
pgs: 618 active+clean
After modifying the file with correct values of Ceph monitors, apply config:
$ kubectl apply -f ceph-rbd-sc.yml storageclass.storage.k8s.io/ceph-rbd created List available StorageClasses:
kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE ceph-rbd ceph.com/rbd Delete Immediate false 17s cephfs ceph.com/cephfs Delete Immediate false 18d Step 5: Create a test Claim and Pod on Kubernetes To confirm everything is working, let’s create a test persistent volume claim.
$ vim ceph-rbd-claim.yml kind: PersistentVolumeClaim apiVersion: v1 metadata:
name: ceph-rbd-claim1
spec:
accessModes:
- ReadWriteOnce
storageClassName: ceph-rbd
resources:
requests:
storage: 1Gi
Apply manifest file to create claim.
$ kubectl apply -f ceph-rbd-claim.yml persistentvolumeclaim/ceph-rbd-claim1 created If it was successful in binding, it should show Bound status.
$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE ceph-rbd-claim1 Bound pvc-c6f4399d-43cf-4fc1-ba14-cc22f5c85304 1Gi RWO ceph-rbd 43s Nice!.. We are able create dynamic Persistent Volume Claims on Ceph RBD backend. Notice we didn’t have to manually create a Persistent Volume before a Claim. How cool is that?..
We can then deploy a test pod using the claim we created. First create a file to hold the data:
$ vim rbd-test-pod.yaml Add:
--- kind: Pod apiVersion: v1 metadata:
name: rbd-test-pod
spec:
containers:
- name: rbd-test-pod
image: busybox
command:
- "/bin/sh"
args:
- "-c"
- "touch /mnt/RBD-SUCCESS && exit 0 || exit 1"
volumeMounts:
- name: pvc
mountPath: "/mnt"
restartPolicy: "Never"
volumes:
- name: pvc
persistentVolumeClaim:
claimName: ceph-rbd-claim1
Create pod:
$ kubectl apply -f rbd-test-pod.yaml pod/rbd-test-pod created If you describe the Pod, you’ll see successful attachment of the Volume.
$ kubectl describe pod rbd-test-pod ..... vents:
Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> default-scheduler Successfully assigned default/rbd-test-pod to rke-worker-02 Normal SuccessfulAttachVolume 3s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-c6f4399d-43cf-4fc1-ba14-cc22f5c85304"
If you have Ceph Dashboard, you can see a new block image created.
ceph kubernetes rbd