Ceph-CSI Provisioner for Kubernetes: Difference between revisions
DrEdWilliams (talk | contribs) mNo edit summary |
DrEdWilliams (talk | contribs) mNo edit summary |
||
| Line 1: | Line 1: | ||
Excerpted from https://computingforgeeks.com/persistent-storage-for-kubernetes-with-ceph-rbd/ | |||
== Step 1: Deploy Ceph Provisioner on Kubernetes == | == Step 1: Deploy Ceph Provisioner on Kubernetes == | ||
Login to your Kubernetes cluster and Create a manifest file for deploying RBD provisioner which is an out-of-tree dynamic provisioner for Kubernetes 1.5+. | Login to your Kubernetes cluster and Create a manifest file for deploying RBD provisioner which is an out-of-tree dynamic provisioner for Kubernetes 1.5+. | ||
Revision as of 20:17, 7 May 2021
Excerpted from https://computingforgeeks.com/persistent-storage-for-kubernetes-with-ceph-rbd/
Step 1: Deploy Ceph Provisioner on Kubernetes
Login to your Kubernetes cluster and Create a manifest file for deploying RBD provisioner which is an out-of-tree dynamic provisioner for Kubernetes 1.5+.
$ vim ceph-rbd-provisioner.yml
Add the following contents to the file. Notice our deployment uses RBAC, so we’ll create cluster role and bindings before creating service account and deploying Ceph RBD provisioner.
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rbd-provisioner
namespace: kube-system
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
- apiGroups: [""]
resources: ["services"]
resourceNames: ["kube-dns","coredns"]
verbs: ["list", "get"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rbd-provisioner
namespace: kube-system
subjects:
- kind: ServiceAccount
name: rbd-provisioner
namespace: kube-system
roleRef:
kind: ClusterRole
name: rbd-provisioner
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: rbd-provisioner
namespace: kube-system
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: rbd-provisioner
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: rbd-provisioner
subjects:
- kind: ServiceAccount
name: rbd-provisioner
namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: rbd-provisioner
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app: rbd-provisioner
strategy:
type: Recreate
template:
metadata:
labels:
app: rbd-provisioner
spec:
containers:
- name: rbd-provisioner
image: "quay.io/external_storage/rbd-provisioner:latest"
env:
- name: PROVISIONER_NAME
value: ceph.com/rbd
serviceAccount: rbd-provisioner
Apply the file to create the resources.
$ kubectl apply -f ceph-rbd-provisioner.yml clusterrole.rbac.authorization.k8s.io/rbd-provisioner created clusterrolebinding.rbac.authorization.k8s.io/rbd-provisioner created role.rbac.authorization.k8s.io/rbd-provisioner created rolebinding.rbac.authorization.k8s.io/rbd-provisioner created deployment.apps/rbd-provisioner created
Confirm that RBD volume provisioner pod is running.
$ kubectl get pods -l app=rbd-provisioner -n kube-system NAME READY STATUS RESTARTS AGE rbd-provisioner-75b85f85bd-p9b8c 1/1 Running 0 3m45s
Step 2: Get Ceph Admin Key and create Secret on Kubernetes
Login to your Ceph Cluster and get the admin key for use by RBD provisioner.
$ sudo ceph auth get-key client.admin
Save the Value of the admin user key printed out by the command above. We’ll add the key as a secret in Kubernetes.
$ kubectl create secret generic ceph-admin-secret \
--type="kubernetes.io/rbd" \
--from-literal=key='<key-value>' \
--namespace=kube-system
Where <key-value> is your ceph admin key. You can confirm creation with the command below.
$ kubectl get secrets ceph-admin-secret -n kube-system NAME TYPE DATA AGE ceph-admin-secret kubernetes.io/rbd 1 5m
Step 3: Create Ceph pool for Kubernetes & client key
Next is to create a new Ceph Pool for Kubernetes.
$ sudo ceph ceph osd pool create <pool-name> <pg-number>
Example:
$ sudo ceph ceph osd pool create k8s 100
Then create a new client key with access to the pool created.
$ sudo ceph auth add client.kube mon 'allow r' osd 'allow rwx pool=<pool-name>'
Example
$ sudo ceph auth add client.kube mon 'allow r' osd 'allow rwx pool=k8s'
Where k8s is the name of pool created in Ceph.
You can then associate the pool with an application and initialize it.
sudo ceph osd pool application enable <pool-name> rbd sudo rbd pool init <pool-name>
Get the client key on Ceph.
$ sudo ceph auth get-key client.kube
Create client secret on Kubernetes
kubectl create secret generic ceph-k8s-secret \ --type="kubernetes.io/rbd" \ --from-literal=key='<key-value>' \ --namespace=kube-system
Where <key-value> is your Ceph client key.
Step 4: Create a RBD Storage Class
A StorageClass provides a way for you to describe the “classes” of storage you offer in Kubernetes. We’ll create a storageclass called ceph-rbd.
$ vim ceph-rbd-sc.yml
The contents to be added to file:
--- kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: ceph-rbd provisioner: ceph.com/rbd parameters: monitors: 10.10.10.11:6789, 10.10.10.12:6789, 10.10.10.13:6789 pool: k8s-uat adminId: admin adminSecretNamespace: kube-system adminSecretName: ceph-admin-secret userId: kube userSecretNamespace: kube-system userSecretName: ceph-k8s-secret imageFormat: "2" imageFeatures: layering
Where:
- ceph-rbd is the name of the StorageClass to be created.
- 10.10.10.11, 10.10.10.12 & 10.10.10.13 are the IP address of Ceph Monitors.
You can list them with the command:
$ sudo ceph -s
cluster:
id: 7795990b-7c8c-43f4-b648-d284ef2a0aba
health: HEALTH_OK
services:
mon: 3 daemons, quorum cephmon01,cephmon02,cephmon03 (age 32h)
mgr: cephmon01(active, since 30h), standbys: cephmon02
mds: cephfs:1 {0=cephmon01=up:active} 1 up:standby
osd: 9 osds: 9 up (since 32h), 9 in (since 32h)
rgw: 3 daemons active (cephmon01, cephmon02, cephmon03)
data:
pools: 8 pools, 618 pgs
objects: 250 objects, 76 KiB
usage: 9.6 GiB used, 2.6 TiB / 2.6 TiB avail
pgs: 618 active+clean
After modifying the file with correct values of Ceph monitors, apply config:
$ kubectl apply -f ceph-rbd-sc.yml storageclass.storage.k8s.io/ceph-rbd created
List available StorageClasses:
kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE ceph-rbd ceph.com/rbd Delete Immediate false 17s cephfs ceph.com/cephfs Delete Immediate false 18d
Step 5: Create a test Claim and Pod on Kubernetes
To confirm everything is working, let’s create a test persistent volume claim.
$ vim ceph-rbd-claim.yml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: ceph-rbd-claim1
spec:
accessModes:
- ReadWriteOnce
storageClassName: ceph-rbd
resources:
requests:
storage: 1Gi
Apply manifest file to create claim.
$ kubectl apply -f ceph-rbd-claim.yml persistentvolumeclaim/ceph-rbd-claim1 created
If it was successful in binding, it should show Bound status.
$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE ceph-rbd-claim1 Bound pvc-c6f4399d-43cf-4fc1-ba14-cc22f5c85304 1Gi RWO ceph-rbd 43s
Nice!.. We are able create dynamic Persistent Volume Claims on Ceph RBD backend. Notice we didn’t have to manually create a Persistent Volume before a Claim. How cool is that?..
We can then deploy a test pod using the claim we created. First create a file to hold the data:
$ vim rbd-test-pod.yaml
Add:
---
kind: Pod
apiVersion: v1
metadata:
name: rbd-test-pod
spec:
containers:
- name: rbd-test-pod
image: busybox
command:
- "/bin/sh"
args:
- "-c"
- "touch /mnt/RBD-SUCCESS && exit 0 || exit 1"
volumeMounts:
- name: pvc
mountPath: "/mnt"
restartPolicy: "Never"
volumes:
- name: pvc
persistentVolumeClaim:
claimName: ceph-rbd-claim1
Create pod:
$ kubectl apply -f rbd-test-pod.yaml pod/rbd-test-pod created
If you describe the Pod, you’ll see successful attachment of the Volume.
$ kubectl describe pod rbd-test-pod ..... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> default-scheduler Successfully assigned default/rbd-test-pod to rke-worker-02 Normal SuccessfulAttachVolume 3s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-c6f4399d-43cf-4fc1-ba14- cc22f5c85304"
If you have Ceph Dashboard, you can see a new block image created.