Ceph

    PLEASE NOTE: This document applies to v1.0 version and not to the latest stable release v1.8

    Documentation for other releases can be found by using the version selector in the top right of any doc page.

    Running Ceph CSI drivers with Rook

    Here is a guide on how to use Rook to deploy ceph-csi drivers on a Kubernetes cluster.

    Prerequisites

    1. a Kubernetes v1.13+ is needed in order to support CSI Spec 1.0.
    2. --allow-privileged flag set to true in kubelet and your API server
    3. An up and running Rook instance (see Rook - Ceph quickstart guide)

    CSI Drivers Enablement

    Create RBAC used by CSI drivers in the same namespace as Rook Ceph Operator

    # create rbac. Since rook operator is not permitted to create rbac rules,
    # these rules have to be created outside of operator
    kubectl apply -f cluster/examples/kubernetes/ceph/csi/rbac/rbd/
    kubectl apply -f cluster/examples/kubernetes/ceph/csi/rbac/cephfs/
    

    Start Rook Ceph Operator

    kubectl apply -f cluster/examples/kubernetes/ceph/operator-with-csi.yaml
    

    Verify CSI drivers and Operator are up and running

    # kubectl get all -n rook-ceph
    NAME                                       READY   STATUS      RESTARTS   AGE
    pod/csi-cephfsplugin-nd5tv                 2/2     Running     1          4m5s
    pod/csi-cephfsplugin-provisioner-0         2/2     Running     0          4m5s
    pod/csi-rbdplugin-provisioner-0            4/4     Running     1          4m5s
    pod/csi-rbdplugin-wr78j                    2/2     Running     1          4m5s
    pod/rook-ceph-agent-bf772                  1/1     Running     0          7m57s
    pod/rook-ceph-mgr-a-7f86bb4968-wdd4l       1/1     Running     0          5m28s
    pod/rook-ceph-mon-a-648b78fc99-jthsz       1/1     Running     0          6m1s
    pod/rook-ceph-mon-b-6f55c9b6fc-nlp4r       1/1     Running     0          5m55s
    pod/rook-ceph-mon-c-69f4f466d5-4q2jk       1/1     Running     0          5m45s
    pod/rook-ceph-operator-7464bd774c-scb5c    1/1     Running     0          4m7s
    pod/rook-ceph-osd-0-7bfdf45977-n5tt9       1/1     Running     0          2m8s
    pod/rook-ceph-osd-1-88f95577d-27jk4        1/1     Running     0          2m8s
    pod/rook-ceph-osd-2-674b4dcd4c-5wzz9       1/1     Running     0          2m8s
    pod/rook-ceph-osd-3-58f6467f6b-q5wwf       1/1     Running     0          2m8s
    pod/rook-discover-6t644                    1/1     Running     0          7m57s
    
    NAME                                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
    service/csi-cephfsplugin-provisioner   ClusterIP   10.100.46.135    <none>        1234/TCP   4m5s
    service/csi-rbdplugin-provisioner      ClusterIP   10.110.210.40    <none>        1234/TCP   4m5s
    service/rook-ceph-mgr                  ClusterIP   10.104.191.254   <none>        9283/TCP   5m13s
    service/rook-ceph-mgr-dashboard        ClusterIP   10.97.152.26     <none>        8443/TCP   5m13s
    service/rook-ceph-mon-a                ClusterIP   10.108.83.214    <none>        6789/TCP   6m4s
    service/rook-ceph-mon-b                ClusterIP   10.104.64.44     <none>        6789/TCP   5m56s
    service/rook-ceph-mon-c                ClusterIP   10.103.170.196   <none>        6789/TCP   5m45s
    
    NAME                              DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
    daemonset.apps/csi-cephfsplugin   1         1         1       1            1           <none>          4m5s
    daemonset.apps/csi-rbdplugin      1         1         1       1            1           <none>          4m5s
    daemonset.apps/rook-ceph-agent    1         1         1       1            1           <none>          7m57s
    daemonset.apps/rook-discover      1         1         1       1            1           <none>          7m57s
    
    NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/rook-ceph-mgr-a      1/1     1            1           5m28s
    deployment.apps/rook-ceph-mon-a      1/1     1            1           6m2s
    deployment.apps/rook-ceph-mon-b      1/1     1            1           5m55s
    deployment.apps/rook-ceph-mon-c      1/1     1            1           5m45s
    deployment.apps/rook-ceph-operator   1/1     1            1           10m
    deployment.apps/rook-ceph-osd-0      1/1     1            1           2m8s
    deployment.apps/rook-ceph-osd-1      1/1     1            1           2m8s
    deployment.apps/rook-ceph-osd-2      1/1     1            1           2m8s
    deployment.apps/rook-ceph-osd-3      1/1     1            1           2m8s
    
    NAME                                            DESIRED   CURRENT   READY   AGE
    replicaset.apps/rook-ceph-mgr-a-7f86bb4968      1         1         1       5m28s
    replicaset.apps/rook-ceph-mon-a-648b78fc99      1         1         1       6m1s
    replicaset.apps/rook-ceph-mon-b-6f55c9b6fc      1         1         1       5m55s
    replicaset.apps/rook-ceph-mon-c-69f4f466d5      1         1         1       5m45s
    replicaset.apps/rook-ceph-operator-6c49994c4f   0         0         0       10m
    replicaset.apps/rook-ceph-operator-7464bd774c   1         1         1       4m7s
    replicaset.apps/rook-ceph-osd-0-7bfdf45977      1         1         1       2m8s
    replicaset.apps/rook-ceph-osd-1-88f95577d       1         1         1       2m8s
    replicaset.apps/rook-ceph-osd-2-674b4dcd4c      1         1         1       2m8s
    replicaset.apps/rook-ceph-osd-3-58f6467f6b      1         1         1       2m8s
    
    NAME                                            READY   AGE
    statefulset.apps/csi-cephfsplugin-provisioner   1/1     4m5s
    statefulset.apps/csi-rbdplugin-provisioner      1/1     4m5s
    

    Once the plugin is successfully deployed, test it by running the following example.

    Test RBD CSI Driver

    Create RBD StorageClass

    This storageclass expects a pool named rbd in your Ceph cluster. You can create this pool using rook pool CRD.

    Please update monitors to reflect the Ceph monitors.

    kubectl create -f cluster/examples/kubernetes/ceph/csi/example/rbd/storageclass.yaml
    

    Create RBD Secret

    Create a Secret that matches adminid or userid specified in the storageclass.

    Find a Ceph operator pod (in the following example, the pod is rook-ceph-operator-7464bd774c-scb5c) and create a Ceph user for that pool called kubernetes:

    kubectl exec -ti -n rook-ceph rook-ceph-operator-7464bd774c-scb5c -- bash -c "ceph -c /var/lib/rook/rook-ceph/rook-ceph.config auth get-or-create-key client.kubernetes mon \"allow profile rbd\" osd \"profile rbd pool=rbd\""
    

    Then create a Secret using admin and kubernetes keyrings:

    In secret, you need your Ceph admin/user password encoded in base64.

    Run ceph auth ls in your rook ceph operator pod, to encode the key of your admin/user run echo -n KEY|base64 and replace BASE64-ENCODED-PASSWORD by your encoded key.

    kubectl exec -ti -n rook-ceph rook-ceph-operator-6c49994c4f-pwqcx /bin/sh
    sh-4.2# ceph auth ls
    installed auth entries:
    
    osd.0
    	key: AQA3pa1cN/fODBAAc/jIm5IQDClm+dmekSmSlg==
    	caps: [mgr] allow profile osd
    	caps: [mon] allow profile osd
    	caps: [osd] allow *
    osd.1
    	key: AQBXpa1cTjuYNRAAkohlInoYAa6A3odTRDhnAg==
    	caps: [mgr] allow profile osd
    	caps: [mon] allow profile osd
    	caps: [osd] allow *
    osd.2
    	key: AQB4pa1cvJidLRAALZyAtuOwArO8JZfy7Y5pFg==
    	caps: [mgr] allow profile osd
    	caps: [mon] allow profile osd
    	caps: [osd] allow *
    osd.3
    	key: AQCcpa1cFFQRHRAALBYhqO3m0FRA9pxTOFT2eQ==
    	caps: [mgr] allow profile osd
    	caps: [mon] allow profile osd
    	caps: [osd] allow *
    client.admin
    	key: AQD0pK1cqcBDCBAAdXNXfgAambPz5qWpsq0Mmw==
    	auid: 0
    	caps: [mds] allow *
    	caps: [mgr] allow *
    	caps: [mon] allow *
    	caps: [osd] allow *
    client.bootstrap-mds
    	key: AQD6pK1crJyZCxAA1UTGwtyFv3YYFcBmhWHyoQ==
    	caps: [mon] allow profile bootstrap-mds
    client.bootstrap-mgr
    	key: AQD6pK1c2KaZCxAATWi/I3i0/XEesSipy/HeIA==
    	caps: [mon] allow profile bootstrap-mgr
    client.bootstrap-osd
    	key: AQD6pK1cwa+ZCxAA7XKXRyLQpaHZ+lRXeUk8xQ==
    	caps: [mon] allow profile bootstrap-osd
    client.bootstrap-rbd
    	key: AQD6pK1cULmZCxAA4++Ch/iRKa52297/rbHP+w==
    	caps: [mon] allow profile bootstrap-rbd
    client.bootstrap-rgw
    	key: AQD6pK1cbMKZCxAAGKj5HaMoEl41LHqEafcfPA==
    	caps: [mon] allow profile bootstrap-rgw
    mgr.a
    	key: AQAZpa1chl+DAhAAYyolLBrkht+0sH0HljkFIg==
    	caps: [mds] allow *
    	caps: [mon] allow *
    	caps: [osd] allow *
    
    #encode admin/user key
    sh-4.2# echo -n AQD0pK1cqcBDCBAAdXNXfgAambPz5qWpsq0Mmw==|base64
    QVFEMHBLMWNxY0JEQ0JBQWRYTlhmZ0FhbWJQejVxV3BzcTBNbXc9PQ==
    #or
    sh-4.2# ceph auth get-key client.admin|base64
    QVFEMHBLMWNxY0JEQ0JBQWRYTlhmZ0FhbWJQejVxV3BzcTBNbXc9PQ==
    
    kubectl create -f cluster/examples/kubernetes/ceph/csi/example/rbd/secret.yaml
    

    Create RBD PersistentVolumeClaim

    Make sure your storageClassName is the name of the StorageClass previously defined in storageclass.yaml

    kubectl create -f cluster/examples/kubernetes/ceph/csi/example/rbd/pvc.yaml
    

    Verify RBD PVC has successfully been created

    # kubectl get pvc
    NAME      STATUS   VOLUME                 CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    rbd-pvc   Bound    pvc-c20495c0d5de11e8   1Gi        RWO            csi-rbd        21s
    

    If your PVC status isn’t Bound, check the csi-rbdplugin logs to see what’s preventing the PVC from being up and bound.

    Create RBD demo Pod

    kubectl create -f cluster/examples/kubernetes/ceph/csi/example/rbd/pod.yaml
    

    When running rbd list block --pool [yourpool] in one of your Ceph pods you should see the created PVC:

    # rbd list block --pool rbd
    pvc-c20495c0d5de11e8
    

    Additional features

    RBD Snapshots

    Since this feature is still in alpha stage (k8s 1.12+), make sure to enable VolumeSnapshotDataSource feature gate in your Kubernetes cluster.

    create RBD snapshot-class

    You need to create the SnapshotClass. The purpose of a SnapshotClass is defined in the kubernetes documentation. In short, as the documentation describes it:

    Just like StorageClass provides a way for administrators to describe the “classes” of storage they offer when provisioning a volume, VolumeSnapshotClass provides a way to describe the “classes” of storage when provisioning a volume snapshot.

    In snapshotClass, the csi.storage.k8s.io/snapshotter-secret-name parameter should reference the name of the secret created for the rbdplugin. The monitors are a comma separated list of your Ceph monitors and pool to reflect the Ceph pool name.

    kubectl create -f cluster/examples/kubernetes/ceph/csi/example/rbd/snapshotclass.yaml
    

    create volumesnapshot

    In snapshot, snapshotClassName should be the name of the VolumeSnapshotClass previously created. The source name should be the name of the PVC you created earlier.

    kubectl create -f cluster/examples/kubernetes/ceph/csi/example/rbd/snapshot.yaml
    

    Verify RBD Snapshot has successfully been created

    # kubectl get volumesnapshotclass
    NAME                      AGE
    csi-rbdplugin-snapclass   4s
    # kubectl get volumesnapshot
    NAME               AGE
    rbd-pvc-snapshot   6s
    

    In one of your Ceph pod, run rbd snap ls [name-of-your-pvc]. The output should be similar to this:

    # rbd snap ls pvc-c20495c0d5de11e8
    SNAPID NAME                                                                      SIZE TIMESTAMP
         4 csi-rbd-pvc-c20495c0d5de11e8-snap-4c0b455b-d5fe-11e8-bebb-525400123456 1024 MB Mon Oct 22 13:28:03 2018
    

    Restore the snapshot to a new PVC

    In pvc-restore, dataSource should be the name of the VolumeSnapshot previously created. The kind should be the VolumeSnapshot.

    Create a new PVC from the snapshot

    kubectl create -f cluster/examples/kubernetes/ceph/csi/example/rbd/pvc-restore.yaml
    

    Verify RBD clone PVC has successfully been created

    # kubectl get pvc
    NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    rbd-pvc           Bound    pvc-84294e34-577a-11e9-b34f-525400581048   1Gi        RWO            csi-rbd        34m
    rbd-pvc-restore   Bound    pvc-575537bf-577f-11e9-b34f-525400581048   1Gi        RWO            csi-rbd        8s
    

    RBD resource Cleanup

    To clean your cluster of the resources created by this example, run the following:

    if you have tested snapshot, delete snapshotclass, snapshot and pvc-restore created to test snapshot feature

    kubectl delete -f cluster/examples/kubernetes/ceph/csi/example/rbd/pvc-restore.yaml
    kubectl delete -f cluster/examples/kubernetes/ceph/csi/example/rbd/snapshot.yaml
    kubectl delete -f cluster/examples/kubernetes/ceph/csi/example/rbd/snapshotclass.yaml
    
    kubectl delete -f cluster/examples/kubernetes/ceph/csi/example/rbd/pod.yaml
    kubectl delete -f cluster/examples/kubernetes/ceph/csi/example/rbd/pvc.yaml
    kubectl delete -f cluster/examples/kubernetes/ceph/csi/example/rbd/secret.yaml
    kubectl delete -f cluster/examples/kubernetes/ceph/csi/example/rbd/storageclass.yaml
    

    Test CephFS CSI Driver

    Create CephFS StorageClass

    This storageclass expect a pool named cephfs_data in your Ceph cluster. You can create this pool using rook file-system CRD.

    Please update monitors to reflect the Ceph monitors.

    kubectl create -f cluster/examples/kubernetes/ceph/csi/example/cephfs/storageclass.yaml
    

    Create CephFS Secret

    Create a Secret that matches provisionVolume type specified in the storageclass.

    In secret you need your Ceph admin/user ID and password encoded in base64. Encode admin/user ID in base64 format and replace BASE64-ENCODED-USER.

    $echo -n admin|base64
    YWRtaW4=
    

    Run ceph auth ls in your rook ceph operator pod, to encode the key of your admin/user run echo -n KEY|base64 and replace BASE64-ENCODED-PASSWORD by your encoded key.

    kubectl exec -ti -n rook-ceph rook-ceph-operator-6c49994c4f-pwqcx /bin/sh
    sh-4.2# ceph auth ls
    installed auth entries:
    
    osd.0
    	key: AQA3pa1cN/fODBAAc/jIm5IQDClm+dmekSmSlg==
    	caps: [mgr] allow profile osd
    	caps: [mon] allow profile osd
    	caps: [osd] allow *
    osd.1
    	key: AQBXpa1cTjuYNRAAkohlInoYAa6A3odTRDhnAg==
    	caps: [mgr] allow profile osd
    	caps: [mon] allow profile osd
    	caps: [osd] allow *
    osd.2
    	key: AQB4pa1cvJidLRAALZyAtuOwArO8JZfy7Y5pFg==
    	caps: [mgr] allow profile osd
    	caps: [mon] allow profile osd
    	caps: [osd] allow *
    osd.3
    	key: AQCcpa1cFFQRHRAALBYhqO3m0FRA9pxTOFT2eQ==
    	caps: [mgr] allow profile osd
    	caps: [mon] allow profile osd
    	caps: [osd] allow *
    client.admin
    	key: AQD0pK1cqcBDCBAAdXNXfgAambPz5qWpsq0Mmw==
    	auid: 0
    	caps: [mds] allow *
    	caps: [mgr] allow *
    	caps: [mon] allow *
    	caps: [osd] allow *
    client.bootstrap-mds
    	key: AQD6pK1crJyZCxAA1UTGwtyFv3YYFcBmhWHyoQ==
    	caps: [mon] allow profile bootstrap-mds
    client.bootstrap-mgr
    	key: AQD6pK1c2KaZCxAATWi/I3i0/XEesSipy/HeIA==
    	caps: [mon] allow profile bootstrap-mgr
    client.bootstrap-osd
    	key: AQD6pK1cwa+ZCxAA7XKXRyLQpaHZ+lRXeUk8xQ==
    	caps: [mon] allow profile bootstrap-osd
    client.bootstrap-rbd
    	key: AQD6pK1cULmZCxAA4++Ch/iRKa52297/rbHP+w==
    	caps: [mon] allow profile bootstrap-rbd
    client.bootstrap-rgw
    	key: AQD6pK1cbMKZCxAAGKj5HaMoEl41LHqEafcfPA==
    	caps: [mon] allow profile bootstrap-rgw
    mgr.a
    	key: AQAZpa1chl+DAhAAYyolLBrkht+0sH0HljkFIg==
    	caps: [mds] allow *
    	caps: [mon] allow *
    	caps: [osd] allow *
    
    #encode admin/user key
    sh-4.2#echo -n AQD0pK1cqcBDCBAAdXNXfgAambPz5qWpsq0Mmw==|base64
    QVFEMHBLMWNxY0JEQ0JBQWRYTlhmZ0FhbWJQejVxV3BzcTBNbXc9PQ==
    #or
    sh-4.2#ceph auth get-key client.admin|base64
    QVFEMHBLMWNxY0JEQ0JBQWRYTlhmZ0FhbWJQejVxV3BzcTBNbXc9PQ==
    
    kubectl create -f cluster/examples/kubernetes/ceph/csi/example/cephfs/secret.yaml
    

    Create CephFS PersistentVolumeClaim

    In pvc, make sure your storageClassName is the name of the StorageClass previously defined in storageclass.yaml

    kubectl create -f cluster/examples/kubernetes/ceph/csi/example/cephfs/pvc.yaml
    

    Verify CephFS PVC has successfully been created

    # kubectl get pvc
    NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    cephfs-pvc   Bound    pvc-6bc76846-3a4a-11e9-971d-525400c2d871   1Gi        RWO            csi-cephfs     25s
    
    

    If your PVC status isn’t Bound, check the csi-cephfsplugin logs to see what’s preventing the PVC from being up and bound.

    Create CephFS demo Pod

    kubectl create -f cluster/examples/kubernetes/ceph/csi/example/cephfs/pod.yaml
    

    Once the PVC is attached to the pod, pod creation process will continue

    CephFS resource Cleanup

    To clean your cluster of the resources created by this example, run the following:

    kubectl delete -f cluster/examples/kubernetes/ceph/csi/example/cephfs/pod.yaml
    kubectl delete -f cluster/examples/kubernetes/ceph/csi/example/cephfs/pvc.yaml
    kubectl delete -f cluster/examples/kubernetes/ceph/csi/example/cephfs/secret.yaml
    kubectl delete -f cluster/examples/kubernetes/ceph/csi/example/cephfs/storageclass.yaml