Ceph

EdgeFS Rook integrated CSI driver, provisioner, attacher and snapshotter

Container Storage Interface (CSI) driver, provisioner, attacher and snapshotter for EdgeFS Scale-Out NFS/ISCSI services

Overview

EdgeFS CSI plugins implement an interface between CSI enabled Container Orchestrator (CO) and EdgeFS local cluster site. It allows dynamic and static provisioning of EdgeFS NFS exports and ISCSI LUNs, and attaching them to application workloads. With EdgeFS NFS/ISCSI implementation, I/O load can be spread-out across multiple PODs, thus eliminating I/O bottlenecks of classing single-node NFS/ISCSI and providing highly available persistent volumes. Current implementation of EdgeFS CSI plugins was tested in Kubernetes environment (requires Kubernetes 1.13+)

Prerequisites

• Ensure your kubernetes cluster version is 1.13+
• Kubernetes cluster must allow privileged pods, this flag must be set for the API server and the kubelet (instructions):

  --allow-privileged=true
  

• Required the API server and the kubelet feature gates (instructions):

  --feature-gates=VolumeSnapshotDataSource=true,CSIDriverRegistry=true
  

• Mount propagation must be enabled, the Docker daemon for the cluster must allow shared mounts (instructions)
• Kubernetes CSI drivers require CSIDriver and CSINodeInfo resource types to be defined on the cluster. Check if they are already defined:

  kubectl get customresourcedefinition.apiextensions.k8s.io/csidrivers.csi.storage.k8s.io
  kubectl get customresourcedefinition.apiextensions.k8s.io/csinodeinfos.csi.storage.k8s.io
  

  If the cluster doesn’t have “csidrivers” and “csinodeinfos” resource types, create them:

  kubectl create -f https://raw.githubusercontent.com/kubernetes/csi-api/release-1.13/pkg/crd/manifests/csidriver.yaml
  kubectl create -f https://raw.githubusercontent.com/kubernetes/csi-api/release-1.13/pkg/crd/manifests/csinodeinfo.yaml
  

• Depends on preferred CSI driver type, following utilities must be installed on each Kubernetes node (For Debian/Ubuntu based systems):

  # for NFS
  apt install -y nfs-common rpcbind
  # for ISCSI
  apt install -y open-iscsi
  

EdgeFS CSI drivers configuration

For each driver type (NFS/ISCSI) we have already prepared configuration files examples, there are:

EdgeFS CSI NFS driver config

EdgeFS CSI ISCSI driver config

Secret file configuration options example:

# EdgeFS k8s cluster options
k8sEdgefsNamespaces: ["rook-edgefs"]          # edgefs cluster namespace
k8sEdgefsMgmtPrefix: rook-edgefs-mgr     # edgefs cluster management prefix

# EdgeFS csi operations options
cluster: cltest           # substitution edgefs cluster name for csi operations
tenant: test              # substitution edgefs tenant name for csi operations
#serviceFilter: "nfs01"   # comma delimited list of allowed service names for filtering

# EdgeFS GRPC security options
username: admin           # edgefs k8s cluster grpc service username
password: admin           # edgefs k8s cluster grpc service password

Options for NFS and ISCSI configuration files

By using k8sEdgefsNamespaces and k8sEdgefsMgmtPrefix parameters, driver is capable of detecting ClusterIPs and Endpoint IPs to provision and attach volumes.

Apply EdgeFS CSI NFS driver configuration

Check configuration options and create kubernetes secret for Edgefs CSI NFS plugin

cd cluster/examples/kubernetes/edgefs/csi/nfs
kubectl create secret generic edgefs-nfs-csi-driver-config --from-file=./edgefs-nfs-csi-driver-config.yaml

Deploy EdgeFS CSI NFS driver

After secret is created successfully, deploy EdgeFS CSI plugin, provisioner and attacher using the following command

cd cluster/examples/kubernetes/edgefs/csi/nfs
kubectl apply -f edgefs-nfs-csi-driver.yaml

There should be number of EdgeFS CSI plugin PODs available running as a DaemonSet

...
NAMESPACE     NAME                           READY   STATUS    RESTARTS   AGE
default       edgefs-nfs-csi-controller-0    4/4     Running   0          33s
default       edgefs-nfs-csi-node-9st9n      2/2     Running   0          33s
default       edgefs-nfs-csi-node-js7jp      2/2     Running   0          33s
default       edgefs-nfs-csi-node-lhjgr      2/2     Running   0          33s
...

At this point configuration is all ready and available for consumption by applications.

Pre-provisioned volumes (NFS)

This method allows to use already created exports in EdgeFS services. This method keeps exports provisioned after application PODs terminated. Read more on how to create PersistentVolume specification for pre-provisioned volumes:

link to Pre-provisioned volumes manifest specification

To test creation and mount pre-provisioned volume to pod execute example

Note:

Make sure that volumeHandle: segment:service@cluster/tenant/bucket in nginx.yaml already exist on EdgeFS cluster and served via any Edgefs NFS service. Any volumeHandle’s parameters may be omitted and will be substituted via CSI configuration file parameters.

Examples:

cd cluster/examples/kubernetes/edgefs/csi/nfs/examples
kubectl apply -f ./preprovisioned-edgefs-volume-nginx.yaml

Dynamically provisioned volumes (NFS)

To setup the system for dynamic provisioning, administrator needs to setup a StorageClass pointing to the CSI driver’s external-provisioner and specifying any parameters required by the driver

link to dynamically provisioned volumes specification

Note:

For dynamically provisioned volumes kubernetes will generate volume name automatically (for example pvc-871068ed-8b5d-11e8-9dae-005056b37cb2) Additional creation options should be passed as parameters in storage class definition i.e :

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: edgefs-nfs-csi-storageclass
provisioner: io.edgefs.csi.nfs
parameters:
  segment: rook-edgefs
  service: nfs01
  tenant: ten1
  encryption: true

Options:

Note:

Options are case sensitive and should be in lower case

Example:

cd cluster/examples/kubernetes/edgefs/csi/nfs/examples
kubectl apply -f ./dynamic-nginx.yaml

Apply Edgefs CSI ISCSI driver configuration

Check configuration options and create kubernetes secret for Edgefs CSI ISCSI plugin

cd cluster/examples/kubernetes/edgefs/csi/iscsi
kubectl create secret generic edgefs-iscsi-csi-driver-config --from-file=./edgefs-iscsi-csi-driver-config.yaml

Deploy Edgefs CSI ISCSI driver

After secret is created successfully, deploy EdgeFS CSI plugin, provisioner, attacher and snapshotter using the following command

cd cluster/examples/kubernetes/edgefs/csi/iscsi
kubectl apply -f edgefs-iscsi-csi-driver.yaml

There should be number of EdgeFS CSI ISCSI plugin PODs available running as a DaemonSet

...
NAMESPACE     NAME                             READY   STATUS    RESTARTS   AGE
default       edgefs-iscsi-csi-controller-0    4/4     Running   0          12s
default       edgefs-iscsi-csi-node-26464      2/2     Running   0          12s
default       edgefs-iscsi-csi-node-p5r58      2/2     Running   0          12s
default       edgefs-iscsi-csi-node-ptn2m      2/2     Running   0          12s
...

At this point configuration is all ready and available for consumption by applications.

Pre-provisioned volumes (ISCSI)

This method allows to use already created exports in EdgeFS ISCSI services. This method keeps exports provisioned after application PODs terminated. Read more on how to create PersistentVolume specification for pre-provisioned volumes:

link to Pre-provisioned volumes manifest specification

To test creation and mount pre-provisioned volume to pod execute example

Note:

Make sure that volumeHandle: segment:service@cluster/tenant/bucket/lun in nginx.yaml already exist on EdgeFS cluster and served via any Edgefs ISCSI service. Any volumeHandle’s parameters may be omitted and will be substituted via CSI configuration file parameters.

Examples:

cd cluster/examples/kubernetes/edgefs/csi/iscsi/examples
kubectl apply -f ./preprovisioned-edgefs-volume-nginx.yaml

Dynamically provisioned volumes (ISCSI)

For dynamic volume provisioning, the administrator needs to set up a StorageClass pointing to the driver. In this case Kubernetes generates volume name automatically (for example pvc-ns-cfc67950-fe3c-11e8-a3ca-005056b857f8). Default driver configuration may be overwritten in parameters section:

link to dynamically provisioned volumes specification

Note:

For dynamically provisioned volumes kubernetes will generate volume name automatically (for example pvc-871068ed-8b5d-11e8-9dae-005056b37cb2) Additional creation options should be passed as parameters in storage class definition i.e :

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: edgefs-iscsi-csi-storageclass
provisioner: io.edgefs.csi.nfs
parameters:
  segment: rook-edgefs
  service: iscsi01
  cluster: cltest
  tenant: test
  bucket: bk1
  encryption: true

Parameters:

Note:

Options are case sensitive and should be in lower case

Example:

cd cluster/examples/kubernetes/edgefs/csi/nfs/examples
kubectl apply -f ./dynamic-nginx.yaml

EdgeFS CSI ISCSI driver snapshots and clones

Getting information about existing snapshots

# snapshot classes
kubectl get volumesnapshotclasses.snapshot.storage.k8s.io

# snapshot list
kubectl get volumesnapshots.snapshot.storage.k8s.io

# volumesnapshotcontents
kubectl get volumesnapshotcontents.snapshot.storage.k8s.io

To create volume’s clone from existing snapshot you should:

• Create snapshotter storage class Example yaml
• Have an existing PVC based on EdgeFS ISCSI LUN
• Take snapshot from volume Example yaml
• Clone volume from existing snapshot Example yaml

Troubleshooting and log collection

For details about other configuration and deployment of NFS, ISCSI and EdgeFS CSI plugin, see Wiki pages:

• Quick Start Guide

Please submit an issue at: Issues

Troubleshooting

• Show installed drivers:

  kubectl get csidrivers.csi.storage.k8s.io
  kubectl describe csidrivers.csi.storage.k8s.io
  

• Error:

  MountVolume.MountDevice failed for volume "pvc-ns-<...>" :
  driver name io.edgefs.csi.iscsi not found in the list of registered CSI drivers
  

  Make sure kubelet is configured with --root-dir=/var/lib/kubelet, otherwise update paths in the driver yaml file (all requirements).

• “VolumeSnapshotDataSource” feature gate is disabled:

  vim /var/lib/kubelet/config.yaml
  # ...
  # featureGates:
  #   VolumeSnapshotDataSource: true
  # ...
  vim /etc/kubernetes/manifests/kube-apiserver.yaml
  # ...
  #     - --feature-gates=VolumeSnapshotDataSource=true
  # ...
  

• Driver logs (for ISCSI driver, to get NFS driver logs substitute iscsi to nfs)

  kubectl logs -f edgefs-iscsi-csi-controller-0 driver
  kubectl logs -f $(kubectl get pods | awk '/edgefs-iscsi-csi-node-/ {print $1;exit}') driver
  # combine all pods:
  kubectl get pods | awk '/edgefs-iscsi-csi-node-/ {system("kubectl logs " $1 " driver &")}'
  

• Show termination message in case driver failed to run:

  kubectl get pod edgefs-iscsi-csi-controller-0 -o go-template="{{range .status.containerStatuses}}{{.lastState.terminated.message}}{{end}}"