Ceph

    PLEASE NOTE: This document applies to an unreleased version of Rook. It is strongly recommended that you only use official releases of Rook, as unreleased versions are subject to changes and incompatibilities that will not be supported in the official releases.

    If you are using an official release version of Rook, you should refer to the documentation for your specific version.

    Documentation for other releases can be found by using the version selector in the bottom left of any doc page.

    Direct Tools

    Rook is designed with Kubernetes design principles from the ground up. This topic is going to escape the bounds of Kubernetes storage and show you how to use block and file storage directly from a pod without any of the Kubernetes magic. The purpose of this topic is to help you quickly test a new configuration, although it is not meant to be used in production. All of the benefits of Kubernetes storage including failover, detach, and attach will not be available. If your pod dies, your mount will die with it.

    Start the Direct Mount Pod

    To test mounting your Ceph volumes, start a pod with the necessary mounts. An example is provided in the examples test directory:

    kubectl create -f deploy/examples/direct-mount.yaml
    

    After the pod is started, connect to it like this:

    kubectl -n rook-ceph get pod -l app=rook-direct-mount
    $ kubectl -n rook-ceph exec -it <pod> bash
    

    Block Storage Tools

    After you have created a pool as described in the Block Storage topic, you can create a block image and mount it directly in a pod. This example will show how the Ceph rbd volume can be mounted in the direct mount pod.

    Create the Direct Mount Pod.

    Create a volume image (10MB):

    rbd create replicapool/test --size 10
    rbd info replicapool/test
    
    # Disable the rbd features that are not in the kernel module
    rbd feature disable replicapool/test fast-diff deep-flatten object-map
    

    Map the block volume and format it and mount it:

    # Map the rbd device. If the Direct Mount Pod was started with "hostNetwork: false" this hangs and you have to stop it with Ctrl-C,
    # however the command still succeeds; see https://github.com/rook/rook/issues/2021
    rbd map replicapool/test
    
    # Find the device name, such as rbd0
    lsblk | grep rbd
    
    # Format the volume (only do this the first time or you will lose data)
    mkfs.ext4 -m0 /dev/rbd0
    
    # Mount the block device
    mkdir /tmp/rook-volume
    mount /dev/rbd0 /tmp/rook-volume
    

    Write and read a file:

    echo "Hello Rook" > /tmp/rook-volume/hello
    cat /tmp/rook-volume/hello
    

    Unmount the Block device

    Unmount the volume and unmap the kernel device:

    umount /tmp/rook-volume
    rbd unmap /dev/rbd0
    

    Shared Filesystem Tools

    After you have created a filesystem as described in the Shared Filesystem topic, you can mount the filesystem from multiple pods. The the other topic you may have mounted the filesystem already in the registry pod. Now we will mount the same filesystem in the Direct Mount pod. This is just a simple way to validate the Ceph filesystem and is not recommended for production Kubernetes pods.

    Follow Direct Mount Pod to start a pod with the necessary mounts and then proceed with the following commands after connecting to the pod.

    # Create the directory
    mkdir /tmp/registry
    
    # Detect the mon endpoints and the user secret for the connection
    mon_endpoints=$(grep mon_host /etc/ceph/ceph.conf | awk '{print $3}')
    my_secret=$(grep key /etc/ceph/keyring | awk '{print $3}')
    
    # Mount the filesystem
    mount -t ceph -o mds_namespace=myfs,name=admin,secret=$my_secret $mon_endpoints:/ /tmp/registry
    
    # See your mounted filesystem
    df -h
    

    Now you should have a mounted filesystem. If you have pushed images to the registry you will see a directory called docker.

    ls /tmp/registry
    

    Try writing and reading a file to the shared filesystem.

    echo "Hello Rook" > /tmp/registry/hello
    cat /tmp/registry/hello
    
    # delete the file when you're done
    rm -f /tmp/registry/hello
    

    Unmount the Filesystem

    To unmount the shared filesystem from the Direct Mount Pod:

    umount /tmp/registry
    rmdir /tmp/registry
    

    No data will be deleted by unmounting the filesystem.