OpenEBS (storage-app)

gravity
#1

I am preparing to deploy a new application, and need a solution for in cluster (easy to implement) persistent storage. I have identified and would like to deploy OpenEBS. I have noticed that there is a gravitational repo called storage-app that is described as a System Application that provides OpenEBS storage for clusters. Would anyone be able to give me any info on this or a nudge in the right direction…

Thanks for any help you can provide…

(P.S. if there is a better in cluster persistent storage solution that is easy and proven, that would also be of interest)

#2

Hello @54321dev!

Yes, storage-app repository contains a Gravity system application that will provide persistent storage using OpenEBS in Gravity clusters. It will be integrated in the platform starting from Gravity release 7.0 (which isn’t out yet), the work is still going on this one.

In the meanwhile, feel free to deploy OpenEBS yourself, just keep in mind that you will need to enable privileged containers (which are by default disabled) in order for OpenEBS Node Device Manager to work properly. See Running Privileged Containers for details.

Hope this helps,
Roman

#3

@r0mant
Hi, I am facing some issues in the ndm pod being in the containercreating stage. Following is some of the important parameters in app.yaml which allows privileged containers as well

providers:
  default: "generic"
hooks:
  install:
    job: file://install.yaml

systemOptions:
  allowPrivileged: true

Install.yaml is running a script which is installing openEBS

apiVersion: batch/v1
 kind: Job
 metadata:
   name: install
 spec:
   template:
     metadata:
       name: install
     spec:
       restartPolicy: OnFailure
       containers:
         - name: install
           image: ubuntu:18.04
           command: ["/bin/bash", "-c", "/var/lib/gravity/resources/install.sh"]

Install.sh is the following script

#!/bin/bash

set -x

echo -e "Installing OpenEBS...\n"
/usr/local/bin/kubectl create -f /var/lib/gravity/resources/openebs-operator.yaml

The script is at openebs-operator.yaml

Following are the details of the pods

ubuntu@aws08:~$ kubectl get pods -n openebs
NAME                                          READY   STATUS              RESTARTS   AGE
maya-apiserver-5b5879656d-p52q7               1/1     Running             2          10h
openebs-admission-server-7fc49b89b6-lqdhv     1/1     Running             0          10h
openebs-localpv-provisioner-554d878c7-hctxz   1/1     Running             0          10h
openebs-ndm-kl7hm                             0/1     ContainerCreating   0          10h
openebs-ndm-operator-58b8987557-znk4x         1/1     Running             1          10h
openebs-provisioner-64766cfd58-6ptgh          1/1     Running             0          10h
openebs-snapshot-operator-5565bbdc9c-rpbdj    2/2     Running             0          10h

The events from kubectl get events which are failing are

44s         Warning   FailedMount         pod/openebs-ndm-kl7hm                              MountVolume.SetUp failed for volume "udev" : hostPath type check failed: /run/udev is not a directory
7h30m       Warning   FailedMount         pod/openebs-ndm-kl7hm                              MountVolume.SetUp failed for volume "config" : configmap "openebs-ndm-config" not found
25m         Warning   FailedMount         pod/openebs-ndm-kl7hm                              Unable to attach or mount volumes: unmounted volumes=[config udev], unattached volumes=[config udev procmount sparsepath openebs-maya-operator-token-2khww]: timed out waiting for the condition
120m        Warning   FailedMount         pod/openebs-ndm-kl7hm                              Unable to attach or mount volumes: unmounted volumes=[config udev], unattached volumes=[sparsepath openebs-maya-operator-token-2khww config udev procmount]: timed out waiting for the condition
30m         Warning   FailedMount         pod/openebs-ndm-kl7hm                              Unable to attach or mount volumes: unmounted volumes=[config udev], unattached volumes=[openebs-maya-operator-token-2khww config udev procmount sparsepath]: timed out waiting for the condition
95m         Warning   FailedMount         pod/openebs-ndm-kl7hm                              Unable to attach or mount volumes: unmounted volumes=[udev config], unattached volumes=[udev procmount sparsepath openebs-maya-operator-token-2khww config]: timed out waiting for the condition
5m          Warning   FailedMount         pod/openebs-ndm-kl7hm                              Unable to attach or mount volumes: unmounted volumes=[config udev], unattached volumes=[procmount sparsepath openebs-maya-operator-token-2khww config udev]: timed out waiting for the condition
#4

Yeah, unfortunately the OpenEBS NDM component requires some additional directories to be mounted to the planet container and won’t work out of the box. Here are the changes we had to make in order to make it work if you’re interested: https://github.com/gravitational/planet/pull/529/files.

Some customers who I know use OpenEBS with Gravity do so without Node Device Manager component but that unfortunately means that it won’t be able to detect block devices so you’d have to create them yourself. However, if you’re looking to just use the local provisioner, that should work w/o a problem.

Thanks,
Roman