Hostpath issues during Gravity upgrade

gravity
#1

We have few files and jars including JDBC copied to a host path that are read by few pods, on upgrade to 6.0.1 those things are not available in new deployment, luckily I saw old installation is not deleted so I manually copied them from the old path to the new one,
how can I avoid these cases or what is the best way to deal with upgrades?

cp /var/lib/gravity/local/packages/unpacked/gravitational.io/planet/5.5.20-11306/rootfs/mnt/hostpath/* /var/lib/gravity/local/packages/unpacked/gravitational.io/planet/6.0.6-11402/rootfs/mnt/hostpath/

#2

@mtariq Please take a look at the application manifest docs, specifically for a nodeProfile to create a volume mount. https://gravitational.com/gravity/docs/pack/#image-manifest

The nodeProfiles.volumes section will create a mount point between the host system and planet which is where we run kubelet. It can also define a number of install time requirements. Pod hostpaths can then use this volume for persistent data. The only mount we consistently offer by default is /var/lib/gravity/planet/share -> /ext/share.

#3

thanks will give it a try.

#4

This seems did not work, I do not see anything on host system but I do see data here /var/lib/gravity/local/packages/unpacked/gravitational.io/planet/6.0.6-11402/rootfs/mnt/, I used following configuration to mount, Am I missing something?

nodeProfiles:

  • name: node
    description: “Generic Linux node”
    ######### gravity will validate that the nodes have the requested amounts
    ######### of RAM/CPU
    requirements:
    cpu:
    min: 2

    ram:
    min: “2GB”

    volumes:
    # This directive tells the installer to ensure that /mnt directory
    # exists & created with 50GB of space:
    - name: app-data
    path: /mnt
    targetPath: /mnt
    capacity: “50GB”
    createIfMissing: true
    # UID and GID set linux UID and GID on the directory if specified
    uid: 114
    gid: 114
    # Unix file permissions mode to set on the directory
    mode: “0777”

#5

Hmm, based on the information presented, I’m not sure why this mapping didn’t work. The configuration appears to be correct as far as I can tell.

After changing the application manifest what happened next? The application was rebuilt? Was the existing node reinstalled with the new installer, or upgraded? Are there any other node profiles in your manifest, that could accidentally be getting activated? I’m not sure what it might be, but it’s probably worth double checking any assumptions.

#6

If it helps, here is our node profile:

nodeProfiles:
  - name: master
    description: "Generic Linux Master Node"
    labels:
      node-role.kubernetes.io/master: "true"
    requirements:
      volumes:
        - name: foo-logs
          path: /var/log/foo
          targetPath: /var/log/foo
          createIfMissing: true
          recursive: true
          capacity: "15GB"
          mode: "0755"
        - name: foo-lib
          path: /var/lib/foo
          targetPath: /var/lib/foo
          createIfMissing: true
          recursive: true
          capacity: "15GB"
          mode: "0755"

And then the volume definitions in a pod spec:

      volumes:
      - name: foo-lib
        hostPath:
          path: /var/lib/foo
          type: DirectoryOrCreate
      - name: foo-log
        hostPath:
          path: /var/log/foo
          type: DirectoryOrCreate

and the volume mounts (in the same pod spec):

        volumeMounts:
        - mountPath: /var/lib/foo
          name: foo-lib
        - mountPath: /var/log/foo
          name: foo-log

Note we use DirectoryOrCreate because we also have the same volumes defined on non-Gravity clusters where we can’t depend on them being created ahead of time. It works just fine on Gravity with just Directory.

#7

The appliance was rebuild, there is only one profile, we upgraded it instead of reinstalling, I can try reinstalling it if it changes anything?

#8

It is suppose to apply any new requirements through the upgrade.

The only thing I can think of, is the mounts are done through the planet configuration, so if the upgrade didn’t change the version of planet, it may not have regenerated the planet configuration and restarted planet during upgrade. If that’s the case I would consider this a bug in the upgrade process in relation to planet configuration changes. It’s also possible it did regenerate the config but then didn’t restart planet, which would only require restarting planet on the node for the change to take effect.

I’m not sure which it is without digging through the sources. If you confirm it does work on a fresh install or after restarting planet, I’ll port this issue to the github bug tracker.

#9

it worked on a Fresh install so likely issue is with the upgrade only, I tried to kill planet as well as rebooted the machine but could not see /mnt on upgraded node.