Bootstrap failing on upgrade

gravity
#1

I am running a test to upgrade 1 node cluster from 6.0.9 to 6.1.8 and running into Bootstrap failure, resulting upgrade failed, do we have any clue what bootstrapping does and why it is failing?

Phase               Description                                     State         Node             Requires                          Updated
-----               -----------                                     -----         ----             --------                          ------- 
* init              Initialize update operation                     Completed     10.201.64.74     -                                 Wed Feb 19 18:11 UTC
* checks            Run preflight checks                            Completed     -                /init                             Wed Feb 19 18:11 UTC
* pre-update        Run pre-update application hook                 Completed     -                /init                             Wed Feb 19 18:11 UTC
* bootstrap         Bootstrap update operation on nodes             Failed        -                /init                             Wed Feb 19 18:11 UTC
  * node1           Bootstrap node "node1"                          Failed        10.201.64.74     -                                 Wed Feb 19 18:11 UTC
* masters           Update master nodes                             Unstarted     -                /checks,/bootstrap,/pre-update    -
#2

On looking running of gravity plan resume it errors out with

./gravity plan resume
Executing "/bootstrap/node1" locally
Fri Feb 21 20:28:14 UTC Operation failure: exit status 1
operation(update(b3052d6c-96c9-4582-bfa1-3cd532df543a), cluster=pricelessarchimedes8132, created=2020-02-21 19:57) finished in 1 second 
[ERROR]: failed to execute phase "/bootstrap"
        failed to run {[chown -R 1000:1000 /var/lib/gravity/local] setting ownership of /var/lib/gravity/local to 1000:1000 false}:

I tried running this command chown -R 1000:1000 /var/lib/gravity/local and run into Read-only file system error not sure any processing is not allowing it or what, I am running it as root user.

chown: changing ownership of ‘/var/lib/gravity/local/packages/unpacked/gravitational.io/planet/6.0.12-XXXX/rootfs/tmp/journal/833e0926ee21aed71ec075d726cbcfe0/system@3ddd0cf87a1d4bd883f0396b1184cae4-000000000974bc2b-00059f1960064e0c.journal’: Read-only file system
chown: changing ownership of ‘/var/lib/gravity/local/packages/unpacked/gravitational.io/planet/6.0.12-XXXX/rootfs/tmp/journal/833e0926ee21aed71ec075d726cbcfe0’: Read-only file system
chown: changing ownership of ‘/var/lib/gravity/local/packages/unpacked/gravitational.io/planet/6.0.12-XXXX/rootfs/tmp/journal’: Read-only file system
#3

Hi @mtariq,

How was the upgrade started? Via the upgrade script or any other way? And was it just initiated once?

Can you share the output of the following command at the failure point:

sudo gravity package list --insecure --ops-url=https://gravity-site.kube-system.svc.cluster.local:3009

I am wondering if you are running into a bootstrap issue we have seen before, where bootstrap pulls update packages just by selecting the package with the highest available version - if an attempt to upgrade has happened prior, it ends up pulling the wrong versions of the packages so bootstrap fails with the message when it does not find the correct package.

#4

sorry for late reply, it was started with script ./upgrade and we tried few times too my doing gravity plan complete. the way we recovered is by doing a node restart.