Gravity Upgrade Failing

I am running a gravity cluster upgrade. Upgrading from version. 6.1.29 to 6.1.41. Seems like it is stuck in the upgrade.

Cluster name: pensiveblack5716
Cluster status: updating
Application: test, version 1.29.0-11rc
Gravity version: 6.1.41 (client) / 6.1.29 (server)
Join token: 8sCZ8H891n
Periodic updates: Not Configured
Remote support: Not Configured
Last completed operation:
* Upgrade to version test
ID: 5e1495fb-12a4-485a-bcf1-4d36b52f22c2
Started: Thu Oct 15 21:11 UTC (1 hour ago)
Failed: Thu Oct 15 21:35 UTC (1 hour ago)
Cluster endpoints:
* Authentication gateway:
- 10.151.229.186:32009
* Cluster management URL:
- https://10.151.229.186:32009
Cluster nodes:
Masters:
* ip-10-151-229-186 / 10.151.229.186 / node
Status: healthy
Remote access: online

gravity-system.log;

ERROR REPORT:
Original Error: *trace.CompareFailedError Upgrade operation can only be triggered for active clusters. This cluster is currently updating.
Use "gravity status" to see the cluster status and make sure that the cluster is active and healthy before retrying.
Stack Trace:
/gopath/src/github.com/gravitational/gravity/lib/ops/opsservice/operationgroup.go:173 github.com/gravitational/gravity/lib/ops/opsservice.(*operationGroup).canCreateUpgradeOperation
/gopath/src/github.com/gravitational/gravity/lib/ops/opsservice/operationgroup.go:147 github.com/gravitational/gravity/lib/ops/opsservice.(*operationGroup).canCreateOperation
/gopath/src/github.com/gravitational/gravity/lib/ops/opsservice/operationgroup.go:80 github.com/gravitational/gravity/lib/ops/opsservice.(*operationGroup).createSiteOperationWithOptions
/gopath/src/github.com/gravitational/gravity/lib/ops/opsservice/update.go:376 github.com/gravitational/gravity/lib/ops/opsservice.(*site).createUpdateOperation
/gopath/src/github.com/gravitational/gravity/lib/ops/opsservice/service.go:1016 github.com/gravitational/gravity/lib/ops/opsservice.(*Operator).CreateSiteAppUpdateOperation
/gopath/src/github.com/gravitational/gravity/tool/gravity/cli/clusterupdate.go:459 github.com/gravitational/gravity/tool/gravity/cli.clusterInitializer.newOperation
/gopath/src/github.com/gravitational/gravity/tool/gravity/cli/update.go:66 github.com/gravitational/gravity/tool/gravity/cli.newUpdater
/gopath/src/github.com/gravitational/gravity/tool/gravity/cli/clusterupdate.go:103 github.com/gravitational/gravity/tool/gravity/cli.newClusterUpdater
/gopath/src/github.com/gravitational/gravity/tool/gravity/cli/clusterupdate.go:74 github.com/gravitational/gravity/tool/gravity/cli.updateTrigger
/gopath/src/github.com/gravitational/gravity/tool/gravity/cli/run.go:354 github.com/gravitational/gravity/tool/gravity/cli.Execute
/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/run.go:171 github.com/gravitational/gravity/e/tool/gravity/cli.execute
/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/run.go:51 github.com/gravitational/gravity/e/tool/gravity/cli.getExec.func1
/gopath/src/github.com/gravitational/gravity/tool/gravity/cli/logging.go:71 github.com/gravitational/gravity/tool/gravity/cli.(*CmdExecer).Execute
/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/run.go:46 github.com/gravitational/gravity/e/tool/gravity/cli.Run
/gopath/src/github.com/gravitational/gravity/e/tool/gravity/main.go:43 main.run
/gopath/src/github.com/gravitational/gravity/e/tool/gravity/main.go:27 main.main
/go/src/runtime/proc.go:200 runtime.main
/go/src/runtime/asm_amd64.s:1337 runtime.goexit

Hey @AhmedShahid, apologies for the delayed reply here.

If you are still running into this issue, have you tried looking at the gravity plan? You can then attempt gravity plan resume or execute the phase at which is stuck manually using gravity plan execute --phase=<phase>

If you can share gravity-install.log that would help as well.

Thanks

-Abdu

Also, it is important to use the upgrade package gravity binary for the upgrade operation - so make sure you’re prefixing the commands with ./ or use the full path to explicitly specify the binary.
Once triggered, the operation will run from a systemd unit either until completion or until it fails on any of the steps - to verify where the operation has paused, use the plan command:

./gravity plan [display]

Also, see this documentation section for more detail on working with operations.
If you’re still stuck - please share the relevant part of /var/log/gravity-system.log describing the issue you’re hitting together with the output of ./gravity plan so we can help.