Random installation failure

mtariq · September 11, 2019, 9:00am

Hi
Seeing random Gravity installation failure, with [ERROR]: context deadline exceeded, I am not sure what to look into?

Wed Sep 11 08:23:42 UTC Starting installer

To abort the installation and clean up the system,
press Ctrl+C two times in a row.

If the you get disconnected from the terminal, you can reconnect to the installer
agent by issuing ‘gravity resume’ command.

If the installation fails, use ‘gravity plan’ to inspect the state and
‘gravity resume’ to continue the operation.
See https://gravitational.com/gravity/docs/cluster/#managing-an-ongoing-operation for details.

Wed Sep 11 08:23:42 UTC Connecting to installer
[ERROR]: context deadline exceeded

From Gravity system logs, seeing following errors

2019-09-11T01:24:40-07:00 DEBU [PROC:1]    Service is completed and removed. service:gravity.listener utils/logginghook.go:56
2019-09-11T01:24:40-07:00 WARN             Failed to run. error:[
ERROR REPORT:
Original Error: *trace.ConnectionProblemError dial tcp 192.168.86.71:61009: connect: connection refused
Stack Trace:
        /gopath/src/github.com/gravitational/gravity/vendor/github.com/gravitational/teleport/lib/httplib/httplib.go:110 github.com/gravitational/gravity/vendor/github.com/gravitational/teleport/lib/httplib.ConvertResponse
        /gopath/src/github.com/gravitational/gravity/lib/ops/opsclient/opsclient.go:1563 github.com/gravitational/gravity/lib/ops/opsclient.(*Client).PostJSON
        /gopath/src/github.com/gravitational/gravity/lib/ops/opsclient/opsclient.go:205 github.com/gravitational/gravity/lib/ops/opsclient.(*Client).CreateInstallToken
        /gopath/src/github.com/gravitational/gravity/tool/gravity/cli/config.go:971 github.com/gravitational/gravity/tool/gravity/cli.generateInstallToken
        /gopath/src/github.com/gravitational/gravity/tool/gravity/cli/config.go:357 github.com/gravitational/gravity/tool/gravity/cli.(*InstallConfig).NewInstallerConfig
        /gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/config.go:109 github.com/gravitational/gravity/e/tool/gravity/cli.(*InstallConfig).NewInstallerConfig
        /gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/install.go:81 github.com/gravitational/gravity/e/tool/gravity/cli.startInstallFromService
        /gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/install.go:28 github.com/gravitational/gravity/e/tool/gravity/cli.startInstall
        /gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/run.go:88 github.com/gravitational/gravity/e/tool/gravity/cli.execute
        /gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/run.go:38 github.com/gravitational/gravity/e/tool/gravity/cli.Run
        /gopath/src/github.com/gravitational/gravity/e/tool/gravity/main.go:38 main.run
        /gopath/src/github.com/gravitational/gravity/e/tool/gravity/main.go:26 main.main
        /go/src/runtime/proc.go:201 runtime.main
        /go/src/runtime/asm_amd64.s:1333 runtime.goexit
User Message: Post https://192.168.86.71:61009/portal/v1/tokens/install: dial tcp 192.168.86.71:61009: connect: connection refused
] utils/logginghook.go:56
2019-09-11T01:24:41-07:00 DEBU [CLIENT:IN] Connect to installer service. addr:/home/user/.gravity/installer.sock error:[dial unix /home/user/.gravity/installer.sock: connect: no such file or directory] utils/logginghook.go:56
2019-09-11T01:24:42-07:00 WARN             Failed to run. error:[
ERROR REPORT:
Original Error: context.deadlineExceededError context deadline exceeded
Stack Trace:
        /gopath/src/github.com/gravitational/gravity/lib/install/proto/client.go:53 github.com/gravitational/gravity/lib/install/proto.NewClient
        /gopath/src/github.com/gravitational/gravity/lib/install/client/install.go:53 github.com/gravitational/gravity/lib/install/client.(*InstallerStrategy).connect
        /gopath/src/github.com/gravitational/gravity/lib/install/client/client.go:47 github.com/gravitational/gravity/lib/install/client.New
        /gopath/src/github.com/gravitational/gravity/tool/gravity/cli/install.go:818 github.com/gravitational/gravity/tool/gravity/cli.installerClient
        /gopath/src/github.com/gravitational/gravity/tool/gravity/cli/install.go:662 github.com/gravitational/gravity/tool/gravity/cli.InstallerClient
        /gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/install.go:38 github.com/gravitational/gravity/e/tool/gravity/cli.startInstall
        /gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/run.go:88 github.com/gravitational/gravity/e/tool/gravity/cli.execute
        /gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/run.go:38 github.com/gravitational/gravity/e/tool/gravity/cli.Run
        /gopath/src/github.com/gravitational/gravity/e/tool/gravity/main.go:38 main.run
        /gopath/src/github.com/gravitational/gravity/e/tool/gravity/main.go:26 main.main
        /go/src/runtime/proc.go:201 runtime.main
        /go/src/runtime/asm_amd64.s:1333 runtime.goexit
User Message: context deadline exceeded
] utils/logginghook.go:56
2019-09-11T01:24:46-07:00 DEBU             got search paths: [/var/lib/gravity assets/local] utils/logginghook.go:56
2019-09-11T01:24:46-07:00 DEBU             look up configs in /var/lib/gravity utils/logginghook.go:56
2019-09-11T01:24:46-07:00 DEBU             /var/lib/gravity/gravity.yaml not found in search path utils/logginghook.go:56
2019-09-11T01:24:46-07:00 DEBU             look up configs in assets/local utils/logginghook.go:56
2019-09-11T01:24:46-07:00 DEBU             assets/local/gravity.yaml not found in search path utils/logginghook.go:56
2019-09-11T01:24:46-07:00 DEBU [LOCAL]     Creating local environment. args:{ /var/lib/gravity/local false false false 0s 0s 0x23a54d0 {[] 0} false} utils/logginghook.go:56
..
..
..
2019-09-11T01:25:30-07:00 INFO [AGENT:RPC] Fetched cloud metadata. metadata:CloudMetadata(<empty>) provider:onprem utils/logginghook.go:56
2019-09-11T01:25:30-07:00 INFO [INSTALLER] Failed to query install operation. error:[
ERROR REPORT:
Original Error: *trace.BadParameterError expected 1 cluster, got: []
Stack Trace:
        /gopath/src/github.com/gravitational/gravity/lib/ops/utils.go:228 github.com/gravitational/gravity/lib/ops.GetWizardOperation
        /gopath/src/github.com/gravitational/gravity/lib/install/install.go:234 github.com/gravitational/gravity/lib/install.(*Installer).maybeStartAgent
        /gopath/src/github.com/gravitational/gravity/lib/install/install.go:73 github.com/gravitational/gravity/lib/install.New
        /gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/install.go:119 github.com/gravitational/gravity/e/tool/gravity/cli.newCLInstaller
        /gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/install.go:88 github.com/gravitational/gravity/e/tool/gravity/cli.startInstallFromService
        /gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/install.go:28 github.com/gravitational/gravity/e/tool/gravity/cli.startInstall
        /gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/run.go:88 github.com/gravitational/gravity/e/tool/gravity/cli.execute
        /gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/run.go:38 github.com/gravitational/gravity/e/tool/gravity/cli.Run
        /gopath/src/github.com/gravitational/gravity/e/tool/gravity/main.go:38 main.run
        /gopath/src/github.com/gravitational/gravity/e/tool/gravity/main.go:26 main.main
        /go/src/runtime/proc.go:201 runtime.main
        /go/src/runtime/asm_amd64.s:1333 runtime.goexit
User Message: expected 1 cluster, got: []
] utils/logginghook.go:56
2019-09-11T01:26:04-07:00 DEBU [KEYGEN]    generated user key for [root] with expiry on (1568226364) 2019-09-11 11:26:04.934541157 -0700 PDT m=+36064.668526804 utils/logginghook.go:56
2019-09-11T01:26:04-07:00 INFO [CA]        Generating TLS certificate {0x6065a68 0xc000c3a860 CN=opscenter@gravitational.io,O=@teleadmin+O=default-implicit-role,L=root 2019-09-11 18:26:04.945158481 +0000 UTC []}. common_name:opscenter@gravitational.io dns_names:[] locality:[root] not_after:2019-09-11 18:26:04.945158481 +0000 UTC org:[@teleadmin default-implicit-role] org_unit:[] utils/logginghook.go:56

s.antipov · September 13, 2019, 4:54pm

Hi @mtariq,

From the error above looks like it is from joining node and the node cannot connect to installer node. Probably because of firewall between them. All network ports requirements you can find here .

maaz · September 13, 2019, 5:17pm

Hi @s.antipov
sudo ./gravity install --advertise-addr=192.168.86.11 --token=secret --cluster="test-maaz" --cloud-provider=generic
I see above issue during installation on first node, I tried couple of times and second time it gave error on prefiight check that minimum CPUs count did not match, so my concerns is it’s failing without giving proper error message,
I played around with install script and it seems to fail if cluster-name have space or special chars in secrets
like
sudo ./gravity install --advertise-addr=192.168.86.71 --token=secret --cluster="test maaz" --cloud-provider=generic
should I file a GitHub issue?

s.antipov · September 13, 2019, 7:42pm

Yes please. With additional related logs from /var/log/gravity-system.log if you can.

abdu · September 18, 2019, 8:45pm

Hey @maaz, were you able to get past the error on preflight check? If not feel free to create a Github issue and we can get it resolved. Please do include the logs. Thanks!