Debug Gravity Application Installation failure

maaz · March 11, 2019, 8:47pm

I am trying to deploy by Gravity Build but it seems to have some failure, UI also gave me a crash report logs too, gravity plan showing all successful but looking /var/log/telekube-system.log I see issue in install hook. Can you guide what to look into for more details or reason of failure.

here are the logs

2019-03-11T19:28:48Z DEBU [FSM:INSTA] Applied StateChange(Phase=/app/my-appliance, State=in_progress). opid:c3de5c4d-b1a2-406e-8aaa-0402287ac138 install/hook.go:56
2019-03-11T19:28:48Z INFO Executing phase: /app/my-appliance. phase:/app/my-appliance install/hook.go:56
2019-03-11T19:28:48Z INFO Executing install hook for my-appliance:0.0.1. phase:/app/my-appliance install/hook.go:56
2019-03-11T19:28:48Z ERRO Phase execution failed: gravitational.io/my-appliance:0.0.1 install hook failed. phase:/app/my-appliance install/hook.go:56
2019-03-11T19:28:49Z DEBU [FSM:INSTA] Applied StateChange(Phase=/app/my-appliance, State=failed, Error=gravitational.io/my-appliance:0.0.1 install hook failed). opid:c3de5c4d-b1a2-406e-8aaa-0402287ac138 install/hook.go:56
2019-03-11T19:28:49Z ERRO [INSTALLER] “Failed to execute plan: \nERROR REPORT:\nOriginal Error: *errors.errorString error unmarshaling JSON: json: cannot unmarshal string into Go value of type v1.Job\nStack Trace:\n\t/gopath/src/github.com/gravitational/gravity/lib/schema/hooks.go:116 github.com/gravitational/gravity/lib/schema.Hook.GetJob\n\t/gopath/src/github.com/gravitational/gravity/lib/app/hooks/hooks.go:144 github.com/gravitational/gravity/lib/app/hooks.(*Runner).Start\n\t/gopath/src/github.com/gravitational/gravity/lib/app/service/app.go:311 github.com/gravitational/gravity/lib/app/service.(*applications).StartAppHook\n\t/gopath/src/github.com/gravitational/gravity/lib/app/apps.go:46 github.com/gravitational/gravity/lib/app.StreamAppHook\n\t/gopath/src/github.com/gravitational/gravity/lib/install/phases/app.go:129 github.com/gravitational/gravity/lib/install/phases.(*hookExecutor).runHooks\n\t/gopath/src/github.com/gravitational/gravity/lib/install/phases/app.go:88 github.com/gravitational/gravity/lib/install/phases.(*hookExecutor).Execute\n\t/gopath/src/github.com/gravitational/gravity/lib/fsm/fsm.go:417 github.com/gravitational/gravity/lib/fsm.(*FSM).executeOnePhase\n\t/gopath/src/github.com/gravitational/gravity/lib/fsm/fsm.go:351 github.com/gravitational/gravity/lib/fsm.(*FSM).executePhaseLocally\n\t/gopath/src/github.com/gravitational/gravity/lib/fsm/fsm.go:311 github.com/gravitational/gravity/lib/fsm.(*FSM).executePhase\n\t/gopath/src/github.com/gravitational/gravity/lib/fsm/fsm.go:188 github.com/gravitational/gravity/lib/fsm.(*FSM).ExecutePhase\n\t/gopath/src/github.com/gravitational/gravity/lib/fsm/fsm.go:362 github.com/gravitational/gravity/lib/fsm.(*FSM).executeSubphasesSequentially\n\t/gopath/src/github.com/gravitational/gravity/lib/fsm/fsm.go:356 github.com/gravitational/gravity/lib/fsm.(*FSM).executePhaseLocally\n\t/gopath/src/github.com/gravitational/gravity/lib/fsm/fsm.go:283 github.com/gravitational/gravity/lib/fsm.(*FSM).executePhase\n\t/gopath/src/github.com/gravitational/gravity/lib/fsm/fsm.go:188 github.com/gravitational/gravity/lib/fsm.(*FSM).ExecutePhase\n\t/gopath/src/github.com/gravitational/gravity/lib/fsm/fsm.go:146 github.com/gravitational/gravity/lib/fsm.(*FSM).ExecutePlan\n\t/gopath/src/github.com/gravitational/gravity/lib/install/flow.go:336 github.com/gravitational/gravity/lib/install.(*Installer).startFSM\n\t/go/src/runtime/asm_amd64.s:2362 runtime.goexit\nUser Message: gravitational.io/my-appliance:0.0.1 install hook failed, failed to execute phase “/app”\n.” install/hook.go:56
2019-03-11T19:28:49Z INFO [OPS] ops.SetOperationStateRequest{State:“failed”, Progress:(*ops.ProgressEntry)(0xc420a74f00)} install/hook.go:56
2019-03-11T19:28:49Z DEBU [OPS] Created: ops.ProgressEntry{ID:"", SiteDomain:“stoicpare8560”, OperationID:“c3de5c4d-b1a2-406e-8aaa-0402287ac138”, Created:time.Time{wall:0x6eccb97, ext:63687929329, loc:(*time.Location)(nil)}, Completion:100, Step:9, State:“failed”, Message:“Operation failure: error unmarshaling JSON: json: cannot unmarshal string into Go value of type v1.Job”}. install/hook.go:56
2019-03-11T19:28:49Z DEBU [FSM:INSTA] Marked operation complete. opid:c3de5c4d-b1a2-406e-8aaa-0402287ac138 install/hook.go:56
2019-03-11T19:28:49Z INFO Operation failed. install/hook.go:56
2019-03-11T19:28:51Z DEBU [PROXY:SER] ping <- 192.168.86.157:54206 latency:242.735µs cluster:stoicpare8560 install/hook.go:56
2019-03-11T19:28:56Z DEBU [PROXY:SER] ping <- 192.168.86.157:54206 latency:391.699µs cluster:stoicpare8560 install/hook.go:56
2019-03-11T19:29:01Z DEBU [PROXY:SER] ping <- 192.168.86.157:54206 latency:268.051µs cluster:stoicpare8560 install/hook.go:56
2019-03-11T19:29:06Z DEBU [PROXY:SER] ping <- 192.168.86.157:54206 latency:282.305µs cluster:stoicpare8560 install/hook.go:56
2019-03-11T19:29:11Z DEBU [PROXY:SER] ping <- 192.168.86.157:54206 latency:508.534µs cluster:stoicpare8560 install/hook.go:56
2019-03-11T19:29:14Z DEBU [KEYGEN] generated user key for [root] with expiry on (1552368554) 2019-03-12 05:29:14.484180749 +0000 UTC m=+36363.153037053 install/hook.go:56
2019-03-11T19:29:14Z DEBU [AUDITLOG] EmitAuditEvent(user.login: map[user:opscenter@gravitational.io method:local]) install/hook.go:56
2019-03-11T19:29:14Z DEBU [TELEPORT] generated certificate for opscenter@gravitational.io install/hook.go:56
2019-03-11T19:29:15Z INFO [INSTALLER] Received “interrupt” signal. install/hook.go:56
2019-03-11T19:29:15Z INFO [INSTALLER] Shutdown. install/hook.go:56
2019-03-11T19:29:15Z DEBU Operation context closed. install/hook.go:56
2019-03-11T19:46:22Z DEBU got search paths: [/var/lib/gravity assets/local] install/hook.go:56
2019-03-11T19:46:22Z DEBU look up configs in /var/lib/gravity install/hook.go:56
2019-03-11T19:46:22Z DEBU /var/lib/gravity/gravity.yaml not found in search path install/hook.go:56
2019-03-11T19:46:22Z DEBU look up configs in assets/local install/hook.go:56
2019-03-11T19:46:22Z DEBU assets/local/gravity.yaml not found in search path install/hook.go:56

knisbet · March 11, 2019, 9:44pm

Can you please share what version of gravity you are using?

Also, can you please share the install hook (redact internal information if needed)? Looking at the stack trace, it looks like the install failed to parse the install hook job for some reason, but it’s not evident from the exception why it wasn’t able to do so. I’m also not sure, why it wasn’t detected when tele build built the installer, but having a version of the job would help isolate why it’s failing.

maaz · March 11, 2019, 9:53pm

@knisbet I am using 5.4.6 and now giving 5.5.0 a try shortly

tele version
Edition: enterprise
Version: 5.4.6
Git Commit: af617669d2d5d9ce55e45b05bfa7a1ef58f1d7cd

Job is

apiVersion: batch/v1
kind: Job
metadata:
  name: install
spec:
  template:
    metadata:
      name: install
    spec:
      restartPolicy: OnFailure
      containers:
        - name: install
          image: quay.io/gravitational/debian-tall:0.0.1
          command: ["/usr/local/bin/helm", "install", "/var/lib/gravity/resources/", "--set", "global.registry=leader.telekube.local:5000" ]

maaz · March 12, 2019, 3:40am

@knisbet, it is Fixed, I had app.yaml and helm charts in same directory, build was fine but during deployment I got error
error unmarshaling JSON: json: cannot unmarshal string into Go value of type v1.Job
I was trying to troubleshoot but no helpful logs, in the end moving all helm charts into a separate folder fixed it. updated command looks like

["/usr/local/bin/helm", “install”, “/var/lib/gravity/resources/folder”, “–set”, “global.registry=leader.telekube.local:5000” ]