Deploy opscenter locally

maaz · March 1, 2019, 4:16pm

I wanted to deploy OpsCenter locally to evaluate it but could not find a way, is there a way I can deploy it in-house (relax OIDC, DNS, and SSL requirement), Right now I have One Mattermost Cluster and One OpsCenter deployed locally, I can login to gravity site on port 3009, I also have the tele license but not sure what to do next? I wanted to test all enterprise features like the remote update, ssh etc.

I have followed this page https://gravitational.com/gravity/docs/opscenter/

sasha · March 1, 2019, 10:41pm

Unfortunately right now ops center deployment requires all the setup with DNS, OIDC and SSL.

maaz · March 3, 2019, 3:00am

@sasha I tried setting up opscenter on AWS node, and configure DNS, OIDC and SSL but nothing seems to be loading, I put --ops-advertise-addr=server:443 but nothing seems to be listening on 443 and even gravity-site:3009 stop working and I thought may be issue with endpoints config but there seems to be no endpoints definition, am I missing something?

Gravity Installed using below command
sudo ./gravity install --debug --advertise-addr=serverip --token=token --flavor=standalone --cluster=clustername --ops-advertise-addr=server:443 --cloud-provider=generic

Trying to access portal

curl -k https://localhost:3009
curl: (16) SSL_write() returned SYSCALL, errno = 32

curl -k https://localhost:443
curl: (7) Failed to connect to localhost port 443: Connection refused`

Endpoint resource also did not work

    2019-03-03T02:54:52Z DEBU             got search paths: [/var/lib/gravity assets/local] processconfig/config.go:47
2019-03-03T02:54:52Z DEBU             look up configs in /var/lib/gravity processconfig/config.go:60
2019-03-03T02:54:52Z DEBU             /var/lib/gravity/gravity.yaml not found in search path processconfig/config.go:67
2019-03-03T02:54:52Z DEBU             look up configs in assets/local processconfig/config.go:60
2019-03-03T02:54:52Z DEBU             assets/local/gravity.yaml not found in search path processconfig/config.go:67
2019-03-03T02:54:52Z DEBU [LOCAL]     Creating local env: localenv.LocalEnvironmentArgs{LocalKeyStoreDir:"", StateDir:"/var/lib/gravity/local", Insecure:false, Silent:false, Debug:true, EtcdRetryTimeout:0, Reporter:(pack.ProgressReporterFn)(0x1d18c60), DNS:localenv.DNSConfig{Addrs:[]string(nil), Port:0}}. localenv/localenv.go:148
2019-03-03T02:54:52Z DEBU             dialing gravity-site.kube-system.svc.cluster.local:3009 httplib/client.go:216
2019-03-03T02:54:52Z DEBU             Resolve gravity-site.kube-system.svc.cluster.local took 165.78µs. utils/dns.go:39
2019-03-03T02:54:52Z DEBU             Resolved gravity-site.kube-system.svc.cluster.local to 10.100.197.55. utils/dns.go:52
2019-03-03T02:54:52Z DEBU             dialing 10.100.197.55:3009 httplib/client.go:253
2019-03-03T02:54:52Z ERRO             "\nERROR REPORT:\nOriginal Error: *trace.NotImplementedError unsupported resource \"endpoints\", supported are: [cluster_auth_preference github auth_connector user token logforwarder smtp alert alerttarget tlskeypair runtimeenvironment role oidc saml trusted_cluster endpoints]\nStack Trace:\n\t/gopath/src/github.com/gravitational/gravity/lib/ops/resources/gravity/gravity.go:214 github.com/gravitational/gravity/lib/ops/resources/gravity.(*Resources).Create\n\t/gopath/src/github.com/gravitational/gravity/e/lib/ops/resources/gravity/gravity.go:116 github.com/gravitational/gravity/e/lib/ops/resources/gravity.(*Resources).Create\n\t/gopath/src/github.com/gravitational/gravity/lib/ops/resources/resources.go:167 github.com/gravitational/gravity/lib/ops/resources.(*ResourceControl).Create\n\t/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/resources.go:59 github.com/gravitational/gravity/e/tool/gravity/cli.createResource.func1\n\t/gopath/src/github.com/gravitational/gravity/lib/ops/resources/resources.go:253 github.com/gravitational/gravity/lib/ops/resources.ForEach\n\t/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/resources.go:49 github.com/gravitational/gravity/e/tool/gravity/cli.createResource\n\t/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/run.go:135 github.com/gravitational/gravity/e/tool/gravity/cli.execute\n\t/gopath/src/github.com/gravitational/gravity/e/tool/gravity/cli/run.go:39 github.com/gravitational/gravity/e/tool/gravity/cli.Run\n\t/gopath/src/github.com/gravitational/gravity/e/tool/gravity/main.go:33 main.run\n\t/gopath/src/github.com/gravitational/gravity/e/tool/gravity/main.go:24 main.main\n\t/go/src/runtime/proc.go:207 runtime.main\n\t/go/src/runtime/asm_amd64.s:2362 runtime.goexit\nUser Message: unsupported resource \"endpoints\", supported are: [cluster_auth_preference github auth_connector user token logforwarder smtp alert alerttarget tlskeypair runtimeenvironment role oidc saml trusted_cluster endpoints]\n" gravity/main.go:24
[ERROR]: unsupported resource "endpoints", supported are: [cluster_auth_preference github auth_connector user token logforwarder smtp alert alerttarget tlskeypair runtimeenvironment role oidc saml trusted_cluster endpoints]`

Dmitri_Shelenin · March 4, 2019, 8:29am

@maaz Which version of Gravity are you using?

knisbet · March 4, 2019, 3:10pm

In regards to:

curl -k https://localhost:3009

curl: (16) SSL_write() returned SYSCALL, errno = 32

Try curl with --http1.1 option. IIRC there is a bug in curl (technically a dependency) that causes it to select the wrong ciphers when doing an http2 connection. I believe I’ve fixed this by forcing server side server selection in a recent pr, but it’s likely not fixed in the version you are using.

r0mant · March 4, 2019, 5:52pm

Hi @maaz!

Let me try to address your questions in order.

When installing on AWS with “generic” cloud provider, you need to configure the access to you Ops Center using AWS control panel by either creating a load balancer or opening up respective ports in your instance’s security group. Take a look at the gravity-public Kubernetes service in the kube-system namespace to see which ports are needed for the Ops Center to function properly. The “–ops-advertise-addr” parameter does not actually configure access, it basically just tells the Ops Center that it will be made available on this host/port so Ops Center will advertise this URL to the clients.

Next, it is actually possible to deploy an Ops Center to play with locally albeit with a few tricks. The way I do it, is I install it in a VM and then configure my /etc/hosts to point its advertise hostname to the VMs IP, for example:

$ ./gravity install ... --ops-advertise-addr=example.gravitational.io:32009
$ cat /etc/hosts
<vm-ip> example.gravitational.io

I also configure my local DNS so other VMs can “see” the Ops Center at that hostname. You still need to apply a proper certificate to the Ops Center though. The OIDC requirement can actually be relaxed by configuring a local-only authentication and creating a local user. Take a look at the Cluster Authentication Preference docs to see how to enable local auth (and Configuring Users to see how to create a local user).

Finally, the issue with creating the endpoints resource may be a bug as it should be supported if you’re using an enterprise version of gravity. Could you please run gravity version like Dmitry suggested above and show us the output so we can investigate?

Thanks,
Roman

maaz · March 5, 2019, 1:31am

@r0mant thanks for the answering it, it worked once I used 3009 port during installation, but I lose gravity-site access now (like monitoring/logs etc), I am confused with 32009 port in your comment as I assume it should be 3009 if we are not using AWS load balancer?

below is the gravity-site service ports.

Before (443 was used)

NAMESPACE     NAME             TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                                       AGE
default       kubernetes       ClusterIP      10.100.0.1       &lt;none&gt;        443/TCP                                       44h
kube-system   bandwagon        ClusterIP      10.100.234.154   &lt;none&gt;        80/TCP                                        44h
kube-system   gravity-public   LoadBalancer   10.100.121.87    &lt;pending&gt;     443:30220/TCP,3024:30069/TCP,3023:31965/TCP   44h
kube-system   gravity-site     LoadBalancer   10.100.197.55    &lt;pending&gt;     3009:32009/TCP                                44h

After (3009 is use)

NAMESPACE     NAME             TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                                        AGE
default       kubernetes       ClusterIP      10.100.0.1       &lt;none&gt;        443/TCP                                        5h38m
kube-system   bandwagon        ClusterIP      10.100.62.249    &lt;none&gt;        80/TCP                                         5h37m
kube-system   gravity-public   LoadBalancer   10.100.120.220   &lt;pending&gt;     3009:32302/TCP,3024:30857/TCP,3023:32388/TCP   5h38m
kube-system   gravity-site     LoadBalancer   10.100.59.224    &lt;pending&gt;     3009:32009/TCP                                 5h36m

also below is the gravity version @r0mant @Dmitri_Shelenin

gravity version
Edition: enterprise
Version: 5.4.6
Git Commit: af617669d2d5d9ce55e45b05bfa7a1ef58f1d7cd

r0mant · March 6, 2019, 5:37pm

I’d say it is better to use port 32009 because this is a Kubernetes service port so it always points to the currently active gravity-site process (there is always only 1 active) while by using 3009 you’re talking directly to the process running on the respective machine.

Thank you for letting us know your gravity version, I believe we’ve identified the issue with resource creation and working on a fix.

Thanks,
Roman