Teleport deployment on OpenShift

balpert89 · July 24, 2019, 4:19pm

Hi guys,

I am in the process of deploying teleport (auth & proxy) to an OpenShift cluster. The auth component is running happily and I am also able to join a node to the cluster. However, when I do a tctl nodes ls I cannot see the node:

INFO [AUTH]      Node "aott006-mst-01" [11324dc1-d9a1-4f3c-bb70-8ecd50d188d4] is trying to join with role: Node. auth/auth.go:1093
INFO [CA]        Generating TLS certificate {0x35076a0 0xc001050e00 CN=11324dc1-d9a1-4f3c-bb70-8ecd50d188d4.teleport-poc,O=Node 2029-07-21 15:55:09.775312835 +0000 UTC [aott006-mst-01]}. common_name:11324dc1-d9a1-4f3c-bb70-8ecd50d188d4.teleport-poc dns_names:[aott006-mst-01] locality:[] not_after:2029-07-21 15:55:09.775312835 +0000 UTC org:[Node] org_unit:[] tlsca/ca.go:176
INFO [AUTH]      Node "aott006-mst-01" [11324dc1-d9a1-4f3c-bb70-8ecd50d188d4] has joined the cluster. auth/auth.go:1126

But no nodes are showing up:

# tctl nodes ls
Nodename UUID Address Labels
-------- ---- ------- ------

My assumption is the proxy is botched, as it is constantly logging:

WARN [NODE:BEAT] Heartbeat failed all SubConns are in TransientFailure, latest connection error: <nil>. logrus/entry.go:188
WARN [NODE:2:CA] Re-init the cache on error: all SubConns are in TransientFailure, latest connection error: <nil>. logrus/entry.go:188
WARN [PROXY:2]   Re-init the watcher on error: all SubConns are in TransientFailure, latest connection error: <nil>. services/proxywatcher.go:180
WARN [PROXY:2:C] Re-init the cache on error: all SubConns are in TransientFailure, latest connection error: <nil>. logrus/entry.go:188
WARN [REVERSE:T] Re-init the cache on error: all SubConns are in TransientFailure, latest connection error: <nil>. logrus/entry.go:188
WARN [NODE:BEAT] Heartbeat failed all SubConns are in TransientFailure, latest connection error: <nil>. logrus/entry.go:188
WARN [PROC:2]    Sync rotation state cycle failed: all SubConns are in TransientFailure, latest connection error: <nil>, going to retry after 10s. logrus/entry.go:188

Edit: I can enter the web ui and login with my created user. There is also no nodes shown.

Here is a gist of both configurations I am using. Turning Debug level on does not help.

gist.github.com

https://gist.github.com/balpert89/861d44ac29f92e64ea9499e0cbfddbdd

teleport-auth.yaml

# By default, this file should be stored in /etc/teleport.yaml

# This section of the configuration file applies to all teleport
# services.
teleport:
    # Data directory where Teleport daemon keeps its data.
    # See "Filesystem Layout" section above for more details.
    data_dir: /var/lib/teleport

    # Invitation token used to join a cluster. it is not used on

This file has been truncated. show original

teleport-proxy.yaml

# By default, this file should be stored in /etc/teleport.yaml

# This section of the configuration file applies to all teleport
# services.
teleport:
    # Data directory where Teleport daemon keeps its data.
    # See "Filesystem Layout" section above for more details.
    data_dir: /var/lib/teleport

    # Invitation token used to join a cluster. it is not used on

This file has been truncated. show original

I hope someone can direct me to the mistake I am making

Best

gus · July 24, 2019, 7:59pm

I note that your ports in the config are set to 32025, 32023 etc but your listen addresses are still configured to the defaults (3025, 3023) - presumably you’re putting these behind a load balancer or some equivalent on OpenShift? At it stands, it doesn’t look like the config would work because of this.

balpert89 · July 24, 2019, 8:11pm

Heyo gus!

Yep the ports are mapped. The 32xxx are node ports that are available as listeners on my Load Balancer. Those are mapped to the local (30xx) ports of the pod.

I have updated the gist as I have managed to get a step further, I can now login with my user:

[root@aott006-mst-01 ~]# tsh --proxy=teleport-proxy.test.otc.appagile:32080 login --user=balpert
Enter password for Teleport user balpert:
Enter your OTP token:
824440
> Profile URL:  https://teleport-proxy.test.otc.appagile:32080
  Logged in as: balpert
  Cluster:      teleport-proxy.test.otc.appagile
  Roles:        admin*
  Logins:       admin
  Valid until:  2019-07-25 08:10:01 +0000 UTC [valid for 12h0m0s]
  Extensions:   permit-agent-forwarding, permit-port-forwarding, permit-pty


* RBAC is only available in Teleport Enterprise
  https://gravitational.com/teleport/docs/enterprise

This means the proxy “works”, but as it stands I still cant see any nodes:

[root@aott006-mst-01 ~]# tsh --proxy teleport-proxy.test.otc.appagile:32080 clusters
Cluster Name Status
------------ ------
teleport-poc online

[root@aott006-mst-01 ~]# tsh --proxy teleport-proxy.test.otc.appagile:32080 ls
Node Name Address Labels
--------- ------- ------

Best
balpert

gus · July 24, 2019, 8:22pm

Good to know that you’ve got further. Could you post the logs from the Teleport node? If there’s insufficient detail, it might be worth running the process with teleport start --debug and seeing whether it connects to the auth server correctly or not. My guess is that there’s something not quite right there at the moment.

balpert89 · July 24, 2019, 8:40pm

I have done that, added the logs of the proxy (clean start) to the gist. There is an authentication attempt with tsh at the end and in the middle.

Best
balpert

gus · July 25, 2019, 8:54pm

Looking at the config files you posted, it doesn’t look like you’re actually running the node service anywhere. You have auth_service: yes in your auth config and proxy_service: yes in your proxy config but no mention of ssh_service (other than to disable it in the auth config)

Assuming you want to be able to tsh ssh into the proxy pod, try setting ssh_service: yes in the proxy server’s config file and restarting Teleport.

(I understand that the config example says the node service is enabled by default, but never hurts to be completely explicit)

balpert89 · July 27, 2019, 5:01pm

Hm changing one or both configs to also activate the node service doesn’t change it. Also, installing teleport node service on a freshly deployed VM doesnt make that VM appear in the ls command, although I see the successful registration message in the auth service Is there something I do wrong?

balpert89 · July 28, 2019, 12:27pm

Ok I have tinkered alot with the configs and now I can see actual nodes in tctl nodes ls

However, the discovery uses the IP Address of the service inside the cluster:

# tctl nodes ls
Nodename               UUID                                 Address        Labels
---------------------- ------------------------------------ -------------- ------
aott006-mst-01         3588815f-11cb-42d1-ac23-841fa3dc2d04 10.1.0.1:3022
teleport-proxy-3-ns6c2 a6550237-48a5-479d-a1b1-201c8a48b8af 10.1.10.1:3022

Is it possible to somehow force an arbitrary IP address (like the host name)?

gus · July 29, 2019, 6:48pm

Good news!

You should be able to do this with advertise_ip in the Teleport config file:

teleport
  advertise_ip: 10.1.0.5

balpert89 · July 29, 2019, 9:14pm

Nice, thank you.

Okay one last question (maybe ): where do I find a complete documentation about what I can put into the teleport.yaml config file? The Admin Guide includes a config file, but are there additional parameters that can be put into it?

gus · July 29, 2019, 9:16pm

I think the listing at https://gravitational.co/teleport/docs/admin-guide/#configuration-file (as you mentioned) is pretty comprehensive. You can run teleport configure to output an example configuration file if you desire, but I’m pretty sure the one in the admin guide is more detailed.