Teleport behind Nginx Ingress on port 443 - node fails to connect via reverse tunnel

Hello Everybody,

I am attempting to run Teleport behind Nginx ingress, on port 443. Basically, both the web and the web interface are listening on the same port, except they now sit behind the Nginx Ingress Controller. AFAIK nginx does support WebSockets. Isn’t that what the reverse tunnel is established over?

Just for more context, please find a chunk of the client logs:

    INFO [PROC]      Generating new host UUID: ab488667-079f-4b3c-955c-3878b07d9103. service/service.go:554
DEBU [SQLITE]    Connected to: file:/var/lib/teleport/proc/sqlite.db?_busy_timeout=10000&_sync=OFF, poll stream period: 1s lite/lite.go:173
DEBU [SQLITE]    Synchronous: 0, busy timeout: 10000 lite/lite.go:218
DEBU [KEYGEN]    SSH cert authority started with no keys pre-compute. native/native.go:107
DEBU [PROC]      Adding service to supervisor. service:register.node service/supervisor.go:181
DEBU [PROC]      Adding service to supervisor. service:ssh.node service/supervisor.go:181
DEBU [PROC]      Adding service to supervisor. service:ssh.shutdown service/supervisor.go:181
DEBU [PROC]      Adding service to supervisor. service:common.rotate service/supervisor.go:181
DEBU [PROC:1]    Service has started. service:ssh.node service/supervisor.go:242
DEBU [PROC:1]    Service has started. service:ssh.shutdown service/supervisor.go:242
DEBU [PROC:1]    Service has started. service:common.rotate service/supervisor.go:242
DEBU [PROC:1]    No signal pipe to import, must be first Teleport process. service/service.go:709
DEBU [PROC:1]    Service has started. service:register.node service/supervisor.go:242
INFO [PROC:1]    Joining the cluster with a secure token. service/connect.go:349
DEBU [PROC:1]    Generating new key pair for Node first-time-connect. service/connect.go:256
DEBU [AUTH]      Attempting to register through auth server. auth/register.go:179
ERRO [AUTH]      Failed to register through auth server: invalid character '<' looking for beginning of value; falling back to trying the proxy server auth/register.go:126
DEBU [AUTH]      Attempting to register through proxy server. auth/register.go:143
DEBU [CLIENT]    HTTPS client init(proxyAddr=<teleport-ingress-hostname>:443, insecure=false) client/weblogin.go:295
DEBU [AUTH]      Successfully registered through proxy server. auth/register.go:132
DEBU [PROC:1]    Deleted generated key pair Node first-time-connect. service/connect.go:242
INFO [PROC]      Node has obtained credentials to connect to cluster. service/connect.go:377
DEBU [PROC]      Attempting to connect to Auth Server directly. service/connect.go:793
DEBU [PROC]      Attempting to connect to Auth Server through tunnel. service/connect.go:801
DEBU [CLIENT]    HTTPS client init(proxyAddr=<teleport-ingress-hostname>:443, insecure=false) client/weblogin.go:295
DEBU [PROC]      Discovered address for reverse tunnel server: <teleport-ingress-hostname>:443. service/connect.go:881
DEBU [HTTP:PROX] No valid environment variables found. proxy/proxy.go:222
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:137
ERRO [PROC:1]    Node failed to establish connection to cluster: ssh: handshake failed: ssh: overflow reading version string. time/sleep.go:149
INFO [PROC:1]    Joining the cluster with a secure token. service/connect.go:349
DEBU [PROC:1]    Generating new key pair for Node first-time-connect. service/connect.go:256
DEBU [AUTH]      Attempting to register through auth server. auth/register.go:179
ERRO [AUTH]      Failed to register through auth server: invalid character '<' looking for beginning of value; falling back to trying the proxy server auth/register.go:126

The reverse tunnel is established over SSH and runs on port 3024 by default. You can expose this port via a regular TCP load balancer in k8s and still have the web interface going via nginx Ingress.

Teleport’s web UI uses websockets to make its terminal connections, but nodes rely on SSH when connecting to the proxy.

Thank you for the answer! I thought it uses some sort of mechanism similar to Chisel (https://github.com/jpillora/chisel) for the reverse tunnel. That works behind the ingress just fine, and it leverages websockets for the full duplex communication channel. The main advantage with that IMHO is that it works well with proxies, and it also works well with services such as CloudFlare. Do you usually accept outside contributions? It would be cool to only. require a single outbound port opened… something as common as 443 / 80, what do you think?

Best Regards,

Alex.

That’s a cool project. I absolutely agree that websocket support in Teleport would be awesome.

We are always very happy to review PRs, thank you for considering!

We would like to switch to GRPC for our reverse tunnels:

This will allow us to support proxies like nginx better