Not able to remote shell to a node behind firewall

sgambient · September 11, 2019, 7:44pm

The remote was activated as per the documentation (e.g. Adding a node located behind NAT). But looks like the Teleport Proxy directly ssh into the remote node behind firewall.

What is the right command ?

sudo teleport start --debug --roles=node --token=xxx --ca-pin=sha256:xxx --auth-server=:3080 --insecure
sudo teleport start --debug --roles=node --token=xxx --ca-pin=sha256:xxx --auth-server=:3025 --insecure

sasha · August 29, 2019, 11:57pm

If you would like to connect the node behind the firewall you should use the first command - the one that connects to the proxy. When it comes to the error, can you paste some logs of the connection failure here?

sgambient · September 3, 2019, 10:40pm

Get this error on the node:

WARNING: You are using insecure connection to SSH proxy https://:3080
DEBU [PROC] Discovered address for reverse tunnel server: :3024. service/connect.go:883
DEBU [HTTP:PROX] No valid environment variables found. proxy/proxy.go:217
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:137
ERRO [PROC:1] Node failed to establish connection to cluster: ssh: handshake failed: no matching keys found. service/connect.go:65
^CINFO [PROC:1] Got signal “interrupt”, exiting immediately. service/signals.go:87

Get this error on the auth,proxy server:

019/09/03 22:39:56 http: TLS handshake error from :46564: remote error: tls: bad certificate

sasha · September 4, 2019, 11:46pm

It seems like you have some leftover state on the node, can you clean up the node /var/lib/teleport and try again?

sgambient · September 5, 2019, 12:56am

Ok, tried, but still not working. The following are the logs.

Node:
WARN [NODE:BEAT] Heartbeat failed all SubConns are in TransientFailure, latest connection error: . logrus/entry.go:188
WARN [NODE:BEAT] Heartbeat failed all SubConns are in TransientFailure, latest connection error: . logrus/entry.go:188
DEBU [PROXY:AGE] Outbound tunnel for f7107f67-9479-4954-b72e-1660ed914c36.remotes1vm2cluster connected to 1 proxies. cluster:remotes1vm2cluster reversetunnel/agentpool.go:412
WARN [PROC:1] Sync rotation state cycle failed: all SubConns are in TransientFailure, latest connection error: , going to retry after 10s. logrus/entry.go:188
DEBU [PROXY:AGE] Outbound tunnel for f7107f67-9479-4954-b72e-1660ed914c36.remotes1vm2cluster connected to 1 proxies. cluster:remotes1vm2cluster reversetunnel/agentpool.go:412
WARN [NODE:BEAT] Heartbeat failed all SubConns are in TransientFailure, latest connection error: . logrus/entry.go:188
DEBU [PROXY:AGE] Outbound tunnel for f7107f67-9479-4954-b72e-1660ed914c36.remotes1vm2cluster connected to 1 proxies. cluster:remotes1vm2cluster reversetunnel/agentpool.go:412
WARN [NODE:BEAT] Heartbeat failed all SubConns are in TransientFailure, latest connection error: . logrus/entry.go:188
WARN [PROC:1] Sync rotation state cycle failed: all SubConns are in TransientFailure, latest connection error: , going to retry after 10s. logrus/entry.go:188
WARN [NODE:BEAT] Heartbeat failed all SubConns are in TransientFailure, latest connection error: . logrus/entry.go:188
DEBU [PROXY:AGE] Outbound tunnel for f7107f67-9479-4954-b72e-1660ed914c36.remotes1vm2cluster connected to 1 proxies. cluster:remotes1vm2cluster reversetunnel/agentpool.go:412
WARN [NODE:BEAT] Heartbeat failed all SubConns are in TransientFailure, latest connection error: . logrus/entry.go:188
DEBU [PROXY:AGE] Outbound tunnel for f7107f67-9479-4954-b72e-1660ed914c36.remotes1vm2cluster connected to 1 proxies. cluster:remotes1vm2cluster reversetunnel/agentpool.go:412
WARN [PROC:1] Sync rotation state cycle failed: all SubConns are in TransientFailure, latest connection error: , going to retry after 10s. logrus/entry.go:188
WARN [NODE:BEAT] Heartbeat failed all SubConns are in TransientFailure, latest connection error: . logrus/entry.go:188
DEBU [PROXY:AGE] Outbound tunnel for f7107f67-9479-4954-b72e-1660ed914c36.remotes1vm2cluster connected to 1 proxies. cluster:remotes1vm2cluster reversetunnel/agentpool.go:412

auth,proxy server:

WARN [NODE:BEAT] Heartbeat failed all SubConns are in TransientFailure, latest connection error: . logrus/entry.go:188
WARN [NODE:BEAT] Heartbeat failed all SubConns are in TransientFailure, latest connection error: . logrus/entry.go:188
WARN [NODE:1:CA] Re-init the cache on error: all SubConns are in TransientFailure, latest connection error: . logrus/entry.go:188
DEBU [NODE:1:CA] Reloading Linear(attempt=89, duration=10s). cache/cache.go:347
WARN [PROXY:1:C] Re-init the cache on error: all SubConns are in TransientFailure, latest connection error: . logrus/entry.go:188
DEBU [PROXY:1:C] Reloading Linear(attempt=89, duration=10s). cache/cache.go:347
WARN [PROXY:1] Re-init the watcher on error: all SubConns are in TransientFailure, latest connection error: . services/proxywatcher.go:180
DEBU [PROXY:1] Reloading Linear(attempt=89, duration=10s). services/proxywatcher.go:184
WARN [REVERSE:R] Re-init the cache on error: all SubConns are in TransientFailure, latest connection error: . logrus/entry.go:188
DEBU [REVERSE:R] Reloading Linear(attempt=89, duration=10s). cache/cache.go:347
WARN [NODE:BEAT] Heartbeat failed all SubConns are in TransientFailure, latest connection error: . logrus/entry.go:188
WARN [NODE:BEAT] Heartbeat failed all SubConns are in TransientFailure, latest connection error: . logrus/entry.go:188

Claire · February 5, 2020, 10:24pm

Hey @sgambient! Are you still having this issue?