In company proxy env, Trusted-clusters - reversetunnel, Failed to create remote tunnel: ssh: handshake failed: read tcp OFFICE_IP:60354->AWS_IPv4_Public_IP:3024: i/o timeout

teleport
#1

Hi, Team,
There’s company proxy in our office network. When i set up a Teleport cluster to be trusted by a Teleport cluster(main) located in AWS N.V region, this error came up. Any tips? Thanks!

May 20 21:59:33 ubuntu teleport[12240]: WARN [PROXY:AGE] Failed to create remote tunnel: ssh: handshake failed: read tcp OFFICE_Private_IP:60354->AWS_IPv4_Public_IP:3024: i/o timeout, conn: <nil>. target:teleport.example.com:3024 reversetunnel/agent.go:448
May 20 21:59:33 ubuntu teleport[12240]: DEBU [PROXY:AGE] changing state connecting -> disconnected target:teleport.example.com:3024 reversetunnel/agent.go:199
May 20 21:59:37 ubuntu teleport[12240]: DEBU [PROXY:AGE] Pool is closing agent. target:teleport.example.com:3024 reversetunnel/agentpool.go:253
May 20 21:59:42 ubuntu teleport[12240]: DEBU [PROXY:AGE] Adding agent(connecting) -> teleport.example.com:teleport.example.com:3024. cluster:teleport.example reversetunnel/agentpool.go:309
May 20 21:59:42 ubuntu teleport[12240]: DEBU [PROXY:AGE] Outbound tunnel for teleport.example.com connected to 1 proxies. cluster:teleport.example reversetunnel/agentpool.go:341
May 20 21:59:42 ubuntu teleport[12240]: DEBU [PROXY:AGE] changing state connecting -> connecting target:teleport.example.com:3024 reversetunnel/agent.go:190
May 20 21:59:42 ubuntu teleport[12240]: DEBU [HTTP:PROX] No valid environment variables found. proxy/proxy.go:217
May 20 21:59:42 ubuntu teleport[12240]: DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:137
May 20 21:59:47 ubuntu teleport[12240]: DEBU [PROXY:AGE] Outbound tunnel for teleport.example.com connected to 1 proxies. cluster:teleport.example reversetunnel/agentpool.go:341
May 20 21:59:52 ubuntu teleport[12240]: DEBU [PROXY:AGE] Outbound tunnel for teleport.example.com connected to 1 proxies. cluster:teleport.example reversetunnel/agentpool.go:341
May 20 21:59:57 ubuntu teleport[12240]: DEBU [PROXY:AGE] Outbound tunnel for teleport.example.com connected to 1 proxies. cluster:teleport.example reversetunnel/agentpool.go:341
May 20 22:00:02 ubuntu teleport[12240]: DEBU [PROXY:AGE] Outbound tunnel for teleport.example.com connected to 1 proxies. cluster:teleport.example reversetunnel/agentpool.go:341
May 20 22:00:07 ubuntu teleport[12240]: DEBU [PROXY:AGE] Outbound tunnel for teleport.example.com connected to 1 proxies. cluster:teleport.example reversetunnel/agentpool.go:341
May 20 22:00:12 ubuntu teleport[12240]: DEBU [PROXY:AGE] Outbound tunnel for teleport.example.com connected to 1 proxies. cluster:teleport.example reversetunnel/agentpool.go:341

Btw,

From the Teleport Proxy host in office, i can

1). Manually setup the reverse tunnel with “ssh -R 60354:localhost:22 ubuntu@AWS_IPv4_Public_IP”, and “ssh ubuntu@localhost -p 60354” back from AWS Teleport cluster(main) succesfully

2). Check “nc -p 60354 -w 5 AWS_IPv4_Public_IP 3024 -v” successfully
Connection to AWS_IPv4_Public_IP 3024 port [tcp/*] succeeded!

From AWS side Teleport cluster log, I can see

“ERRO read tcp AWS_Private_IP:3024->OFFICE_WAN_IP:19993: i/o timeout sshutils/server.go:531”, the “OFFICE_WAN_IP” is the same as the result of “curl ifconfig.me” from office.
Is this expected? Should OFFICE_Private_IP instead of OFFICE_WAN_IP be presented here?

Did company firewall block the talk between Trusted clusters? or misconfiguration?

P.S. Two clusters have same versions

$ teleport version
Teleport v3.2.4 git:v3.2.4-0-g339827c6 go1.11.5

$ tsh version
Teleport v3.2.4 git:v3.2.4-0-g339827c6 go1.11.5

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.5 LTS
Release: 16.04
Codename: xenial

#2

Could it be a routing problem? E.g. teleport if bound to private IP could not route to public IP?

In your netcat you tried to use generic nc, but what if you try to bind to private IP, would it route?

nc -s OFFICE_Private_IP -p 60354 -w 5 AWS_IPv4_Public_IP 3024 -v
#3

Thanks! Just tried, got this

Connection to AWS_IPv4_Public_IP 3024 port [tcp/*] succeeded!

Btw, I only set/use OFFICE_Private_IP in office side cluster configuration.

#4

@sasha and team,

Here’re the content of /etc/teleport.yaml in Trusted clusters. Anything missing here? Thanks!

/etc/teleport.yaml of Teleport cluster located inside of company

/etc/teleport.yaml of Teleport cluster located in AWS N.V region

#5

In the end this turned out to be the HTTP_PROXY and HTTPS_PROXY environment variables not being set in Teleport’s environment.

The fix was to add these environment variables to the systemd unit file:

[Service]
Environment="HTTP_PROXY=http://proxy.example.com:8080/"
Environment="HTTPS_PROXY=http://proxy.example.com:8080/"
1 Like
#6

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.