X509: certificate signed by unknown authority error during adding a node

teleport
#1

hi,

My cluster (node, proxy, auth) is up and running on https://stage.advasmart.in. I can ssh to the node stage.advasmart.in from my laptop.

I am adding a new node and generated a node token on stage.advasmart.in

I am trying below on another node teleport.advarisk.com

sudo teleport start --roles=node --token=2xxxxxxxxxxxxxxxxxxxxxxxxf --auth-server=https://stage.advasmart.in --ca-pin=sha256:b235xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx243

But getting below error

x509: certificate signed by unknown authority

My yaml file on teleport.advarisk.com looks like -

teleport:
    nodename: teleport.advarisk.com
    data_dir: /var/lib/teleport
    auth_token: 2a544df661690f836ef79f889db2bf5f
    auth_servers:
      - stage.advasmart.in:3025
    ca_pin: "sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
auth_service:
    enabled: "no"
    cluster_name: "teleport-test"
    listen_addr: 0.0.0.0:3025
    tokens:
    - proxy,node,app:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
ssh_service:
    enabled: "yes"
    labels:
        env: staging
app_service:
    enabled: "no"
    debug_app: true
proxy_service:
    enabled: "no"
    listen_addr: 0.0.0.0:3023
    web_listen_addr: 0.0.0.0:3080
    tunnel_listen_addr: 0.0.0.0:3024
    public_addr: teleport.advarisk.com:3080
    https_keypairs:
      - key_file: /etc/letsencrypt/live/teleport.advarisk.com/privkey.pem
        cert_file: /etc/letsencrypt/live/teleport.advarisk.com/fullchain.pem

Any ideas what is going wrong here?

Regards,
Suraj

#2

Hi @surajmundada,

Is this a self-signed cert?

#3

Hi Alen,

Both the Certs are obtained from LetsEncrypt.

Suraj

#4

Any thoughts on how I can debug this?

Regards,
Suraj

#5

Suraj - Could you provide us with more context from the Teleport logs? We would need to see the full error message, and surrounding log lines, to debug this further.

#6

Hi Grav,

I am getting following logs on my proxy in /var/lib/teleport/teleport.log

WARN [MXTLS:2]   Handshake failed. error:remote error: tls: bad certificate multiplexer/tls.go:143
    [[AERRO [AUTH:2]    "Failed to retrieve client pool. Client cluster teleport-test, target cluster stage-teleport-cluster, error:  \nERROR REPORT:\nOriginal Error: *trace.NotFoundError key \"/authorities/host/teleport-test\" is not found\nStack Trace:\n\t/go/src/github.com/gravitational/teleport/lib/backend/memory/memory.go:186 github.com/gravitational/teleport/lib/backend/memory.(*Memory).Get\n\t/go/src/github.com/gravitational/teleport/lib/backend/report.go:159 github.com/gravitational/teleport/lib/backend.(*Reporter).Get\n\t/go/src/github.com/gravitational/teleport/lib/backend/wrap.go:89 github.com/gravitational/teleport/lib/backend.(*Wrapper).Get\n\t/go/src/github.com/gravitational/teleport/lib/services/local/trust.go:207 github.com/gravitational/teleport/lib/services/local.(*CA).GetCertAuthority\n\t/go/src/github.com/gravitational/teleport/lib/cache/cache.go:892 github.com/gravitational/teleport/lib/cache.(*Cache).GetCertAuthority\n\t/go/src/github.com/gravitational/teleport/lib/auth/middleware.go:546 github.com/gravitational/teleport/lib/auth.ClientCertPool\n\t/go/src/github.com/gravitational/teleport/lib/auth/middleware.go:253 github.com/gravitational/teleport/lib/auth.(*TLSServer).GetConfigForClient\n\t/opt/go/src/crypto/tls/handshake_server.go:141 crypto/tls.(*Conn).readClientHello\n\t/opt/go/src/crypto/tls/handshake_server.go:40 crypto/tls.(*Conn).serverHandshake\n\t/opt/go/src/crypto/tls/conn.go:1362 crypto/tls.(*Conn).Handshake\n\t/go/src/github.com/gravitational/teleport/lib/multiplexer/tls.go:141 github.com/gravitational/teleport/lib/multiplexer.(*TLSListener).detectAndForward\n\t/opt/go/src/runtime/asm_amd64.s:1375 runtime.goexit\nUser Message: key \"/authorities/host/teleport-test\" is not found\n." auth/middleware.go:261
WARN [MXTLS:2]   Handshake failed. error:remote error: tls: bad certificate multiplexer/tls.go:143
WARN [MXTLS:2]   Handshake failed. error:remote error: tls: bad certificate multiplexer/tls.go:143
ERRO [AUTH:2]    "Failed to retrieve client pool. Client cluster teleport-test, target cluster stage-teleport-cluster, error:  \nERROR REPORT:\nOriginal Error: *trace.NotFoundError key \"/authorities/host/teleport-test\" is not found\nStack Trace:\n\t/go/src/github.com/gravitational/teleport/lib/backend/memory/memory.go:186 github.com/gravitational/teleport/lib/backend/memory.(*Memory).Get\n\t/go/src/github.com/gravitational/teleport/lib/backend/report.go:159 github.com/gravitational/teleport/lib/backend.(*Reporter).Get\n\t/go/src/github.com/gravitational/teleport/lib/backend/wrap.go:89 github.com/gravitational/teleport/lib/backend.(*Wrapper).Get\n\t/go/src/github.com/gravitational/teleport/lib/services/local/trust.go:207 github.com/gravitational/teleport/lib/services/local.(*CA).GetCertAuthority\n\t/go/src/github.com/gravitational/teleport/lib/cache/cache.go:892 github.com/gravitational/teleport/lib/cache.(*Cache).GetCertAuthority\n\t/go/src/github.com/gravitational/teleport/lib/auth/middleware.go:546 github.com/gravitational/teleport/lib/auth.ClientCertPool\n\t/go/src/github.com/gravitational/teleport/lib/auth/middleware.go:253 github.com/gravitational/teleport/lib/auth.(*TLSServer).GetConfigForClient\n\t/opt/go/src/crypto/tls/handshake_server.go:141 crypto/tls.(*Conn).readClientHello\n\t/opt/go/src/crypto/tls/handshake_server.go:40 crypto/tls.(*Conn).serverHandshake\n\t/opt/go/src/crypto/tls/conn.go:1362 crypto/tls.(*Conn).Handshake\n\t/go/src/github.com/gravitational/teleport/lib/multiplexer/tls.go:141 github.com/gravitational/teleport/lib/multiplexer.(*TLSListener).detectAndForward\n\t/opt/go/src/runtime/asm_amd64.s:1375 runtime.goexit\nUser Message: key \"/authorities/host/teleport-test\" is not found\n." auth/middleware.go:261
WARN [MXTLS:2]   Handshake failed. error:remote error: tls: bad certificate multiplexer/tls.go:143
WARN [MXTLS:2]   Handshake failed. error:remote error: tls: bad certificate multiplexer/tls.go:143

teleport.yaml on proxy/auth (stage.advasmart.in)

teleport:
    nodename: stage.advasmart.in
    data_dir: /var/lib/teleport
    log:
      output: '/var/lib/teleport/teleport.log'
      severity: INFO
    storage:
        audit_events_uri:  ['file:///var/lib/teleport/audit/events', 'stdout://']
auth_service:
    enabled: "yes"
    cluster_name: "stage-teleport-cluster"
    listen_addr: 0.0.0.0:3025
    public_addr: stage.advasmart.in:3025
    tokens:
    - proxy,node,app:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    authentication:
    # default authentication type. possible values are 'local' and 'github' for OSS
    #  and 'oidc', 'saml' and 'false' for Enterprise.
    type: local
    # second_factor can be off, otp, or u2f
    second_factor: off

ssh_service:
    enabled: "yes"
    labels:
        env: staging

app_service:
    enabled: "yes"
    debug_app: true

proxy_service:
    enabled: "yes"
    listen_addr: 0.0.0.0:3023
    web_listen_addr: 0.0.0.0:3080
    tunnel_listen_addr: 0.0.0.0:3024
    public_addr: stage.advasmart.in:3080
    tunnel_public_addr: stage.advasmart.in:3024
    https_keypairs:
    - key_file: '/etc/letsencrypt/live/stage.advasmart.in/privkey.pem'
      cert_file: '/etc/letsencrypt/live/stage.advasmart.in/fullchain.pem'

teleport.yaml on node (teleport.advarisk.com)

teleport:
    nodename: teleport.advarisk.com
    data_dir: /var/lib/teleport
    auth_token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    auth_servers:
      - stage.advasmart.in:3025
    ca_pin:     "sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    log:
        output: '/var/lib/teleport/teleport.log'
        severity: INFO
    storage:
        audit_events_uri:  ['file:///var/lib/teleport/audit/events', 'stdout://']
auth_service:
    enabled: "no"
    cluster_name: "stage-teleport-cluster"
    listen_addr: 0.0.0.0:3025
    tokens:
    - proxy,node,app:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
ssh_service:
    enabled: "yes"
    labels:
        env: staging
app_service:
    enabled: "no"
    debug_app: true
proxy_service:
    enabled: "no"
    listen_addr: 0.0.0.0:3023
    web_listen_addr: 0.0.0.0:3080
    tunnel_listen_addr: 0.0.0.0:3024
    public_addr: teleport.advarisk.com:3080
    https_keypairs:
      - key_file: /etc/letsencrypt/live/teleport.advarisk.com/privkey.pem
        cert_file: /etc/letsencrypt/live/teleport.advarisk.com/fullchain.pem
#7

Thanks! That’s helpful. On your node, stop the Teleport service. Clear out the folder contents /var/lib/teleport/. Now, start Teleport. Does the node join the cluster successfully?

#8

Hi Grav,

Cleaning the teleport folder worked. Node “teleport.advarisk.com” got added to the cluster. But next problem is I can not tsh into node “teleport.advarisk.com” from my laptop. But tsh into “stage.advasmart.in” works properly.

I killed the teleport process on “teleport.advarisk.com” and started teleport node again. But it did not work.

On proxy -

root@stage:~# tctl nodes ls

Nodename              UUID                                 Address            Labels      

--------------------- ------------------------------------ ------------------ ----------- 

stage.advasmart.in    95c301cc-c001-4b0b-bdbc-a02b82c60d98 127.0.0.1:3022     env=staging 
teleport.advarisk.com adcb4591-43e3-4285-8753-c49da422ec08 3.238.130.234:3022 env=staging 
teleport.advarisk.com c45e3db3-2052-483a-b9b0-dd9340787fb0 3.238.130.234:3022 env=staging

Logs on Proxy/Auth -

INFO [AUTH]      Node "teleport.advarisk.com" [c45e3db3-2052-483a-b9b0-dd9340787fb0] is trying to join with role: Node. auth/auth.go:1313
INFO [CA]        Generating TLS certificate {0x40bc7f8 0xc000e1e010 1.3.9999.1.7=#131673746167652d74656c65706f72742d636c7573746572,CN=c45e3db3-2052-483a-b9b0-dd9340787fb0.stage-teleport-cluster,O=Node,POSTALCODE=null,STREET= 2030-12-22 13:33:21.942199634 +0000 UTC [teleport.advarisk.com c45e3db3-2052-483a-b9b0-dd9340787fb0]}. common_name:c45e3db3-2052-483a-b9b0-dd9340787fb0.stage-teleport-cluster dns_names:[teleport.advarisk.com c45e3db3-2052-483a-b9b0-dd9340787fb0] locality:[] not_after:2030-12-22 13:33:21.942199634 +0000 UTC org:[Node] org_unit:[] tlsca/ca.go:391
INFO [AUTH]      Node "teleport.advarisk.com" [c45e3db3-2052-483a-b9b0-dd9340787fb0] has joined the cluster. auth/auth.go:1346
tail: /var/lib/teleport/teleport.log: file truncated
INFO [AUDIT]     user.login code:T1000I ei:0 event:user.login method:local success:true time:2020-12-24T13:42:31.359Z uid:635bf355-19f0-4d31-a17e-e534c9f39ae4 user:suraj events/emitter.go:318
INFO [CA]        Generating TLS certificate {0x40bc7f8 0xc00170df40 1.3.9999.1.7=#131673746167652d74656c65706f72742d636c7573746572,1.3.9999.1.1=#1305737572616a,CN=suraj,O=admin,POSTALCODE={\"kubernetes_groups\":null\,\"kubernetes_users\":[\"suraj\"]\,\"logins\":[\"suraj\"\,\"root\"\,\"ubuntu\"]},STREET=stage-teleport-cluster,L=suraj+L=root+L=ubuntu 2020-12-25 01:42:31.367215812 +0000 UTC []}. common_name:suraj dns_names:[] locality:[suraj root ubuntu] not_after:2020-12-25 01:42:31.367215812 +0000 UTC org:[admin] org_unit:[] tlsca/ca.go:391
INFO [SUBSYSTEM] Connected to auth server: 172.16.113.86:3025 trace.fields:map[dst:101.53.142.168:3023 src:49.248.168.170:29955] regular/proxy.go:268
INFO [SUBSYSTEM] Connected to auth server: 172.16.113.86:3025 trace.fields:map[dst:101.53.142.168:3023 src:49.248.168.170:22132] regular/proxy.go:268
WARN [PROXY]     Subsystem request proxySubsys(cluster=default/stage-teleport-cluster, host=teleport.advarisk.com, port=0) failed: dialing through a tunnel: no tunnel connection found: no node reverse tunnel for c45e3db3-2052-483a-b9b0-dd9340787fb0.stage-teleport-cluster found, dialing directly: dial tcp 3.238.130.234:3022: i/o timeout. id:17 local:101.53.142.168:3023 login:suraj remote:49.248.168.170:4991 teleportUser:suraj regular/sshserver.go:1359
ERRO [NODE]      dialing through a tunnel: no tunnel connection found: no node reverse tunnel for c45e3db3-2052-483a-b9b0-dd9340787fb0.stage-teleport-cluster found, dialing directly: dial tcp 3.238.130.234:3022: i/o timeout regular/sshserver.go:1539

My laptop -

suraj@suraj:~$ tsh login --proxy=stage.advasmart.in:3080 --auth=local --user=suraj
Enter password for Teleport user suraj:
Enter your OTP token:
538482
> Profile URL:        https://stage.advasmart.in:3080
  Logged in as:       suraj
  Cluster:            stage-teleport-cluster
  Roles:              admin*
  Logins:             suraj, root, ubuntu
  Kubernetes:         disabled
  Valid until:        2020-12-25 07:12:31 +0530 IST [valid for 12h0m0s]
  Extensions:         permit-agent-forwarding, permit-port-forwarding, permit-pty


* RBAC is only available in Teleport Enterprise
  https://gravitational.com/teleport/docs/enterprise
suraj@suraj:~$ 
suraj@suraj:~$ 
suraj@suraj:~$ tsh ssh suraj@teleport.advarisk.com
error: failed connecting to node teleport.advarisk.com. dialing through a tunnel: no tunnel connection found: no node reverse tunnel for c45e3db3-2052-483a-b9b0-dd9340787fb0.stage-teleport-cluster found, dialing directly: dial tcp 3.238.130.234:3022: i/o timeout
suraj@suraj:~$ tsh ssh suraj@smart.advarisk

Any ideas?

Regards,
Suraj

#9

Hi @surajmundada,

It appears that you have the same node appearing twice as if it is cached. Since it appears twice you may get an error stating the hostname is ambiguous. The work around is to delete the node using tctl rm node/<node-name> and re-add, or to restart the teleport process in the proxy node. If you wish not to delete you can access the node by using the uuid rather than the hostname( for example tsh ssh root@adcb4591-43e3-4285-8753-c49da422ec08 ).

Hope this helps.

Regards,
Jay

#10

Hi @Jay

I restarted both staging.advasmart.in and teleport.advarisk.com and added teleport to the cluster. it worked. :smiley: So, could not try removing the node.

Regards,
Suraj