Getting x509 error during login to Proxy using tsh

Hi,
I installed latest version of Teleport with proxy, auth and node on a remote server. This server already had a domain name and SSL cert installed on it.

Proxy is running successfully and I can login into the user portal. So, i guess setup is done.

Now, I installed Teleport on my local machine to test connecting a server from my local machine using tsh.

suraj@suraj:~/Softwares/Utilities/teleport$ tsh login --proxy=stage.myserver.in:3080 --user=suraj
error: Get “https://stage.myserver.in:3080/v1/webapi/ping”: x509: certificate is valid for e2e-39-168, localhost, localhost.local, not stage.myserver.in

I googled for above issue but not able to get pointers to resolve this.

SSL certificate on stage.myserver.in is active and works fine.

Any thoughts what needs to be fixed?

Regards,
Suraj

Hi,

I assume it is SSL based authentication during login process. Currently SSL certificate is a server certificate and installed on server (Teleport Proxy node that also acts as Auth and User Node)

TO be able to login into Teleport Auth using “tsh”, Do I need ssl certificate or keys on my laptop as well?

Regards,
Suraj

Hi,

Request to help on this.

Regards,
Suraj

Hi Suraj,

It appears that the cert you’re using is valid for localhost and e2e-39-168 hostnames, but not for “stage.myserver.in”. You can confirm by running the following command against your domain:

openssl s_client -connect stage.myserver.in:3080 | openssl x509 -noout -text

To remedy the issue you can add stage.myserver.in to the list of SNIs or regenerate a cert w/ that hostname included.

Hi Alen,

Thanks for revert.
Output of command shows that cert points to the expected domain name as below -

root@e2e-39-168:~# openssl s_client -connect stage.advasmart.in:3080 | openssl x509 -noout     -text
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = stage.advasmart.in
verify return:1
Certificate:
Data:
    Version: 3 (0x2)
    Serial Number:
....
....
....

X509v3 Subject Alternative Name: 
          DNS:stage.advasmart.in

....
....
....

stage.advasmart.in is the correct domain name where I have installed teleport auth, proxy and node and to which I am trying to connect from my laptop using tsh.

So, cert seems to be okay.

What else I can check to identify the issue?

Regards,
Suraj

@surajmundada Do you have an public_addr configured for the proxy service?

Yes. My teleport.yaml file looks as below -

teleport:
    data_dir: /var/lib/teleport
auth_service:
    enabled: true
    cluster_name: "stage-teleport-cluster"
    listen_addr: 0.0.0.0:3025
    tokens:
    - proxy,node,app:f7adb7ccdf04037bcd2b52ec6010fd6f0caec94ba190b765
    authentication:
    # default authentication type. possible values are 'local' and 'github' for OSS
    #  and 'oidc', 'saml' and 'false' for Enterprise.
    type: local
    # second_factor can be off, otp, or u2f
    second_factor: off
ssh_service:
    enabled: true
    labels:
        env: staging
app_service:
    enabled: true
    debug_app: true
proxy_service:
    enabled: true
    listen_addr: 0.0.0.0:3023
    web_listen_addr: 0.0.0.0:3080
    tunnel_listen_addr: 0.0.0.0:3024
    public_addr: stage.advasmart.in:3080
    https_keypairs:
    -  key_file: '/etc/letsencrypt/live/stage.advasmart.in/privkey.pem'
       cert_file: '/etc/letsencrypt/live/stage.advasmart.in/fullchain.pem'

Hi @surajmundada

Can you please change all the services to enabled: yes rather than enabled: true, for we have found some odd behaviors with setting to true. Additionally, I suggest the below changes to the proxy section:

proxy_service:
    enabled: yes
    listen_addr: 0.0.0.0:3023
    web_listen_addr: 0.0.0.0:3080
    tunnel_listen_addr: 0.0.0.0:3024
    public_addr: stage.advasmart.in:3080
    ssh_public_addr: stage.advasmart.in:3023
    tunnel_public_addr: stage.advasmart.in:3024
    https_keypairs:
    -  key_file: '/etc/letsencrypt/live/stage.advasmart.in/privkey.pem'
       cert_file: '/etc/letsencrypt/live/stage.advasmart.in/fullchain.pem'

Then reload the daemon and restart the process.

systemctl daemon-reload
systemctl restart teleport
  • Jay

I changed all “enabled” flags to “yes” from “true” and also added tunnel_public_addr: stage.advasmart.in:3024 to config file and it resolved the x:509 issue. Thanks a lot. :grinning: :+1:

However, when I tried to connect to stage.advasmart.in from my laptop, I got following issue -

suraj@suraj:/$ sudo tsh login --proxy=stage.advasmart.in:3080 --user=suraj
[sudo] password for suraj: 
Enter password for Teleport user suraj:
Enter your OTP token:
254114
> Profile URL:        https://stage.advasmart.in:3080
Logged in as:       suraj
Cluster:            stage-teleport-cluster
Roles:              admin*
Logins:             suraj
Kubernetes:         disabled
Valid until:        2020-12-19 02:43:39 +0530 IST [valid for 12h0m0s]
Extensions:         permit-agent-forwarding, permit-port-forwarding, permit-pty


* RBAC is only available in Teleport Enterprise
https://gravitational.com/teleport/docs/enterprise

Can I not connect to remote server as a regular user without any role through commnnity version?

Regards,
Suraj

I dug deeper into documentation and made few changes. teleport.yaml looks as below after few changes -

teleport:
    data_dir: /var/lib/teleport
    log:
        output: stderr
        severity: INFO

auth_service:
     enabled: "yes"
     cluster_name: "stage-teleport-cluster"
     listen_addr: 0.0.0.0:3025
     public_addr: stage.advasmart.in:3025
     tokens:
     - proxy,node,app:REDACTED
     authentication:
     # default authentication type. possible values are 'local' and 'github' for OSS
     #  and 'oidc', 'saml' and 'false' for Enterprise.
     type: local
     # second_factor can be off, otp, or u2f
     second_factor: off

ssh_service:
      enabled: "yes"
      labels:
             env: staging

app_service:
      enabled: "yes"
      debug_app: true

proxy_service:
     enabled: "yes"
     listen_addr: 0.0.0.0:3023
     web_listen_addr: 0.0.0.0:3080
     tunnel_listen_addr: 0.0.0.0:3024
     public_addr: stage.advasmart.in:3080
     tunnel_public_addr: stage.advasmart.in:3024
     https_keypairs:
     - key_file: '/etc/letsencrypt/live/stage.advasmart.in/privkey.pem'
       cert_file: '/etc/letsencrypt/live/stage.advasmart.in/fullchain.pem'

Created teleport users on stage.advasmart.in as below -

root@e2e-39-168:~# tctl users ls
User  Allowed logins    
----- ----------------- 
suraj suraj,root,ubuntu 

Then started teleport as below -

root@e2e-39-168:~# sudo teleport start --roles=node,auth,proxy 

Logs -

WARN [PROXY:1:C] Failed to set tombstone: database is closed cache/cache.go:655
WARN [PROXY:1]   Re-init the watcher on error: grpc: the client connection is closing.  services/proxywatcher.go:189
WARN [NODE:2:CA] Re-init the cache on error: watcher closed. cache/cache.go:627
WARN [PROXY:2]   Re-init the watcher on error: watcher closed. services/proxywatcher.go:189 
WARN [PROXY:2:C] Re-init the cache on error: watcher closed. cache/cache.go:627
WARN [PROXY:1]   Re-init the watcher on error: grpc: the client connection is closing. services/proxywatcher.go:189
INFO [PROXY:SER] Shutting down gracefully. service/service.go:2624
WARN             Failed to sync reverse tunnels: {"message":"cache is closed"}. reversetunnel/rc_manager.go:138
INFO [AUTH:1]    Shutting down gracefully. service/service.go:1354
WARN [REVERSE:S] Re-init the cache on error: watcher closed. cache/cache.go:627
WARN [PROXY:2]   Re-init the watcher on error: cache is closed. services/proxywatcher.go:189
WARN [NODE:2:CA] Re-init the cache on error: cache is closed. cache/cache.go:627
WARN [PROXY:2:C] Re-init the cache on error: cache is closed. cache/cache.go:627
INFO [PROXY:SER] Exited. service/service.go:2459
INFO [WEB]       Closing session cache. web/sessions.go:357
WARN [REVERSE:S] Re-init the cache on error: {"message":"cache is closed"}. cache/cache.go:627
INFO [KEYGEN]    Stopping key precomputation routine. native/native.go:144
INFO [WEB]       Closing session cache. web/sessions.go:357
INFO [PROXY:SER] Exited. service/service.go:2645
INFO [PROC]      Waiting for services: [auth.tls auth.shutdown] to finish. service/signals.go:43
ERRO [AUTH]      Failed to perform cert rotation check: cache is closed. auth/auth.go:279
INFO [PROC]      Waiting for services: [auth.tls auth.shutdown] to finish. service/signals.go:43
ERRO [AUTH]      Failed to perform cert rotation check: cache is closed. auth/auth.go:279
INFO [PROC]      Waiting for services: [auth.tls auth.shutdown] to finish. service/signals.go:43
INFO [PROC]      Waiting for services: [auth.tls auth.shutdown] to finish. service/signals.go:43
ERRO [AUTH]      Failed to perform cert rotation check: cache is closed. auth/auth.go:279
INFO [PROC]      Waiting for services: [auth.tls auth.shutdown] to finish. service/signals.go:43
ERRO [AUTH]      Failed to perform cert rotation check: cache is closed. auth/auth.go:279
INFO [PROC]      Waiting for services: [auth.tls auth.shutdown] to finish. service/signals.go:43
INFO [AUTH:1]    Exited. service/service.go:1361
WARN [AUTH:1]    TLS server exited with error: http: Server closed. service/service.go:1240
INFO [PROC]      The old service was successfully shut down gracefully. service/service.go:530
WARN [NODE:BEAT] Keep alive has failed: cache is closed. srv/heartbeat.go:461
WARN [NODE:BEAT] Heartbeat failed keep alive channel closed. srv/heartbeat.go:256
INFO [AUDIT]     user.login code:T1000I ei:0 event:user.login method:local success:true time:2020-12-18T14:40:36.445Z uid:224c0c17-5de0-42e7-bd51-712120c6bb58 user:suraj events/emitter.go:318

I am still able to login into the admin panel even after getting above errors. So, tried creating the identity on my laptop as below -

suraj@suraj:~$ tsh ssh --proxy=stage.advasmart.in --user=suraj root@stage.advasmart.in
Enter password for Teleport user suraj:
Enter your OTP token:
180447
error: access denied to root connecting to stage.advasmart.in on cluster stage-teleport-cluster

I want to test if I can connect to node “stage.advasmart.in” from my laptop through proxy “stage.advasmart.in” when auth is running on “stage.advasmart.in”

Any idea why I am denied access as root from my laptop when using teleport client “tsh”?

Regards,
Suraj

Hello @surajmundada,

I have redacted your token as this shall be protected. I would suggest you generate new tokens as this may have been compromised. Below I outlined the login process. This can be accomplished after you joined nodes to your cluster (Add a node to a cluster)

From your laptop you first have to logon to tsh, can be accomplished by:

tsh login --proxy=stage.advasmart.in:3080 --auth=local --user=suraj

To see which clusters you can:

# tsh status 

//Should get an output like below                                                                                                                                                      
> Profile URL:        stage.advasmart.in:3080
  Logged in as:     suraj  
  Cluster:           clustername
  Roles:              admin*
  Logins:             suraj, root, ubuntu
  Kubernetes:         disabled
  Valid until:        2020-12-18 22:01:11 -0600 CST [valid for 12h0m0s]
  Extensions:         permit-agent-forwarding, permit-port-forwarding, permit-pty

After you are authenticated you should be able to:

tsh ssh root@<node-name>

Hopefully this answers your questions.

-Jay

Worked and I was able to login and ssh to stage.advasmart.in from my laptop. Thanks for all the help.

I had to do one more change though. I still got access denied. But I tried to connect root@e2e-36-168 instead of root@stage.advasmart.in and it worked.

Then I added nodename as “stage.advasmart.in” in teleport.yaml and also changed host name of the server as “stage.advasmart.in” in /etc/hosts as well as using hostnamectl command. That allowed to connect from my laptop to root@stage.advasmart.in also.

I am facing another issue while adding a node to my cluster. Will raise it in another thread.

Thanks again.

Regards,
Suraj

1 Like