Kubernetes group not applied

If Teleport is running outside the Kubernetes cluster, you’ll need to generate a kubeconfig for it to use as detailed here: https://gravitational.com/teleport/docs/kubernetes_ssh/#connecting-the-teleport-proxy-to-kubernetes

This script is known to work on GKE and bare metal Kubernetes clusters - I don’t think it’s been tested on EKS. I’d be curious to see whether it works or whether some tweaks might be required.

Thanks again @gus. I have a kubeconfig that works. With EKS you don’t need to login using a certificate. You can assign an IAM role to the EC2 instance that teleport is running on and map that role to a Kubernetes RBAC group.

1 Like

Ok, so I am able to get it to work if I add the --as=teleport --as-group= flags to the kubectl commands. It seems like my GitHub configuration is not getting applied when I login to Teleport with my GitHub account.

If you run tsh status locally, you should get a username shown for Logged in as: - probably the email address you have registered with Github.

What’s the output of tctl get user/<username> on the auth server - specifically the section on roles (and potentially kubernetes_groups if it’s there)?

@gus when I run tsh status it says Not logged in however I am able to login to the web console using my GitHub ID. When I get the info for my GitHub user it shows me the following:

kind: user
metadata:
  expires: 2019-06-22T21:14:26.810812276Z
  id: 1561130066811413274
  name: jicowan
spec:
  created_by:
    connector:
      id: github
      identity: jicowan
      type: github
    time: 2019-06-21T15:14:26.810814876Z
    user:
      name: system
  expires: 0001-01-01T00:00:00Z
  github_identities:
  - connector_id: github
    username: jicowan
  roles:
  - admin
  status:
    is_locked: false
    lock_expires: 0001-01-01T00:00:00Z
    locked_time: 0001-01-01T00:00:00Z
  traits:
    kubernetes_groups:
    - system:masters
    logins:
    - root
version: v2

@jicowan That user output looks fine - it’s what I’d expect, and the system:masters at the bottom indicates that the Kubernetes group mapping is getting applied correctly.

If you haven’t logged into the cluster from the command line with tsh (as indicated by Not logged in) then where are you seeing that the groups aren’t getting applied?

@gus Here a description of the steps:

  1. Goto the login page for Teleport and login using my GitHub credentials
  2. Launch a terminal (SSH) session from within the browser/Teleport console
  3. Run kubectl commands

Do I need to do something different to manage kubernetes?

Yes.

The idea is that you run tsh login --proxy=<proxy_address> at the command line and a web browser should pop up, you log in via Github there, then the CLI will print a message stating that you’re now logged in for the next 12 hours.

$ tsh login --proxy=<cluster>.gravitational.co:3080
If browser window does not open automatically, open it by clicking on the link:
 http://127.0.0.1:33179/d22e85fc-591a-47e4-86bf-929818ce9a3f
> Profile URL:  https://<cluster>.gravitational.co:3080
  Logged in as: <redacted>@gmail.com
  Cluster:      <cluster>.gravitational.co
  Roles:        clusteradmin*
  Logins:       root
  Valid until:  2019-06-22 00:30:22 -0300 ADT [valid for 12h0m0s]
  Extensions:   permit-agent-forwarding, permit-port-forwarding, permit-pty


* RBAC is only available in Teleport Enterprise
  https://gravitational.com/teleport/docs/enterprise

Once that’s done, kubectl config get-contexts on your local machine should show your cluster:

$ kubectl config get-contexts
CURRENT   NAME                        CLUSTER                                               AUTHINFO                                              NAMESPACE
*         <cluster>.gravitational.co  <cluster>.gravitational.co                            <cluster>.gravitational.co

Then running anything via kubectl should go via Teleport to the Kubernetes cluster:

$ kubectl -v=7 get nodes
I0621 12:35:47.131312   27461 loader.go:359] Config loaded from file /home/gus/.kube/config
I0621 12:35:47.132563   27461 loader.go:359] Config loaded from file /home/gus/.kube/config
I0621 12:35:47.133212   27461 round_trippers.go:383] GET <cluster>.gravitational.co:3026/api?timeout=32s
I0621 12:35:47.133220   27461 round_trippers.go:390] Request Headers:
I0621 12:35:47.133224   27461 round_trippers.go:393]     Accept: application/json, */*
I0621 12:35:47.133228   27461 round_trippers.go:393]     User-Agent: kubectl/v1.12.4 (linux/amd64) kubernetes/f49fa02
I0621 12:35:47.592972   27461 round_trippers.go:408] Response Status: 200 OK in 459 milliseconds
<snip>
NAME                                             STATUS   ROLES    AGE   VERSION
gke-teleport-demo-cluster-pool-1-afbf7cbd-96cl   Ready    <none>   53d   v1.12.7-gke.10

Thanks @gus. That’s the step I was missing! I should have read through the user guide more carefully.

No worries - glad I could help :slight_smile:

@gus So now I have slightly different issue. I can login

tsh login --proxy=<instance-id>.compute-1.amazonaws.com:3080 --insecure
WARNING: You are using insecure connection to SSH proxy https://....amazonaws.com:3080
WARNING: You are using insecure connection to SSH proxy https://...:3080
If browser window does not open automatically, open it by clicking on the link:
 http://127.0.0.1:50581/66b71d13-d389-4326-9768-51df2a3b3498
> Profile URL:  https://...compute-1.amazonaws.com:3080
  Logged in as: jicowan
  Cluster:      ....compute-1.amazonaws.com
  Roles:        admin*
  Logins:       root
  Valid until:  2019-06-22 01:44:11 -0500 CDT [valid for 12h0m0s]
  Extensions:   permit-agent-forwarding, permit-port-forwarding, permit-pty 

but I can’t seem to connect to the Kubernetes server

kubectl get nodes -v 7
I0621 13:45:30.005130   44300 loader.go:359] Config loaded from file /Users/jicowan/.kube/config
I0621 13:45:30.013528   44300 round_trippers.go:416] GET https://...:3026/api?timeout=32s
I0621 13:45:30.013556   44300 round_trippers.go:423] Request Headers:
I0621 13:45:30.013565   44300 round_trippers.go:426]     User-Agent: kubectl/v1.14.2 (darwin/amd64) kubernetes/66049e3
I0621 13:45:30.013573   44300 round_trippers.go:426]     Accept: application/json, */*
I0621 13:45:30.248263   44300 round_trippers.go:441] Response Status: 403 Forbidden in 234 milliseconds
I0621 13:45:30.249843   44300 round_trippers.go:416] GET https://....amazonaws.com:3026/apis?timeout=32s

I can see the entries in my local kube config for the teleport proxy, but it’s behaving as if there’s no kube config on the server. When I try get nodes it says the server doesn't have a resource type "nodes". I am able to issue kubectl commands from the proxy to the k8s cluster when I SSH to the proxy, but I can’t seem to do it through a proxy session.

Given that EKS only supports IAM authN, I’d love to create a reference architecture for teleport that allows folks to use GitHub or OIDC.

Looks like it’s a permissions issue. When I issue the kubectl auth can-i "*" "*" command I get users jicowan is forbidden: User system:anonymous cannot impersonate resource users in API group "" at the cluster scope Do I need to create a specific role binding for jicowan or is it supposed to use the role I’ve assigned to it through the github.yaml file?

This is the reason why I thought that a certificate might be needed for authentication in the first place and suggested the use of https://github.com/gravitational/teleport/blob/master/examples/gke-auth/get-kubeconfig.sh (linked from https://gravitational.com/teleport/docs/kubernetes_ssh/#connecting-the-teleport-proxy-to-kubernetes)

What Kubernetes group is assigned to the IAM role on the EC2 instance? It’ll need to be something with high permissions like system:masters so that Teleport is able to use the impersonate verb. I haven’t done anything on EKS myself so I’m unsure whether it does anything funky with permissions.

If the calls are currently being made as system:anonymous then that’s the reason for it getting a 403.

You shouldn’t need to create a RoleBinding in Kubernetes for anything. Teleport will give you the kubernetes_groups that you assigned in the github.yaml file, but ONLY if it’s able to send impersonate requests to the Kubernetes apiserver as a sufficiently highly privileged user.

@sasha did some work with Teleport on EKS and posted about it here - you might find something useful in that thread.

@gus the role assigned to the EC2 instance that teleport runs on is mapped to the Kubernetes teleport group. The teleport group has permissions to impersonate other roles. I can confirm this works by logging in to the instance and running kubectl get nodes --as teleport --as-group system:masters. I used the clusterrole and rolebinding from the documentation to create the role for teleport. Do I need to use system:masters? I’ll take a look at the post @sasha wrote. I wanted to try doing this on an EC2 instance before I tried running teleport on the cluster.

I also noticed that after I use tsh to login to Teleport, all of the user entries in my local kubeconfig get wiped out.

I think I may be running in to this issue, https://github.com/kubernetes/kubernetes/issues/43227

@gus I think I figured it out. It worked once I added system:anonymous to the cluster role binding for teleport

roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: teleport-impersonation
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: teleport-group
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: system:anonymous

Probably not very secure but it seems to work.

Interesting. I presume that EKS binds to a different user by default.

trying the same use case

a) teleport deployed in ec2 , kubernetes enabled
b) created eks cluster (myself) and copied kubeconfig in teleport server
c) created teleport user
d) can login in teleport UI, but not in command line,

tsh login --proxy=teleport:3080 --insecure

ERROR REPORT:

Original Error: *errors.errorString unexpected EOF

Stack Trace:

/tmp/20190827T134557/src/github.com/gravitational/teleport/lib/client/api.go:387 github.com/gravitational/teleport/lib/client.readProfile

/tmp/20190827T134557/src/github.com/gravitational/teleport/lib/client/api.go:483 github.com/gravitational/teleport/lib/client.Status

/tmp/20190827T134557/src/github.com/gravitational/teleport/tool/tsh/tsh.go:382 main.onLogin

/tmp/20190827T134557/src/github.com/gravitational/teleport/tool/tsh/tsh.go:324 main.Run

/tmp/20190827T134557/src/github.com/gravitational/teleport/tool/tsh/tsh.go:174 main.main

/usr/local/go/src/runtime/proc.go:209 runtime.main

/usr/local/go/src/runtime/asm_amd64.s:1338 runtime.goexit

User Message:

Team, trying the same setup but unable to access the cluster,

  • teleport deployed in ec2 instance with Teleport Base AMI from the AWS marketplace.

  • AWS eks cluster using EKSCTL command cli

  • Followed the same Process as mentioned here,
    https://gravitational.com/teleport/docs/aws-oss-guide/#using-teleport-with-eks

  • Able to login from the UI but not from the ec2-instance’s command line as its running on a different server and the url for the brower says to http://127.0.0.1/{token}

  • have added the https certs as mentioned in eks teleport port ( certs using letsencrypt) but still the brower is not able to recognise and its giving self signed certs.

      proxy_service:
         enabled: "yes"
         listen_addr: 0.0.0.0:3023
         web_listen_addr: 0.0.0.0:3080
         tunnel_listen_addr: 0.0.0.0:3024
         public_addr: my-url:3080
         https_key_file: /etc/letsencrypt/live/my-url/privkey.pem
         https_cert_file: /etc/letsencrypt/live/my-url/fullchain.pem 
    
  • Teleport version: 4.3.0

From the UI,
I’m able to login to the proxy server and when i try to do kubectl get pods it give me below error,

Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:teleport:teleport-sa" cannot list resource "pods" in API group "" in the namespace "default"

But when i use the “–as=teleport” i’m able to get the resources and when i add “–as-groups=system:masters” i’m able to access the whole cluster.

kubectl get ns --as=teleport --as-group=system:masters
NAME              STATUS   AGE
default           Active   21h
kube-node-lease   Active   21h
kube-public       Active   21h
kube-system       Active   21h
teleport          Active   19h

My github.yml

kind: github
version: v3
metadata:
  name: github
spec:
  client_id: {id}
  client_secret: {tokrn}
  display: Github
  redirect_url: https://my-url:3080/v1/webapi/github/callback
  teams_to_logins:
  - kubernetes_groups:
    - system:masters
    - teleport-group
  logins:
  - github
  - ec2-user
  - teleport
  organization: abc-project
  team: devops

When i try tsh login from my local system,

login --proxy=my-url:3080 --auth=github --insecure

If browser window does not open automatically, open it by clicking on the link:
 http://127.0.0.1:46865/token
> Profile URL:  https://my-url:3080
Logged in as: Gangaram
Cluster:      my-cluster
Roles:        admin*
Logins:       github, ec2-user, teleport
Valid until:  2020-10-27 21:33:35 +0530 IST [valid for 12h0m0s]
Extensions:   permit-agent-forwarding, permit-port-forwarding, permit-pty
 RBAC is only available in Teleport Enterprise
https://gravitational.com/teleport/docs/enterprise