If Teleport is running outside the Kubernetes cluster, you’ll need to generate a kubeconfig for it to use as detailed here: https://gravitational.com/teleport/docs/kubernetes_ssh/#connecting-the-teleport-proxy-to-kubernetes
This script is known to work on GKE and bare metal Kubernetes clusters - I don’t think it’s been tested on EKS. I’d be curious to see whether it works or whether some tweaks might be required.
Thanks again @gus. I have a kubeconfig that works. With EKS you don’t need to login using a certificate. You can assign an IAM role to the EC2 instance that teleport is running on and map that role to a Kubernetes RBAC group.
Ok, so I am able to get it to work if I add the --as=teleport --as-group= flags to the kubectl commands. It seems like my GitHub configuration is not getting applied when I login to Teleport with my GitHub account.
If you run tsh status locally, you should get a username shown for Logged in as: - probably the email address you have registered with Github.
What’s the output of tctl get user/<username> on the auth server - specifically the section on roles (and potentially kubernetes_groups if it’s there)?
@gus when I run tsh status it says Not logged in however I am able to login to the web console using my GitHub ID. When I get the info for my GitHub user it shows me the following:
kind: user
metadata:
expires: 2019-06-22T21:14:26.810812276Z
id: 1561130066811413274
name: jicowan
spec:
created_by:
connector:
id: github
identity: jicowan
type: github
time: 2019-06-21T15:14:26.810814876Z
user:
name: system
expires: 0001-01-01T00:00:00Z
github_identities:
- connector_id: github
username: jicowan
roles:
- admin
status:
is_locked: false
lock_expires: 0001-01-01T00:00:00Z
locked_time: 0001-01-01T00:00:00Z
traits:
kubernetes_groups:
- system:masters
logins:
- root
version: v2
@jicowan That user output looks fine - it’s what I’d expect, and the system:masters at the bottom indicates that the Kubernetes group mapping is getting applied correctly.
If you haven’t logged into the cluster from the command line with tsh (as indicated by Not logged in) then where are you seeing that the groups aren’t getting applied?
@gus Here a description of the steps:
Do I need to do something different to manage kubernetes?
Yes.
The idea is that you run tsh login --proxy=<proxy_address> at the command line and a web browser should pop up, you log in via Github there, then the CLI will print a message stating that you’re now logged in for the next 12 hours.
$ tsh login --proxy=<cluster>.gravitational.co:3080
If browser window does not open automatically, open it by clicking on the link:
http://127.0.0.1:33179/d22e85fc-591a-47e4-86bf-929818ce9a3f
> Profile URL: https://<cluster>.gravitational.co:3080
Logged in as: <redacted>@gmail.com
Cluster: <cluster>.gravitational.co
Roles: clusteradmin*
Logins: root
Valid until: 2019-06-22 00:30:22 -0300 ADT [valid for 12h0m0s]
Extensions: permit-agent-forwarding, permit-port-forwarding, permit-pty
* RBAC is only available in Teleport Enterprise
https://gravitational.com/teleport/docs/enterprise
Once that’s done, kubectl config get-contexts on your local machine should show your cluster:
$ kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* <cluster>.gravitational.co <cluster>.gravitational.co <cluster>.gravitational.co
Then running anything via kubectl should go via Teleport to the Kubernetes cluster:
$ kubectl -v=7 get nodes
I0621 12:35:47.131312 27461 loader.go:359] Config loaded from file /home/gus/.kube/config
I0621 12:35:47.132563 27461 loader.go:359] Config loaded from file /home/gus/.kube/config
I0621 12:35:47.133212 27461 round_trippers.go:383] GET <cluster>.gravitational.co:3026/api?timeout=32s
I0621 12:35:47.133220 27461 round_trippers.go:390] Request Headers:
I0621 12:35:47.133224 27461 round_trippers.go:393] Accept: application/json, */*
I0621 12:35:47.133228 27461 round_trippers.go:393] User-Agent: kubectl/v1.12.4 (linux/amd64) kubernetes/f49fa02
I0621 12:35:47.592972 27461 round_trippers.go:408] Response Status: 200 OK in 459 milliseconds
<snip>
NAME STATUS ROLES AGE VERSION
gke-teleport-demo-cluster-pool-1-afbf7cbd-96cl Ready <none> 53d v1.12.7-gke.10
Thanks @gus. That’s the step I was missing! I should have read through the user guide more carefully.
No worries - glad I could help 
@gus So now I have slightly different issue. I can login
tsh login --proxy=<instance-id>.compute-1.amazonaws.com:3080 --insecure
WARNING: You are using insecure connection to SSH proxy https://....amazonaws.com:3080
WARNING: You are using insecure connection to SSH proxy https://...:3080
If browser window does not open automatically, open it by clicking on the link:
http://127.0.0.1:50581/66b71d13-d389-4326-9768-51df2a3b3498
> Profile URL: https://...compute-1.amazonaws.com:3080
Logged in as: jicowan
Cluster: ....compute-1.amazonaws.com
Roles: admin*
Logins: root
Valid until: 2019-06-22 01:44:11 -0500 CDT [valid for 12h0m0s]
Extensions: permit-agent-forwarding, permit-port-forwarding, permit-pty
but I can’t seem to connect to the Kubernetes server
kubectl get nodes -v 7
I0621 13:45:30.005130 44300 loader.go:359] Config loaded from file /Users/jicowan/.kube/config
I0621 13:45:30.013528 44300 round_trippers.go:416] GET https://...:3026/api?timeout=32s
I0621 13:45:30.013556 44300 round_trippers.go:423] Request Headers:
I0621 13:45:30.013565 44300 round_trippers.go:426] User-Agent: kubectl/v1.14.2 (darwin/amd64) kubernetes/66049e3
I0621 13:45:30.013573 44300 round_trippers.go:426] Accept: application/json, */*
I0621 13:45:30.248263 44300 round_trippers.go:441] Response Status: 403 Forbidden in 234 milliseconds
I0621 13:45:30.249843 44300 round_trippers.go:416] GET https://....amazonaws.com:3026/apis?timeout=32s
I can see the entries in my local kube config for the teleport proxy, but it’s behaving as if there’s no kube config on the server. When I try get nodes it says the server doesn't have a resource type "nodes". I am able to issue kubectl commands from the proxy to the k8s cluster when I SSH to the proxy, but I can’t seem to do it through a proxy session.
Given that EKS only supports IAM authN, I’d love to create a reference architecture for teleport that allows folks to use GitHub or OIDC.
Looks like it’s a permissions issue. When I issue the kubectl auth can-i "*" "*" command I get users jicowan is forbidden: User system:anonymous cannot impersonate resource users in API group "" at the cluster scope Do I need to create a specific role binding for jicowan or is it supposed to use the role I’ve assigned to it through the github.yaml file?
This is the reason why I thought that a certificate might be needed for authentication in the first place and suggested the use of https://github.com/gravitational/teleport/blob/master/examples/gke-auth/get-kubeconfig.sh (linked from https://gravitational.com/teleport/docs/kubernetes_ssh/#connecting-the-teleport-proxy-to-kubernetes)
What Kubernetes group is assigned to the IAM role on the EC2 instance? It’ll need to be something with high permissions like system:masters so that Teleport is able to use the impersonate verb. I haven’t done anything on EKS myself so I’m unsure whether it does anything funky with permissions.
If the calls are currently being made as system:anonymous then that’s the reason for it getting a 403.
You shouldn’t need to create a RoleBinding in Kubernetes for anything. Teleport will give you the kubernetes_groups that you assigned in the github.yaml file, but ONLY if it’s able to send impersonate requests to the Kubernetes apiserver as a sufficiently highly privileged user.
@sasha did some work with Teleport on EKS and posted about it here - you might find something useful in that thread.
@gus the role assigned to the EC2 instance that teleport runs on is mapped to the Kubernetes teleport group. The teleport group has permissions to impersonate other roles. I can confirm this works by logging in to the instance and running kubectl get nodes --as teleport --as-group system:masters. I used the clusterrole and rolebinding from the documentation to create the role for teleport. Do I need to use system:masters? I’ll take a look at the post @sasha wrote. I wanted to try doing this on an EC2 instance before I tried running teleport on the cluster.
I also noticed that after I use tsh to login to Teleport, all of the user entries in my local kubeconfig get wiped out.
@gus I think I figured it out. It worked once I added system:anonymous to the cluster role binding for teleport
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: teleport-impersonation
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: Group
name: teleport-group
- apiGroup: rbac.authorization.k8s.io
kind: User
name: system:anonymous
Probably not very secure but it seems to work.
Interesting. I presume that EKS binds to a different user by default.
trying the same use case
a) teleport deployed in ec2 , kubernetes enabled
b) created eks cluster (myself) and copied kubeconfig in teleport server
c) created teleport user
d) can login in teleport UI, but not in command line,
tsh login --proxy=teleport:3080 --insecure
ERROR REPORT:
Original Error: *errors.errorString unexpected EOF
Stack Trace:
/tmp/20190827T134557/src/github.com/gravitational/teleport/lib/client/api.go:387 github.com/gravitational/teleport/lib/client.readProfile
/tmp/20190827T134557/src/github.com/gravitational/teleport/lib/client/api.go:483 github.com/gravitational/teleport/lib/client.Status
/tmp/20190827T134557/src/github.com/gravitational/teleport/tool/tsh/tsh.go:382 main.onLogin
/tmp/20190827T134557/src/github.com/gravitational/teleport/tool/tsh/tsh.go:324 main.Run
/tmp/20190827T134557/src/github.com/gravitational/teleport/tool/tsh/tsh.go:174 main.main
/usr/local/go/src/runtime/proc.go:209 runtime.main
/usr/local/go/src/runtime/asm_amd64.s:1338 runtime.goexit
User Message:
Team, trying the same setup but unable to access the cluster,
teleport deployed in ec2 instance with Teleport Base AMI from the AWS marketplace.
AWS eks cluster using EKSCTL command cli
Followed the same Process as mentioned here,
https://gravitational.com/teleport/docs/aws-oss-guide/#using-teleport-with-eks
Able to login from the UI but not from the ec2-instance’s command line as its running on a different server and the url for the brower says to http://127.0.0.1/{token}
have added the https certs as mentioned in eks teleport port ( certs using letsencrypt) but still the brower is not able to recognise and its giving self signed certs.
proxy_service:
enabled: "yes"
listen_addr: 0.0.0.0:3023
web_listen_addr: 0.0.0.0:3080
tunnel_listen_addr: 0.0.0.0:3024
public_addr: my-url:3080
https_key_file: /etc/letsencrypt/live/my-url/privkey.pem
https_cert_file: /etc/letsencrypt/live/my-url/fullchain.pem
Teleport version: 4.3.0
From the UI,
I’m able to login to the proxy server and when i try to do kubectl get pods it give me below error,
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:teleport:teleport-sa" cannot list resource "pods" in API group "" in the namespace "default"
But when i use the “–as=teleport” i’m able to get the resources and when i add “–as-groups=system:masters” i’m able to access the whole cluster.
kubectl get ns --as=teleport --as-group=system:masters
NAME STATUS AGE
default Active 21h
kube-node-lease Active 21h
kube-public Active 21h
kube-system Active 21h
teleport Active 19h
My github.yml
kind: github
version: v3
metadata:
name: github
spec:
client_id: {id}
client_secret: {tokrn}
display: Github
redirect_url: https://my-url:3080/v1/webapi/github/callback
teams_to_logins:
- kubernetes_groups:
- system:masters
- teleport-group
logins:
- github
- ec2-user
- teleport
organization: abc-project
team: devops
When i try tsh login from my local system,
login --proxy=my-url:3080 --auth=github --insecure
If browser window does not open automatically, open it by clicking on the link:
http://127.0.0.1:46865/token
> Profile URL: https://my-url:3080
Logged in as: Gangaram
Cluster: my-cluster
Roles: admin*
Logins: github, ec2-user, teleport
Valid until: 2020-10-27 21:33:35 +0530 IST [valid for 12h0m0s]
Extensions: permit-agent-forwarding, permit-port-forwarding, permit-pty
RBAC is only available in Teleport Enterprise
https://gravitational.com/teleport/docs/enterprise