이번 주차에는 EKS 보안에 대해 배웠다. 먼저, 쿠버네티스의 인증 인가 체계에 대해 배우고, EKS는 어떻게 다른 지 학습한다. 이후 IRSA에 대한 실습을 마지막으로 이번주차가 종료된다.
환경세팅
이번의 EKS 배포환경은 이전 주차와 크게 다르지 않다. EKS 인증/인가 테스트를 위해 작업용 EC2가 하나 추가되었다. 가시다님이 제공해주신 CloudFormation을 통해 배포를 실시한다.
K8S 인증/인가
쿠버네티스의 인증 인가 체계를 살펴보면 아래의 그림과 같다.
인증 단계를 거친다. 인증이 완료되면 인가 단계를 거친다. 인가 단계를 통해 명령이 리소스에 대한 권한이 있는 지 확인한 후, Admission control 을 통해서 etcd에 접근한다.
실습환경은 다음과 같다.
2개의 네임 스페이스가 존재하고 각 네임스페이스에는 서비스어카운트와 파드를 둔다. 서비스어카운트에 롤을 바인딩해보며 인증인가 체계를 확인한다.
- kubeconfig 파일 확인
우선 kubeconfig 파일을 확인하여 현재의 쿠버네티스 접속 상태를 확인한다.
clusters : kubectl 이 사용할 쿠버네티스 API 서버의 접속 정보 목록. 원격의 쿠버네티스 API 서버의 주소를 추가해 사용 가능
users : 쿠버네티스의 API 서버에 접속하기 위한 사용자 인증 정보 목록. (서비스 어카운트의 토큰, 혹은 인증서의 데이터 등)
contexts : cluster 항목과 users 항목에 정의된 값을 조합해 최종적으로 사용할 쿠버네티스 클러스터의 정보(컨텍스트)를 설정.
예를 들어 clusters 항목에 클러스터 A,B 가 정의돼 있고, users 항목에 사용자 a,b 가 정의돼 있다면 cluster A + user a 를 조합해,
'cluster A 에 user a 로 인증해 쿠버네티스를 사용한다' 라는 새로운 컨텍스트를 정의할 수 있습니다.
$cat .kube/config
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJ..
server: https://BF69CC8DDDFB36E86FE01E52B6F5641B.gr7.ap-northeast-2.eks.amazonaws.com
name: myeks.ap-northeast-2.eksctl.io
contexts:
- context:
cluster: myeks.ap-northeast-2.eksctl.io
user: EKS-study@myeks.ap-northeast-2.eksctl.io
name: kane
current-context: kane
kind: Config
preferences: {}
users:
- name: EKS-study@myeks.ap-northeast-2.eksctl.io
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- eks
- get-token
- --output
- json
- --cluster-name
- myeks
- --region
- ap-northeast-2
command: aws
env:
- name: AWS_STS_REGIONAL_ENDPOINTS
value: regional
interactiveMode: IfAvailable
provideClusterInfo: false- 위의 실습 환경과 같이 네임스페이스와 파드, 서비스 어카운트를 생성합니다.
$kubectl create namespace dev-team
namespace/dev-team created
$k create ns infra-team
namespace/infra-team created
# 네임스페이스 생성 확인
$k get ns
NAME STATUS AGE
default Active 34m
dev-team Active 12s #<--
infra-team Active 3s #<--
kube-node-lease Active 34m
kube-public Active 34m
kube-system Active 34m
monitoring Active 7m55s
$k create sa dev-k8s -n dev-team
serviceaccount/dev-k8s created
$k create sa infra-k8s -n infra-team
serviceaccount/infra-k8s created
$k get sa -n infra-team
NAME SECRETS AGE
default 0 37s
infra-k8s 0 7s
- 아래는 서비스 어카운트의 정보를 확인한다. 토큰 값이 없는 것을 확인할 수 있다.
1.24 버전으로 업데이트 되며, 서비스 계정을 생성하면 토큰이 자동으로 생성되는 방식에서 수동으 로 생성해야 하는 방식으로 변경됨.
# 서비스 어카운트 정보확인
#
$k get sa -n infra-team infra-k8s -o yaml | yh
apiVersion: v1
kind: ServiceAccount
metadata:
creationTimestamp: "2023-05-31T11:28:28Z"
name: infra-k8s
namespace: infra-team
resourceVersion: "7932"
uid: f5371b71-860f-48b2-9927-0b3d4e60052a
- “dev-k8s 서비스어카운트의 토큰 정보 확인” 부분은 추후 수동으로 토큰을 생성한 뒤 진행
- 서비스 어카운트를 지정하여 파드 생성 후 권한 테스트
$cat <<EOF | kubectl create -f -
> apiVersion: v1
> kind: Pod
> metadata:
> name: dev-kubectl
> namespace: dev-team
> spec:
> serviceAccountName: dev-k8s
> containers:
> - name: kubectl-pod
> image: bitnami/kubectl:1.24.10
> command: ["tail"]
> args: ["-f", "/dev/null"]
> terminationGracePeriodSeconds: 0
> EOF
pod/dev-kubectl created
$cat <<EOF | kubectl create -f -
> apiVersion: v1
> kind: Pod
> metadata:
> name: infra-kubectl
> namespace: infra-team
> spec:
> serviceAccountName: infra-k8s
> containers:
> - name: kubectl-pod
> image: bitnami/kubectl:1.24.10
> command: ["tail"]
> args: ["-f", "/dev/null"]
> terminationGracePeriodSeconds: 0
> EOF
pod/infra-kubectl created
#확인
$kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
dev-team dev-kubectl 0/1 ContainerCreating 0 12s
infra-team infra-kubectl 0/1 ContainerCreating 0 10s
kube-system aws-load-balancer-controller-5f99d5f58f-lqj8f 1/1 Running 0 19m
kube-system aws-load-balancer-controller-5f99d5f58f-mpmtr 1/1 Running 0 19m
...
monitoring kube-prometheus-stack-prometheus-node-exporter-zv249 1/1 Running 0 18m
monitoring prometheus-kube-prometheus-stack-prometheus-0 2/2 Running 0 18m
# 서비스 어카운트와 파드 확인
$kubectl get pod -o dev-kubectl -n dev-team -o yaml
apiVersion: v1
...
securityContext: {}
serviceAccount: dev-k8s
serviceAccountName: dev-k8s
terminationGracePeriodSeconds: 0
...
$kubectl get pod -o infra-kubectl -n infra-team -o yaml
apiVersion: v1
...
serviceAccount: infra-k8s
serviceAccountName: infra-k8s
...
# 서비스 어카운트 정보 확인
$kubectl exec -it dev-kubectl -n dev-team -- ls /run/secrets/kubernetes.io/serviceaccount
ca.crt namespace token
$kubectl exec -it dev-kubectl -n dev-team -- cat /run/secrets/kubernetes.io/serviceaccount/token
eyJhbGciOiJSUzI1NiIsImtpZCI6ImMwMTM4ZDQ5OGUyYjk0OGE3MzA5M2VkOTI3ZGFiODNjNTE2NGUzZjgifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjIl0sImV4cCI6M...
alias k1='kubectl exec -it dev-kubectl -n dev-team -- kubectl'
alias k2='kubectl exec -it infra-kubectl -n infra-team -- kubectl'
# 권한 테스트
$k1 get pods
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:dev-team:dev-k8s" cannot list resource "pods" in API group "" in the namespace "dev-team"
command terminated with exit code 1
$k1 run nginx --image nginx:1.20-alpine
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:dev-team:dev-k8s" cannot create resource "pods" in API group "" in the namespace "dev-team"
command terminated with exit code 1
$k2 get pods # kubectl exec -it infra-kubectl -n infra-team -- kubectl get pods 와 동일한 실행 명령이다!
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:infra-team:infra-k8s" cannot list resource "pods" in API group "" in the namespace "infra-team"
command terminated with exit code 1
$k2 run nginx --image nginx:1.20-alpine
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:infra-team:infra-k8s" cannot create resource "pods" in API group "" in the namespace "infra-team"
command terminated with exit code 1
# 권한이 없는 것을 확인할 수 있음.
$k1 auth can-i get pods
no
command terminated with exit code 1
$k2 get pods -n kube-system
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:infra-team:infra-k8s" cannot list resource "pods" in API group "" in the namespace "kube-system"
command terminated with exit code 1
- Role을 부여한 뒤 서비스 어카운트에 바인딩 진행한다.
#각각 네임스페이스에 롤(Role)를 생성 후 서비스 어카운트 바인딩
# 모든 권한 부여(*)
$cat <<EOF | kubectl create -f -
> apiVersion: rbac.authorization.k8s.io/v1
> kind: Role
> metadata:
> name: role-dev-team
> namespace: dev-team
> rules:
> - apiGroups: ["*"]
> resources: ["*"]
> verbs: ["*"]
> EOF
role.rbac.authorization.k8s.io/role-dev-team created
$cat <<EOF | kubectl create -f -
> apiVersion: rbac.authorization.k8s.io/v1
> kind: Role
> metadata:
> name: role-infra-team
> namespace: infra-team
> rules:
> - apiGroups: ["*"]
> resources: ["*"]
> verbs: ["*"]
> EOF
role.rbac.authorization.k8s.io/role-infra-team created
$kubectl get roles -n dev-team
NAME CREATED AT
role-dev-team 2023-05-31T11:40:56Z
$kubectl get roles -n infra-team
NAME CREATED AT
role-infra-team 2023-05-31T11:41:11Z
$kubectl get roles -n dev-team -o yaml
apiVersion: v1
items:
- apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
creationTimestamp: "2023-05-31T11:40:56Z"
name: role-dev-team
namespace: dev-team
resourceVersion: "11069"
uid: 02cc4672-f543-4fb6-a770-4352d37f7a7e
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
kind: List
metadata:
resourceVersion: ""
$kubectl describe roles role-dev-team -n dev-team
Name: role-dev-team
Labels: <none>
Annotations: <none>
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
*.* [] [] [*]
# 롤 바인딩
$cat <<EOF | kubectl create -f -
> apiVersion: rbac.authorization.k8s.io/v1
> kind: RoleBinding
> metadata:
> name: roleB-dev-team
> namespace: dev-team
> roleRef:
> apiGroup: rbac.authorization.k8s.io
> kind: Role
> name: role-dev-team
> subjects:
> - kind: ServiceAccount
> name: dev-k8s
> namespace: dev-team
> EOF
rolebinding.rbac.authorization.k8s.io/roleB-dev-team created
$cat <<EOF | kubectl create -f -
> apiVersion: rbac.authorization.k8s.io/v1
> kind: RoleBinding
> metadata:
> name: roleB-infra-team
> namespace: infra-team
> roleRef:
> apiGroup: rbac.authorization.k8s.io
> kind: Role
> name: role-infra-team
> subjects:
> - kind: ServiceAccount
> name: infra-k8s
> namespace: infra-team
> EOF
rolebinding.rbac.authorization.k8s.io/roleB-infra-team created
$kubectl get rolebindings -n dev-team
NAME ROLE AGE
roleB-dev-team Role/role-dev-team 7s
$kubectl get rolebindings -n infra-team
NAME ROLE AGE
roleB-infra-team Role/role-infra-team 7s
$kubectl get rolebindings -n dev-team -o yaml
apiVersion: v1
items:
- apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
creationTimestamp: "2023-05-31T11:41:34Z"
name: roleB-dev-team
namespace: dev-team
resourceVersion: "11233"
uid: b6b8aa7c-637d-4ec9-a95b-1d7af10fb427
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: role-dev-team
subjects:
- kind: ServiceAccount
name: dev-k8s
namespace: dev-team
kind: List
metadata:
resourceVersion: ""
$kubectl describe rolebindings roleB-dev-team -n dev-team
Name: roleB-dev-team
Labels: <none>
Annotations: <none>
Role:
Kind: Role
Name: role-dev-team
Subjects:
Kind Name Namespace
---- ---- ---------
ServiceAccount dev-k8s dev-team
# 테스트 진행!
alias k1='kubectl exec -it dev-kubectl -n dev-team -- kubectl'
alias k2='kubectl exec -it infra-kubectl -n infra-team -- kubectl'
# (옵션) kubectl auth can-i 로 kubectl 실행 사용자가 특정 권한을 가졌는지 확인
k1 auth can-i get podsNAME READY STATUS RESTARTS AGE
dev-kubectl 1/1 Running 0 3m24s
$k1 get pods
NAME READY STATUS RESTARTS AGE
dev-kubectl 1/1 Running 0 3m39s
$k1 run nginx --image nginx:1.20-alpine
pod/nginx created
$k1 get pods
NAME READY STATUS RESTARTS AGE
dev-kubectl 1/1 Running 0 3m46s
nginx 0/1 ContainerCreating 0 3s
$k1 delete pods nginx
pod "nginx" deleted
$k1 get pods -n kube-system
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:dev-team:dev-k8s" cannot list resource "pods" in API group "" in the namespace "kube-system"
command terminated with exit code 1
$k1 get nodes
Error from server (Forbidden): nodes is forbidden: User "system:serviceaccount:dev-team:dev-k8s" cannot list resource "nodes" in API group "" at the cluster scope
command terminated with exit code 1
$k2 get pods
NAME READY STATUS RESTARTS AGE
infra-kubectl 1/1 Running 0 4m2s
$k2 run nginx --image nginx:1.20-alpine
pod/nginx created
$k2 get pods
NAME READY STATUS RESTARTS AGE
infra-kubectl 1/1 Running 0 4m8s
nginx 1/1 Running 0 2s
$k2 delete pods nginx
pod "nginx" deleted
$k2 get pods -n kube-system
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:infra-team:infra-k8s" cannot list resource "pods" in API group "" in the namespace "kube-system"
command terminated with exit code 1
$k2 get nodes
Error from server (Forbidden): nodes is forbidden: User "system:serviceaccount:infra-team:infra-k8s" cannot list resource "nodes" in API group "" at the cluster scope
command terminated with exit code 1
# 파드에 대한 권한은 있지만, 노드에 대한 권한은 없는 모습
# 노드는 클러스터 롤 범위에 있기 때문이다.!
$k1 auth can-i get pods
yes
$k1 auth can-i get nodes
Warning: resource 'nodes' is not namespace scoped
yes
$k1 auth can-i get no
Warning: resource 'nodes' is not namespace scoped
yes
# 리소스 삭제
$kubectl delete ns dev-team infra-team
namespace "dev-team" deleted
namespace "infra-team" deletedEKS 인증/인가
이론
이제, EKS 의 인증/인가 단계에 대해 실습합니다. 아래의 그림과 설명은 유튜브 영상에서 확인할 수 있습니다.
https://youtu.be/bksogA-WXv8?t=669
EKS는 Webhook, OIDC, Service Account을 지원한다.
아래는 RBAC에 대한 설명이다.
이제 EKS의 인증 인가 체계에 대해 알아본다.
핵심은 인증 인가 단계를 AWS IAM 을 통해 진행한다.! 어떻게 진행할 수 있는 지 확인해보면 다음과 같다.
먼저, EKS의 작업용 PC에서 쿠버네티스 명령을 날리면, 자동으로 .kubeconfig 에 입력되어 있는 eks get-token 명령을 실행된다. .
#!/bin/bash
# AWS에서 제공해준 EKS config 파일이다.
read -r -d '' KUBECONFIG <<EOF
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: $certificate_data
server: $cluster_endpoint
name: arn:aws:eks:$region_code:$account_id:cluster/$cluster_name
contexts:
- context:
cluster: arn:aws:eks:$region_code:$account_id:cluster/$cluster_name
user: arn:aws:eks:$region_code:$account_id:cluster/$cluster_name
name: arn:aws:eks:$region_code:$account_id:cluster/$cluster_name
current-context: arn:aws:eks:$region_code:$account_id:cluster/$cluster_name
kind: Config
preferences: {}
users:
- name: arn:aws:eks:$region_code:$account_id:cluster/$cluster_name
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
command: aws
args:
- --region
- $region_code
- eks
- get-token
- --cluster-name
- $cluster_name
# - --role
# - "arn:aws:iam::$account_id:role/my-role"
# env:
# - name: "AWS_PROFILE"
# value: "aws-profile"
EOF
echo "${KUBECONFIG}" > ~/.kube/config해당 명령은 EKS service endpoint로 간다. 요청에 대한 응답으로 토큰값이 전달된다.
토큰값을 디코딩해보면, aws sts get-caller-identity 를 호출하는 pre-signed URL이다
이제 URL을 가지고 아래와 같은 구조로 인증 인가 단계가 진행된다.
쿠버네티스는 CA, bearer token, authenticating proxy 방법을 통해 API 요청을 허용한다. 여기서 EKS는 Bearer Token을 사용하는 것이다.
이후 토큰을 통해 아까 살펴봤던 쿠버네티스 인증단계의 webhook 인증을 선택한다. URL은 sts get-caller-identity 를 호출하고, 이는 AWS IAM에게 인증을 받아 아래와 같은 유저 아이디 혹은 Role에 대한 ARN을 반환받는다.
{
"UserId": "AIDASAMPLEUSERID",
"Account": "123456789012",
"Arn": "arn:aws:iam::123456789012:role/k8s-admin"
}위의 정보를 aws-auth 를 보낸다. aws-auth 는 쿠버네티스 내의 User, Group으로 맵핑하는 개체로 맵핑된 쿠버네티스 개체를 반환한다. 쿠버네티스 개체를 통해 RBAC 인가 단계를 거친 후 etcd에 접근한다.
- 요약 (박준환님이 정리해주셨다.)
- 핵심 : 인증은 AWS IAM, 인가는 K8S RBAC에서 처리
실습
먼저, RBAC 관련 krew 플러그인을 설치합니다.
$**kubectl krew install access-matrix rbac-tool rbac-view rolesum**플러그인 확인하기
$kubectl rbac-tool lookup system:masters
W0531 20:47:56.354831 9284 warnings.go:67] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
SUBJECT | SUBJECT TYPE | SCOPE | NAMESPACE | ROLE
+----------------+--------------+-------------+-----------+---------------+
system:masters | Group | ClusterRole | | cluster-admin
$kubectl rbac-tool lookup system:nodes # eks:node-bootstrapper
W0531 20:47:58.478713 9379 warnings.go:67] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
SUBJECT | SUBJECT TYPE | SCOPE | NAMESPACE | ROLE
+--------------+--------------+-------------+-----------+-----------------------+
system:nodes | Group | ClusterRole | | eks:node-bootstrapper
$kubectl rbac-tool lookup system:bootstrappers # eks:node-bootstrapper
W0531 20:48:02.171737 9432 warnings.go:67] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
SUBJECT | SUBJECT TYPE | SCOPE | NAMESPACE | ROLE
+----------------------+--------------+-------------+-----------+-----------------------+
system:bootstrappers | Group | ClusterRole | | eks:node-bootstrapper
$kubectl describe ClusterRole eks:node-bootstrapper
Name: eks:node-bootstrapper
Labels: eks.amazonaws.com/component=node
Annotations: <none>
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
certificatesigningrequests.certificates.k8s.io/selfnodeserver [] [] [create]
$kubectl rbac-tool whoami
{Username: "kubernetes-admin",
UID: "aws-iam-authenticator:871103481195:AIDA4VUOQIVV5CHOU2JOK",
Groups: ["system:masters",
"system:authenticated"],
Extra: {accessKeyId: ["AKIA4VUOQIVV2CPMGKLE"],
arn: ["arn:aws:iam::871103481195:user/EKS-study"],
canonicalArn: ["arn:aws:iam::871103481195:user/EKS-study"],
principalId: ["AIDA4VUOQIVV5CHOU2JOK"],
sessionName: [""]}}
$kubectl rolesum aws-node -n kube-system
ServiceAccount: kube-system/aws-node
Secrets:
Policies:
• [CRB] */aws-node ⟶ [CR] */aws-node
Resource Name Exclude Verbs G L W C U P D DC
*.extensions [*] [-] [-] ✖ ✔ ✔ ✖ ✖ ✖ ✖ ✖
eniconfigs.crd.k8s.amazonaws.com [*] [-] [-] ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖
events.[,events.k8s.io] [*] [-] [-] ✖ ✔ ✖ ✔ ✖ ✔ ✖ ✖
namespaces [*] [-] [-] ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖
nodes [*] [-] [-] ✔ ✔ ✔ ✖ ✔ ✖ ✖ ✖
pods [*] [-] [-] ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖
$kubectl rolesum -k User system:kube-proxy
User: system:kube-proxy
Policies:
• [CRB] */system:node-proxier ⟶ [CR] */system:node-proxier
Resource Name Exclude Verbs G L W C U P D DC
endpoints [*] [-] [-] ✖ ✔ ✔ ✖ ✖ ✖ ✖ ✖
endpointslices.discovery.k8s.io [*] [-] [-] ✖ ✔ ✔ ✖ ✖ ✖ ✖ ✖
events.[,events.k8s.io] [*] [-] [-] ✖ ✖ ✖ ✔ ✔ ✔ ✖ ✖
nodes [*] [-] [-] ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖
services [*] [-] [-] ✖ ✔ ✔ ✖ ✖ ✖ ✖ ✖
$kubectl rolesum -k Group system:masters
Group: system:masters
Policies:
• [CRB] */cluster-admin ⟶ [CR] */cluster-admin
Resource Name Exclude Verbs G L W C U P D DC
*.* [*] [-] [-] ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔
rbac-view 실행
$kubectl rbac-view
...
INFO[0060] Built Matrix for ClusterRoles
INFO[0064] Built Matrix for Roles
INFO[0064] Matrix for json built
$echo -e "RBAC View Web http://$(curl -s ipinfo.io/ip):8800"
RBAC View Web http://3.36.103.16:8800위의 출력된 URL에 접속하면 아래와 같이 UI를 통해 RBAC를 확인할 수 있다.
인증/인가 완벽 분석 해보기
아래에서는 위에서 설명한 EKS의 인증 인가체계를 코드를 통해 상세하게 분석합니다.
kubectl 명령 → aws eks get-token → EKS Service endpoint(STS)에 토큰 요청 ⇒ 응답값 디코드(Pre-Signed URL 이며 GetCallerIdentity..)
$aws sts get-caller-identity --query Arn
"arn:aws:iam::871103481195:user/EKS-study"
$cat ~/.kube/config | yh
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJek1EVXpN...
server: https://BF69CC8DDDFB36E86FE01E52B6F5641B.gr7.ap-northeast-2.eks.amazonaws.com
name: myeks.ap-northeast-2.eksctl.io
contexts:
- context:
cluster: myeks.ap-northeast-2.eksctl.io
GET-TOKEN() GET-TOKEN()
user: EKS-study@myeks.ap-northeast-2.eksctl.io
name: kane
current-context: kane
kind: Config
preferences: {}
users:
- name: EKS-study@myeks.ap-northeast-2.eksctl.io
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- eks
- get-token
- --output
- json
- --cluster-name
- myeks
- --region
- ap-northeast-2
command: aws
env:
- name: AWS_STS_REGIONAL_ENDPOINTS
value: regional
interactiveMode: IfAvailable
provideClusterInfo: false
$aws eks get-token --cluster-name $CLUSTER_NAME | jq
{
"kind": "ExecCredential",
"apiVersion": "client.authentication.k8s.io/v1beta1",
"spec": {},
"status": {
"expirationTimestamp": "2023-05-31T12:04:36Z",
"token": "k8s-aws-v1.aHR0cHM6Ly9zdHMuYXAtbm9ydGhlYXN0LTIuYW1hem9uYXdzLmNvbS8_QWN0aW9uPUdldENhb..."
}
}
$aws eks get-token --cluster-name $CLUSTER_NAME | jq -r '.status.token'
k8s-aws-v1.aHR0cHM6Ly9zdHMuYXAtbm9ydGhlYXN0LTIuYW1hem9uYXdzLmNvbS8_QWN0aW9uPUdldENhbGxlcklkZW50aXR5JlZlcnNpb249MjAxMS0wNi0xNSZYLUFtei1BbGdvcml0aG09QVdTNC1ITUFDL...
토큰을 변환한 모습
EKS API는 Token Review 를 Webhook token authenticator에 요청 ⇒ (STS GetCallerIdentity 호출) AWS IAM 해당 호출 인증 완료 후 User/Role에 대한 ARN 반환
$kubectl api-resources | grep authentication
tokenreviews authentication.k8s.io/v1 false TokenReview
$kubectl explain tokenreviews
KIND: TokenReview
VERSION: authentication.k8s.io/v1
DESCRIPTION:
TokenReview attempts to authenticate a token to a known user. Note:
TokenReview requests may be cached by the webhook token authenticator
plugin in the kube-apiserver.
...
이제 쿠버네티스 RBAC 인가를 처리합니다. 개인적인 생각이지만 플랫폼간 인증 이외에 인가까지 처리 통합은 쉽지 않은 것 같습니다
$kubectl api-resources | grep Webhook
mutatingwebhookconfigurations admissionregistration.k8s.io/v1 false MutatingWebhookConfiguration
validatingwebhookconfigurations admissionregistration.k8s.io/v1 false ValidatingWebhookConfiguration
$kubectl get validatingwebhookconfigurations
NAME WEBHOOKS AGE
aws-load-balancer-webhook 3 46m
eks-aws-auth-configmap-validation-webhook 1 72m
kube-prometheus-stack-admission 1 45m
vpc-resource-validating-webhook 2 72m
$kubectl get validatingwebhookconfigurations eks-aws-auth-configmap-validation-webhook -o yaml | kubectl neat | yh
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: eks-aws-auth-configmap-validation-webhook
webhooks:
- admissionReviewVersions:
- v1
clientConfig:
caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJek1EVXpN..
url: https://127.0.0.1:21375/validate
failurePolicy: Ignore
matchPolicy: Equivalent
name: eks-aws-auth-configmap-validation-webhook.amazonaws.com
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
rules:
- apiGroups:
- ""
apiVersions:
- v1
operations:
- UPDATE
resources:
- configmaps
scope: '*'
sideEffects: None
timeoutSeconds: 5
$kubectl get cm -n kube-system aws-auth -o yaml | kubectl neat | yh
apiVersion: v1
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::871103481195:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-QR6CCYVFGKRS
username: system:node:{{EC2PrivateDNSName}}
kind: ConfigMap
metadata:
name: aws-auth
namespace: kube-system
$kubectl rbac-tool whoami
{Username: "kubernetes-admin",
UID: "aws-iam-authenticator:871103481195:AIDA4VUOQIVV5CHOU2JOK",
Groups: ["system:masters",
"system:authenticated"],
Extra: {accessKeyId: ["AKIA4VUOQIVV2CPMGKLE"],
arn: ["arn:aws:iam::871103481195:user/EKS-study"],
canonicalArn: ["arn:aws:iam::871103481195:user/EKS-study"],
principalId: ["AIDA4VUOQIVV5CHOU2JOK"],
sessionName: [""]}}
#system:masters , system:authenticated 그룹의 정보 확인
$kubectl rbac-tool lookup system:masters
W0531 21:06:13.318205 11330 warnings.go:67] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
SUBJECT | SUBJECT TYPE | SCOPE | NAMESPACE | ROLE
+----------------+--------------+-------------+-----------+---------------+
system:masters | Group | ClusterRole | | cluster-admin
$kubectl rbac-tool lookup system:authenticated
W0531 21:06:14.357002 11384 warnings.go:67] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
SUBJECT | SUBJECT TYPE | SCOPE | NAMESPACE | ROLE
+----------------------+--------------+-------------+-----------+----------------------------------+
system:authenticated | Group | ClusterRole | | eks:podsecuritypolicy:privileged
system:authenticated | Group | ClusterRole | | system:discovery
system:authenticated | Group | ClusterRole | | system:public-info-viewer
system:authenticated | Group | ClusterRole | | system:basic-user
$kubectl rolesum -k Group system:masters
Group: system:masters
Policies:
• [CRB] */cluster-admin ⟶ [CR] */cluster-admin
Resource Name Exclude Verbs G L W C U P D DC
*.* [*] [-] [-] ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔
$kubectl rolesum -k Group system:authenticated
W0531 21:06:16.145547 11491 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
Group: system:authenticated
Policies:
• [CRB] */eks:podsecuritypolicy:authenticated ⟶ [CR] */eks:podsecuritypolicy:privileged
Name PRIV RO-RootFS Volumes Caps SELinux RunAsUser FSgroup SUPgroup
eks.privileged True False [*] [*] RunAsAny RunAsAny RunAsAny RunAsAny
• [CRB] */system:basic-user ⟶ [CR] */system:basic-user
Resource Name Exclude Verbs G L W C U P D DC
selfsubjectaccessreviews.authorization.k8s.io [*] [-] [-] ✖ ✖ ✖ ✔ ✖ ✖ ✖ ✖
selfsubjectrulesreviews.authorization.k8s.io [*] [-] [-] ✖ ✖ ✖ ✔ ✖ ✖ ✖ ✖
• [CRB] */system:discovery ⟶ [CR] */system:discovery
• [CRB] */system:public-info-viewer ⟶ [CR] */system:public-info-viewer
#system:masters 그룹이 사용 가능한 클러스터 롤 확인 : cluster-admin
$kubectl describe clusterrolebindings.rbac.authorization.k8s.io cluster-admin
Name: cluster-admin
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
Role:
Kind: ClusterRole
Name: cluster-admin
Subjects:
Kind Name Namespace
---- ---- ---------
Group system:masters
$kubectl describe clusterrole cluster-admin
Name: cluster-admin
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
*.* [] [] [*]
[*] [] [*]
$kubectl describe ClusterRole system:discovery
Name: system:discovery
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
[/api/*] [] [get]
[/api] [] [get]
[/apis/*] [] [get]
[/apis] [] [get]
[/healthz] [] [get]
[/livez] [] [get]
[/openapi/*] [] [get]
[/openapi] [] [get]
[/readyz] [] [get]
[/version/] [] [get]
[/version] [] [get]
$kubectl describe ClusterRole system:public-info-viewer
Name: system:public-info-viewer
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
[/healthz] [] [get]
[/livez] [] [get]
[/readyz] [] [get]
[/version/] [] [get]
[/version] [] [get]
$kubectl describe ClusterRole system:basic-user
Name: system:basic-user
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
selfsubjectaccessreviews.authorization.k8s.io [] [] [create]
selfsubjectrulesreviews.authorization.k8s.io [] [] [create]
$kubectl describe ClusterRole eks:podsecuritypolicy:privileged
Name: eks:podsecuritypolicy:privileged
Labels: eks.amazonaws.com/component=pod-security-policy
kubernetes.io/cluster-service=true
Annotations: <none>
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
podsecuritypolicies.policy [] [eks.privileged] [use]
데브온스 신입 사원을 위한 myeks-bastion-2에 설정해보기
- [main bastion] IAM 사용자 생성
$aws iam create-user --user-name testuser
{
"User": {
"Path": "/",
"UserName": "testuser",
"UserId": "AIDA4VUOQIVVX254SX7ZZ",
"Arn": "arn:aws:iam::871103481195:user/testuser",
"CreateDate": "2023-05-31T12:07:09+00:00"
}
}
#사용자에게 프로그래밍 방식 액세스 권한 부여
$aws iam create-access-key --user-name testuser
{
"AccessKey": {
"UserName": "testuser",
"AccessKeyId": "AKIA4VUOQIVV2DMRFWMX",
"Status": "Active",
"SecretAccessKey": "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",
"CreateDate": "2023-05-31T12:07:16+00:00"
}
}
$aws iam attach-user-policy --policy-arn arn:aws:iam::aws:policy/AdministratorAccess --user-name testuser
$aws sts get-caller-identity --query Arn
"arn:aws:iam::871103481195:user/EKS-study"
$aws ec2 describe-instances --query "Reservations[*].Instances[*].{PublicIPAdd:PublicIpAddress,PrivateIPAdd:PrivateIpAddress,InstanceName:Tags[?Key=='Name']|[0].Value,Status:State.Name}" --filters Name=instance-state-name,Values=running --output table
-----------------------------------------------------------------------
| DescribeInstances |
+----------------------+----------------+------------------+----------+
| InstanceName | PrivateIPAdd | PublicIPAdd | Status |
+----------------------+----------------+------------------+----------+
| myeks-ng1-Node | 192.168.3.81 | 43.200.177.234 | running |
| myeks-ng1-Node | 192.168.2.117 | 3.38.186.134 | running |
| myeks-bastion-EC2-2 | 192.168.1.200 | 3.38.105.241 | running |
| myeks-bastion-EC2 | 192.168.1.100 | 3.36.103.16 | running |
| myeks-ng1-Node | 192.168.1.102 | 43.201.254.218 | running |
+----------------------+----------------+------------------+----------+
- 이제 bastion2 에 접속하여, 생성한 testuser AWS config 설정을 하고, 권한을 확인한다.
[root@myeks-bastion-2 ~]# bastion 2로 접속
$aws sts get-caller-identity --query Arn
Unable to locate credentials. You can configure credentials by running "aws configure".
$aws configure
AWS Access Key ID [None]: AKIA4VUOQIVV2DMRFWMX
AWS Secret Access Key [None]: 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
Default region name [None]: ap-northeast-2
Default output format [None]: json
$aws sts get-caller-identity --query Arn
"arn:aws:iam::871103481195:user/testuser"
# 접속 실패
$kubectl get node -v6
I0531 21:11:22.552208 1798 round_trippers.go:553] GET http://localhost:8080/api?timeout=32s in 0 milliseconds
E0531 21:11:22.552311 1798 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
I0531 21:11:22.552333 1798 cached_discovery.go:120] skipped caching discovery info due to Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
I0531 21:11:22.552554 1798 round_trippers.go:553] GET http://localhost:8080/api?timeout=32s in 0 milliseconds
E0531 21:11:22.552594 1798 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
I0531 21:11:22.553487 1798 cached_discovery.go:120] skipped caching discovery info due to Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
I0531 21:11:22.553511 1798 shortcut.go:100] Error loading discovery information: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
I0531 21:11:22.554197 1798 round_trippers.go:553] GET http://localhost:8080/api?timeout=32s in 0 milliseconds
E0531 21:11:22.554252 1798 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
I0531 21:11:22.555354 1798 cached_discovery.go:120] skipped caching discovery info due to Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
I0531 21:11:22.555575 1798 round_trippers.go:553] GET http://localhost:8080/api?timeout=32s in 0 milliseconds
E0531 21:11:22.555682 1798 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
I0531 21:11:22.556788 1798 cached_discovery.go:120] skipped caching discovery info due to Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
I0531 21:11:22.556994 1798 round_trippers.go:553] GET http://localhost:8080/api?timeout=32s in 0 milliseconds
E0531 21:11:22.557035 1798 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
I0531 21:11:22.558150 1798 cached_discovery.go:120] skipped caching discovery info due to Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
I0531 21:11:22.558214 1798 helpers.go:264] Connection error: Get http://localhost:8080/api?timeout=32s: dial tcp 127.0.0.1:8080: connect: connection refused
The connection to the server localhost:8080 was refused - did you specify the right host or port?
# kube config 파일이 없음!
$ls ~/.kube
ls: cannot access /root/.kube: No such file or directory
- [myeks-bastion] testuser에 system:masters 그룹 부여로 EKS 관리자 수준 권한 설정
# 방안1 : eksctl 사용 >> iamidentitymapping 실행 시 aws-auth 컨피그맵 작성해줌
# tesk 유저 권한부여
## Creates a mapping from IAM role or user to Kubernetes user and groups
$eksctl create iamidentitymapping --cluster $CLUSTER_NAME --username testuser --group system:masters --arn arn:aws:iam::$ACCOUNT_ID:user/testuser
2023-05-31 21:12:06 [ℹ] checking arn arn:aws:iam::871103481195:user/testuser against entries in the auth ConfigMap
2023-05-31 21:12:06 [ℹ] adding identity "arn:aws:iam::871103481195:user/testuser" to auth ConfigMap
$kubectl get cm -n kube-system aws-auth -o yaml | kubectl neat | yh
apiVersion: v1
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::871103481195:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-QR6CCYVFGKRS
username: system:node:{{EC2PrivateDNSName}}
mapUsers: |
- groups:
- system:masters
userarn: arn:aws:iam::871103481195:user/testuser
username: testuser
kind: ConfigMap
metadata:
name: aws-auth
namespace: kube-system
$eksctl get iamidentitymapping --cluster $CLUSTER_NAME
ARN USERNAME GROUPS ACCOUNT
arn:aws:iam::871103481195:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-QR6CCYVFGKRS system:node:{{EC2PrivateDNSName}} system:bootstrappers,system:nodes
arn:aws:iam::871103481195:user/testuser testuser system:masters
- 이제 bastion2에 접속하여, eks를 업데이트하고 kubeconfig 파일을 비교 확인한다.
[root@myeks-bastion-2 ~]#
# 업데이트
$aws eks update-kubeconfig --name $CLUSTER_NAME --user-alias testuser
Added new context testuser to /root/.kube/config
# 첫번째 bastic ec2의 config와 비교해보자
$cat ~/.kube/config | yh
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SU
server: https://BF69CC8DDDFB36E86FE01E52B6F5641B.gr7.ap-northeast-2.eks.amazonaws.com
name: arn:aws:eks:ap-northeast-2:871103481195:cluster/myeks
contexts:
- context:
cluster: arn:aws:eks:ap-northeast-2:871103481195:cluster/myeks
user: testuser
name: testuser
current-context: testuser
kind: Config
preferences: {}
users:
- name: testuser
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- --region
- ap-northeast-2
- eks
- get-token
- --cluster-name
- myeks
- --output
- json
command: aws아래는 main bastion 의 kube config 파일이다. 대부분 같은 것을 확인할 수 있다.
#bation 2와 kubeconfig file 확인
$cat ~/.kube/config | yh
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JS..
server: https://BF69CC8DDDFB36E86FE01E52B6F5641B.gr7.ap-northeast-2.eks.amazonaws.com
name: myeks.ap-northeast-2.eksctl.io
contexts:
- context:
cluster: myeks.ap-northeast-2.eksctl.io
user: EKS-study@myeks.ap-northeast-2.eksctl.io
name: kane
current-context: kane
kind: Config
preferences: {}
users:
- name: EKS-study@myeks.ap-northeast-2.eksctl.io
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- eks
- get-token
...
#bastion2와 비교
$kubectl rbac-tool whoami
{Username: "kubernetes-admin",
UID: "aws-iam-authenticator:871103481195:AIDA4VUOQIVV5CHOU2JOK",
Groups: ["system:masters",
"system:authenticated"],
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::871103481195:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-QR6CCYVFGKRS
username: system:node:{{EC2PrivateDNSName}}
mapUsers: |
- groups:
- system:authenticated
userarn: arn:aws:iam::871103481195:user/testuser
username: testuser
...- [myeks-bastion] testuser 의 Group 변경(system:masters → system:authenticated)으로 RBAC 동작 확인
- 데브옵스 신입, config map 수정
# Please edit the object below. Lines beginning with a '#' will be ignored, # and an empty file will abort the edit. If an error occurs while saving this file will be # reopened with the relevant failures. # apiVersion: v1 data: mapRoles: | - groups: - system:bootstrappers - system:nodes rolearn: arn:aws:iam::871103481195:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-QR6CCYVFGKRS username: system:node:{{EC2PrivateDNSName}} mapUsers: | - groups: - system:authenticated userarn: arn:aws:iam::871103481195:user/testuser username: testuser kind: ConfigMap metadata: creationTimestamp: "2023-05-31T11:03:14Z" name: aws-auth namespace: kube-system resourceVersion: "19809" uid: 7e786e3c-cf39-4ade-a86f-ff92a4bcbb39
$kubectl edit cm -n kube-system aws-auth
Edit cancelled, no changes made.
$kubectl edit cm -n kube-system aws-auth
configmap/aws-auth edited
$eksctl get iamidentitymapping --cluster $CLUSTER_NAME
ARN USERNAME GROUPS ACCOUNT
arn:aws:iam::871103481195:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-QR6CCYVFGKRS system:node:{{EC2PrivateDNSName}} system:bootstrappers,system:nodes
arn:aws:iam::871103481195:user/testuser testuser system:authenticated
$eksctl get iamidentitymapping --cluster $CLUSTER_NAME
ARN USERNAME GROUPS ACCOUNT
arn:aws:iam::871103481195:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-QR6CCYVFGKRS system:node:{{EC2PrivateDNSName}} system:bootstrappers,system:nodes
arn:aws:iam::871103481195:user/testuser testuser system:authenticated
$kubectl edit cm -n kube-system aws-auth
Edit cancelled, no changes made.
$kubectl edit cm -n kube-system aws-auth
Edit cancelled, no changes made.
$kubectl get node -v6
I0531 21:18:20.686745 13429 loader.go:374] Config loaded from file: /root/.kube/config
I0531 21:18:21.481898 13429 round_trippers.go:553] GET https://BF69CC8DDDFB36E86FE01E52B6F5641B.gr7.ap-northeast-2.eks.amazonaws.com/api/v1/nodes?limit=500 200 OK in 773 milliseconds
NAME STATUS ROLES AGE VERSION
ip-192-168-1-102.ap-northeast-2.compute.internal Ready <none> 74m v1.24.13-eks-0a21954
ip-192-168-2-117.ap-northeast-2.compute.internal Ready <none> 74m v1.24.13-eks-0a21954
ip-192-168-3-81.ap-northeast-2.compute.internal Ready <none> 74m v1.24.13-eks-0a21954
$kubectl api-resources -v5
NAME SHORTNAMES APIVERSION NAMESPACED KIND
bindings v1 true Binding
componentstatuses cs v1 false ComponentStatus
configmaps cm v1 true ConfigMap
endpoints ep v1 true Endpoints
...
securitygrouppolicies sgp vpcresources.k8s.aws/v1beta1 true SecurityGroupPolicy
$eksctl get iamidentitymapping --cluster $CLUSTER_NAME
ARN USERNAME GROUPS ACCOUNT
arn:aws:iam::871103481195:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-QR6CCYVFGKRS system:node:{{EC2PrivateDNSName}} system:bootstrappers,system:nodes
arn:aws:iam::871103481195:user/testuser testuser system:authenticated
- bastion 2
kubectl사용 확인
$kubectl ns default
Context "testuser" modified.
Active namespace is "default".
(testuser:default) [root@myeks-bastion-2 ~]#
$kubectl get node -v6
I0531 21:13:59.725066 2078 loader.go:373] Config loaded from file: /root/.kube/config
I0531 21:14:00.542215 2078 round_trippers.go:553] GET https://BF69CC8DDDFB36E86FE01E52B6F5641B.gr7.ap-northeast-2.eks.amazonaws.com/api/v1/nodes?limit=500 200 OK in 809 milliseconds
NAME STATUS ROLES AGE VERSION
ip-192-168-1-102.ap-northeast-2.compute.internal Ready <none> 70m v1.24.13-eks-0a21954
ip-192-168-2-117.ap-northeast-2.compute.internal Ready <none> 70m v1.24.13-eks-0a21954
ip-192-168-3-81.ap-northeast-2.compute.internal Ready <none> 70m v1.24.13-eks-0a21954
$kubectl krew install rbac-tool && kubectl rbac-tool whoami
Updated the local copy of plugin index.
Installing plugin: rbac-tool
Installed plugin: rbac-tool
\
| Use this plugin:
| kubectl rbac-tool
| Documentation:
| https://github.com/alcideio/rbac-tool
/
WARNING: You installed plugin "rbac-tool" from the krew-index plugin repository.
These plugins are not audited for security by the Krew maintainers.
Run them at your own risk.
{Username: "testuser",
UID: "aws-iam-authenticator:871103481195:AIDA4VUOQIVVX254SX7ZZ",
Groups: ["system:masters",
"system:authenticated"],
Extra: {accessKeyId: ["AKIA4VUOQIVV2DMRFWMX"],
arn: ["arn:aws:iam::871103481195:user/testuser"],
canonicalArn: ["arn:aws:iam::871103481195:user/testuser"],
principalId: ["AIDA4VUOQIVVX254SX7ZZ"],
sessionName: [""]}}
# 노드에 대한 권한은 클러스터 롤이기에, 차단된 모습이다.
$kubectl get node -v6
I0531 21:16:11.465425 2208 loader.go:373] Config loaded from file: /root/.kube/config
I0531 21:16:12.269649 2208 round_trippers.go:553] GET https://BF69CC8DDDFB36E86FE01E52B6F5641B.gr7.ap-northeast-2.eks.amazonaws.com/api/v1/nodes?limit=500 403 Forbidden in 782 milliseconds
I0531 21:16:12.269935 2208 helpers.go:246] server response object: [{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "nodes is forbidden: User \"testuser\" cannot list resource \"nodes\" in API group \"\" at the cluster scope",
"reason": "Forbidden",
"details": {
"kind": "nodes"
},
"code": 403
}]
Error from server (Forbidden): nodes is forbidden: User "testuser" cannot list resource "nodes" in API group "" at the cluster scope
$kubectl api-resources -v5
NAME SHORTNAMES APIVERSION NAMESPACED KIND
bindings v1 true Binding
componentstatuses cs v1 false ComponentStatus
configmaps cm v1 true ConfigMap
endpoints ep v1 true Endpoints
events ev v1 true Event
...- [myeks-bastion]에서 testuser IAM 맵핑 삭제
$eksctl delete iamidentitymapping --cluster $CLUSTER_NAME --arn arn:aws:iam::$ACCOUNT_ID:user/testuser
2023-05-31 21:21:05 [ℹ] removing identity "arn:aws:iam::871103481195:user/testuser" from auth ConfigMap (username = "testuser", groups = ["system:authenticated"])
$eksctl get iamidentitymapping --cluster $CLUSTER_NAME
ARN USERNAME GROUPS ACCOUNT
arn:aws:iam::871103481195:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-QR6CCYVFGKRS system:node:{{EC2PrivateDNSName}} system:bootstrappers,system:nodes
$kubectl get cm -n kube-system aws-auth -o yaml | yh
apiVersion: v1
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::871103481195:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-QR6CCYVFGKRS
username: system:node:{{EC2PrivateDNSName}}
mapUsers: |
[]
kind: ConfigMap
metadata:
creationTimestamp: "2023-05-31T11:03:14Z"
name: aws-auth
namespace: kube-system
resourceVersion: "21204"
uid: 7e786e3c-cf39-4ade-a86f-ff92a4bcbb39IRSA
IAM Role for Service Account의 약자로 각 파드 당 권한을 IAM 권한을 별도로 부여하는 방법이다.
운영하다보면 사용자가 아닌 하나의 서비스가 AWS에 접근해야 하는 일이 생긴다. ex) RDS or S3 스냅샷을 남기는 파드, 이때 EC2 Instance Profile을 사용하여 노드의 권한을 모든 파드가 공유하면 최소 권한 원칙에 위배된다. 파드마다 역할이 다르니 권한 부여 단위를 파드로 두는 것이다. 파드가 특정 IAM 역할로 Assume 할때 토큰을 AWS에 전송하고, AWS는 토큰과 EKS IdP를 통해 해당 IAM 역할을 사용할 수 있는지 검증한다.
EC2 instance Profile
설정예시를 참고해서 추후 진행
# 설정 예시 1 : eksctl 사용 시
**eksctl create** cluster --name $CLUSTER_NAME ... **--external-dns-access --full-ecr-access --asg-access**
# 설정 예시 2 : eksctl로 yaml 파일로 노드 생성 시
**cat myeks.yaml | yh**
...
managedNodeGroups:
- amiFamily: AmazonLinux2
iam:
withAddonPolicies:
albIngress: false
appMesh: false
appMeshPreview: false
**autoScaler: true**
awsLoadBalancerController: false
**certManager: true**
**cloudWatch: true**
ebs: false
efs: false
**externalDNS: true**
fsx: false
**imageBuilder: true**
xRay: false
...
# 설정 예시 3 : 테라폼
...IRSA
AWS 자원에 접근하는 파드 생성해보기
s3 ls 명령어를 사용하는 파드 생성
$cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: eks-iam-test1
spec:
containers:
- name: my-aws-cli
image: amazon/aws-cli:latest
args: ['s3', 'ls']
restartPolicy: Never
automountServiceAccountToken: false
EOF
pod/eks-iam-test1 created
#확인
$kubectl get pod
NAME READY STATUS RESTARTS AGE
eks-iam-test1 0/1 ContainerCreating 0 1s
$kubectl describe pod
Name: eks-iam-test1
Namespace: default
Priority: 0
Service Account: default
Node: ip-192-168-3-81.ap-northeast-2.compute.internal/192.168.3.81
Start Time: Wed, 31 May 2023 21:28:03 +0900
Labels: <none>
Annotations: kubernetes.io/psp: eks.privileged
Status: Pending
IP:
IPs: <none>
Containers:
my-aws-cli:
Container ID:
Image: amazon/aws-cli:latest
Image ID:
Port: <none>
Host Port: <none>
Args:
s3
ls
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts: <none>
...
#로그 확인
$kubectl logs eks-iam-test1
Error from server (BadRequest): container "my-aws-cli" in pod "eks-iam-test1" is waiting to start: ContainerCreating
#파드1 삭제
$kubectl logs eks-iam-test1
An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied
S3(ListBuckets)로그확인(접속하려하지만, 권한이 없어 접근에 실패한 것을 확인할 수 있다.)
{
"eventVersion": "1.08",
"userIdentity": {
"type": "AssumedRole",
"principalId": "AROASUCZUNGONCRSO3PBP:access-analyzer",
"arn": "arn:aws:sts::180576610716:assumed-role/AWSServiceRoleForAccessAnalyzer/access-analyzer",
"accountId": "180576610716",
"accessKeyId": "ASIASUCZUNGOFEM7DDEV",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "AROASUCZUNGONCRSO3PBP",
"arn": "arn:aws:iam::180576610716:role/aws-service-role/access-analyzer.amazonaws.com/AWSServiceRoleForAccessAnalyzer",
"accountId": "180576610716",
"userName": "AWSServiceRoleForAccessAnalyzer"
},
"webIdFederationData": {},
"attributes": {
"creationDate": "2023-05-31T07:03:36Z",
"mfaAuthenticated": "false"
}
},
"invokedBy": "access-analyzer.amazonaws.com"
},
"eventTime": "2023-05-31T07:03:37Z",
"eventSource": "s3.amazonaws.com",
"eventName": "ListBuckets",
"awsRegion": "ap-northeast-2",
"sourceIPAddress": "access-analyzer.amazonaws.com",
"userAgent": "access-analyzer.amazonaws.com",
"requestParameters": {
"Host": "s3.ap-northeast-2.amazonaws.com"
},
"responseElements": null,
"additionalEventData": {
"SignatureVersion": "SigV4",
"CipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
"bytesTransferredIn": 0,
"AuthenticationMethod": "AuthHeader",
"x-amz-id-2": "3euU4IpaPy9deQh..=",
"bytesTransferredOut": 661
},
"requestID": "CGKW0AV31SR85514",
"eventID": "5afa53c5-31e4-4e22-88d3-324625cccdd0",
"readOnly": true,
"eventType": "AwsApiCall",
"managementEvent": true,
"recipientAccountId": "180576610716",
"eventCategory": "Management"
}이제 서비스 어카운트를 기반으로 접근 권한을 부여해서 진행해본다.
위와 같이 S3에 접근하는 파드 생성
$cat <<EOF | kubectl apply -f -
> apiVersion: v1
> kind: Pod
> metadata:
> name: eks-iam-test2
> spec:
> containers:
> - name: my-aws-cli
> image: amazon/aws-cli:latest
> command: ['sleep', '36000']
> restartPolicy: Never
> EOF
pod/eks-iam-test2 created
$kubectl get pod
NAME READY STATUS RESTARTS AGE
eks-iam-test2 1/1 Running 0 6s
$kubectl describe pod
Name: eks-iam-test2
Namespace: default
...
Containers:
my-aws-cli:
Image: amazon/aws-cli:latest
Image ID: docker.io/amazon/aws-cli@sha256:21e6273f0025755abfc842ca39e8ef4fed3d9d2ce61d93bb16ce86a6c1668ae5
Port: <none>
Host Port: <none>
Command:
sleep
36000
State: Running
Started: Wed, 31 May 2023 21:30:32 +0900
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4x9qc (ro)
...
# 접근 실패
$kubectl exec -it eks-iam-test2 -- aws s3 ls
# 서비스 어카운트 토큰 확인
SA_TOKEN=$(kubectl exec -it eks-iam-test2 -- cat /var/run/secrets/kubernetes.io/serviceaccount/token)
echo $SA_TOKEN
An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied
command terminated with exit code 254
IAM 서비스 어카운트 생성
$eksctl create iamserviceaccount \
> --name my-sa \
> --namespace default \
> --cluster $CLUSTER_NAME \
> --approve \
> --attach-policy-arn $(aws iam list-policies --query 'Policies[?PolicyName==`AmazonS3ReadOnlyAccess`].Arn' --output text)
2023-05-31 21:40:06 [ℹ] 1 existing iamserviceaccount(s) (kube-system/aws-load-balancer-controller) will be excluded
2023-05-31 21:40:06 [ℹ] 1 iamserviceaccount (default/my-sa) was included (based on the include/exclude rules)
2023-05-31 21:40:06 [!] serviceaccounts that exist in Kubernetes will be excluded, use --override-existing-serviceaccounts to override
2023-05-31 21:40:06 [ℹ] 1 task: {
2 sequential sub-tasks: {
create IAM role for serviceaccount "default/my-sa",
create serviceaccount "default/my-sa",
} }2023-05-31 21:40:06 [ℹ] building iamserviceaccount stack "eksctl-myeks-addon-iamserviceaccount-default-my-sa"
2023-05-31 21:40:07 [ℹ] deploying stack "eksctl-myeks-addon-iamserviceaccount-default-my-sa"
2023-05-31 21:40:08 [ℹ] waiting for CloudFormation stack "eksctl-myeks-addon-iamserviceaccount-default-my-sa"
2023-05-31 21:40:38 [ℹ] waiting for CloudFormation stack "eksctl-myeks-addon-iamserviceaccount-default-my-sa"
2023-05-31 21:40:38 [ℹ] created serviceaccount "default/my-sa"
$eksctl get iamserviceaccount --cluster $CLUSTER_NAME
NAMESPACE NAME ROLE ARN
default my-sa arn:aws:iam::871103481195:role/eksctl-myeks-addon-iamserviceaccount-default-Role1-193TXMP0QLZSN
kube-system aws-load-balancer-controller arn:aws:iam::871103481195:role/eksctl-myeks-addon-iamserviceaccount-kube-sy-Role1-M4DYRCMI95LR
$kubectl get sa
NAME SECRETS AGE
default 0 107m
my-sa 0 32s
$kubectl describe sa my-sa
Name: my-sa
Namespace: default
Labels: app.kubernetes.io/managed-by=eksctl
Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::871103481195:role/eksctl-myeks-addon-iamserviceaccount-default-Role1-193TXMP0QLZSN
Image pull secrets: <none>
Mountable secrets: <none>
Tokens: <none>
Events: <none>
콘솔에서 정보 확인
만든 service account
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::871103481195:oidc-provider/oidc.eks.ap-northeast-2.amazonaws.com/id/BF69CC8DDDFB36E86FE01E52B6F5641B"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.ap-northeast-2.amazonaws.com/id/BF69CC8DDDFB36E86FE01E52B6F5641B:sub": "system:serviceaccount:default:my-sa",
"oidc.eks.ap-northeast-2.amazonaws.com/id/BF69CC8DDDFB36E86FE01E52B6F5641B:aud": "sts.amazonaws.com"
}
}
}
]
}테스트를 위해 위와 같이 파드 생성
→ S3 에 접근되는 것을 확인할 수 있다.
$cat <<EOF | kubectl apply -f -
> apiVersion: v1
> kind: Pod
> metadata:
> name: eks-iam-test3
> spec:
> serviceAccountName: **my-sa**
> containers:
> - name: my-aws-cli
> image: amazon/aws-cli:latest
> command: ['sleep', '36000']
> restartPolicy: Never
> EOF
pod/eks-iam-test3 created
$kubectl get mutatingwebhookconfigurations pod-identity-webhook -o yaml | kubectl neat | yh
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
name: pod-identity-webhook
webhooks:
- admissionReviewVersions:
- v1beta1
clientConfig:
caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakN...
url: https://127.0.0.1:23443/mutate
failurePolicy: Ignore
matchPolicy: Equivalent
name: iam-for-pods.amazonaws.com
reinvocationPolicy: IfNeeded
rules:
- apiGroups:
- ""
apiVersions:
- v1
operations:
- CREATE
resources:
- pods
scope: '*'
sideEffects: None
timeoutSeconds: 10
# 파드 확인
$kubectl get pod eks-iam-test3
NAME READY STATUS RESTARTS AGE
eks-iam-test3 1/1 Running 0 9s
# **Pod Identity Webhook**은 **mutating** webhook을 통해 아래 **Env 내용**과 **1개의 볼륨**을 추가함
$kubectl describe pod eks-iam-test3
Name: eks-iam-test3
Namespace: default
Priority: 0
**Service Account: my-sa**
...
Containers:
my-aws-cli:
Container ID: containerd://37b52c6ca91475c8b42f194fc21e32ad905ae8a982110443cbff910152be9264
Image: amazon/aws-cli:latest
Image ID: docker.io/amazon/aws-cli@sha256:21e6273f0025755abfc842ca39e8ef4fed3d9d2ce61d93bb16ce86a6c1668ae5
...
**Environment:
AWS_STS_REGIONAL_ENDPOINTS: regional
AWS_DEFAULT_REGION: ap-northeast-2
AWS_REGION: ap-northeast-2
AWS_ROLE_ARN: arn:aws:iam::...:role/eksctl-myeks-addon-iamserviceaccount-default-Role1-193..
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token**
Mounts:
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2gvz2 (ro)
...
**Volumes:**
**aws-iam-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 86400**
kube-api-access-2gvz2:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
...
$eksctl get iamserviceaccount --cluster $CLUSTER_NAME
NAMESPACE NAME ROLE ARN
default my-sa arn:aws:iam::871103481195:role/eksctl-myeks-addon-iamserviceaccount-default-Role1-193TXMP0QLZSN
kube-system aws-load-balancer-controller arn:aws:iam::871103481195:role/eksctl-myeks-addon-iamserviceaccount-kube-sy-Role1-M4DYRCMI95LR
테스트 진행
$kubectl exec -it eks-iam-test3 -- aws sts get-caller-identity --query Arn
"arn:aws:sts::871103481195:assumed-role/eksctl-myeks-addon-iamserviceaccount-default-Role1-193TXMP0QLZSN/botocore-session-1685537063"
# S3 접근 가능
$kubectl exec -it eks-iam-test3 -- aws s3 ls
2023-05-31 12:35:31 cf-templates-1pjjg014ag81h-ap-northeast-2
# 정책에서 S3 권한만 부여했으니,당연히 나머지 리소스는 접근하지 못한다.
# ec2 접근 불가
$kubectl exec -it eks-iam-test3 -- aws ec2 describe-instances --region ap-northeast-2
An error occurred (UnauthorizedOperation) when calling the DescribeInstances operation: You are not authorized to perform this operation.
command terminated with exit code 254
# vpc 접근 불가
$kubectl exec -it eks-iam-test3 -- aws ec2 describe-vpcs --region ap-northeast-2
An error occurred (UnauthorizedOperation) when calling the DescribeVpcs operation: You are not authorized to perform this operation.
command terminated with exit code 254
AWS CloudTrail 로그를 통해 접근한 것을 확인할 수 있다.
OWASP Kubernetes Top Ten
OWASP 에서 발표한 2022 쿠버네티스 환경 취약점 Top10 중 2가지에 대해 실습을 진행한다. 실습의 대부분은 악분님의 블로그를 참고한다.
https://malwareanalysis.tistory.com/607
https://malwareanalysis.tistory.com/606
Amazon EKS Best Practices Guide for Security
https://aws.github.io/aws-eks-best-practices/security/docs/
Securing Secrets
경험발표에서 공유해주신 하시코프에 Valut 시스템이다. 쿠버네티스의 보안을 지원한다.
- Valut Secret Operator on K8s - Youtube 개요 설치 실습 비교 GitHub
- VSO 실습 코드(Dynamic,PKI,Static) : https://github.com/hashicorp/vault-secrets-operator/tree/main/demo/infra/app
- VSO 실습 코드(Static) : https://github.com/hashicorp-education/learn-vault-secrets-operator/tree/main/vault
- Vault Static Secret 샘플 : https://developer.hashicorp.com/vault/tutorials/kubernetes/vault-secrets-operator
- VSO API Reference : https://developer.hashicorp.com/vault/docs/platform/k8s/vso/api-reference
파드/컨테이너 보안 컨텍스트
https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
참고링크
-
OAuth
제 3자의 서비스에게 계정에 대한 정보를 줄 때, 계정이 아닌 Access Token 을 줌으로 알맞는 권한만 제공하는 방법 -
참고 링크
- AWS
- [용찬호님] - EKS에서 쿠버네티스 포드의 IAM 권한 제어하기 - 링크 / AWS IAM Authenticator - 링크 / OIDC 인증 - 링크
- [커피고래님] - 인증 시리즈 X.509 HTTP인증 OpenID Connect Webhook Proxy인증 , Admisstion Control
- [Youtube] 생활코딩 OAuth 2.0 - 링크
- [learnk8s] User and workload identities in Kubernetes - 링크
- [Youtube] 쿠버네티스 해킹과 방어 (데모 포함)
- [Youtube] Kubecon 2023 Europe
- AWS Blog
- Amazon EKS 환경에서 Pod Security Standard 구현하기 - 링크
- Validating Amazon EKS optimized Bottlerocket AMI against the CIS Benchmark - 링크
- Managing access to Amazon Elastic Kubernetes Service clusters with X.509 certificates - 링크
- Managing Pod Security on Amazon EKS with Kyverno - 링크
- Implementing Pod Security Standards in Amazon EKS - 링크
- Preventing Kubernetes misconfigurations using Datree - 링크
- Building Amazon Linux 2 CIS Benchmark AMIs for Amazon EKS - 링크
- Secure Bottlerocket deployments on Amazon EKS with KubeArmor - 링크
- Diving into IAM Roles for Service Accounts - 링크
- Leverage AWS secrets stores from EKS Fargate with External Secrets Operator - 링크
- Building STIG-compliant AMIs for Amazon EKS - 링크
- (참고) Valut Operator - Youtube 개요 설치 실습
- (참고) OWASP Kubernetes Top Ten - 링크
- (참고) Amazon Linux 2 Security Advisories - 링크
초보 데브옵스 엔지니어, 피드백은 언제나 환영입니다.