AWS EKS

참고링크: https://docs.aws.amazon.com/ko_kr/eks/latest/userguide/getting-started.html

choco install awscli aws-iam-authenticator eksctl kubernetes-helm

AWS에서 사용자 생성(.csv 파일 받기)

aws configure

eksctl create cluster --name myeks --nodes=3 --region=ap-northeast-2

안되는 것들
Load Balancer Service = class lb -> nlb
Ingress: X
kubectl top: X -> HPA X

클러스터 네트워킹 참고링크: https://kubernetes.io/ko/docs/concepts/cluster-administration/networking/

---

YAML 파일을 이용한 EKS 배포

mkdir aws-eks
cd aws-eks

myeks.yaml

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: myeks-custom
  region: ap-northeast-2
  version: "1.22"

# AZ
availabilityZones: ["ap-northeast-2a", "ap-northeast-2b",  "ap-northeast-2c"]

# IAM OIDC & Service Account
iam:
  withOIDC: true
  serviceAccounts:
    - metadata:
        name: aws-load-balancer-controller
        namespace: kube-system
      wellKnownPolicies:
        awsLoadBalancerController: true
    - metadata:
        name: ebs-csi-controller-sa
        namespace: kube-system
      wellKnownPolicies:
        ebsCSIController: true
    - metadata:
        name: cluster-autoscaler
        namespace: kube-system
      wellKnownPolicies:
        autoScaler: true

# Managed Node Groups
managedNodeGroups:
  # On-Demand Instance
  - name: myeks-ng1
    instanceType: t3.medium
    minSize: 2
    desiredCapacity: 3
    maxSize: 4
    privateNetworking: true
    ssh:
      allow: true
      publicKeyPath: ./keypair/myeks.pub
    availabilityZones: ["ap-northeast-2a", "ap-northeast-2b", "ap-northeast-2c"]
    iam:
      withAddonPolicies:
        autoScaler: true
        albIngress: true
        cloudWatch: true
        ebs: true

# Fargate Profiles
fargateProfiles:
  - name: fg-1
    selectors:
    - namespace: dev
      labels:
        env: fargate
        
        
# CloudWatch Logging
cloudWatch:
  clusterLogging:
    enableTypes: ["*"]
    

mkdir keypair
ssh-keygen -f keypair/myssh

eksctl create cluster -f myeks.yaml

Classic LoadBalancer는 ec2에만 작동

NLB for LoadBalancer Service

https://docs.aws.amazon.com/ko_kr/eks/latest/userguide/network-load-balancing.html
https://docs.aws.amazon.com/ko_kr/eks/latest/userguide/alb-ingress.html
https://docs.aws.amazon.com/ko_kr/eks/latest/userguide/aws-load-balancer-controller.html

AWS Load Balancer Controller 설치

helm repo add eks https://aws.github.io/eks-charts
helm repo update

helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=myeks-custom --set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller --set image.repository=602401143452.dkr.ecr.ap-northeast-2.amazonaws.com/amazon/aws-load-balancer-controller

샘플 코드

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myweb-deploy
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: myweb
          image: ghcr.io/c1t1d0s7/go-myweb
          ports:
            - containerPort: 8080

apiVersion: v1
kind: Service
metadata:
  name: myweb-svc-lb
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "external"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "instance"
	service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
spec:
  type: LoadBalancer
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 8080

• service.beta.kubernetes.io/aws-load-balancer-nlb-target-type
  - instance: EC2 타겟
  - ip: Pod 타겟(Fargate)
• service.beta.kubernetes.io/aws-load-balancer-scheme
  - internal: 내부
  - internet-facing: 외부

internet-facing 설정을 안해주면 lb는 private에 설치됨

Ingress for ALB

https://docs.aws.amazon.com/ko_kr/eks/latest/userguide/alb-ingress.html

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myweb-ing
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/target-type: instance
    alb.ingress.kubernetes.io/scheme: internet-facing
spec:
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: myweb-svc-lb
                port:
                  number: 80

• alb.ingress.kubernetes.io/target-type
  - instance: EC2 타겟
  - ip: Pod 타겟(Fargate)
• alb.ingress.kubernetes.io/scheme
  - internal: 내부
  - internet-facing: 외부
  

internet-facing 설정을 안해주면 lb는 private에 설치됨

EBS for CSI

• EBS 스냅샷
• EBS 크기 변경

https://docs.aws.amazon.com/ko_kr/eks/latest/userguide/managing-ebs-csi.html

eksctl get iamserviceaccount --cluster myeks-custom      #arn 확인

NAMESPACE       NAME                            ROLE ARN
kube-system     aws-load-balancer-controller    arn:aws:iam::065144736597:role/eksctl-myeks-custom-addon-iamserviceaccount-Role1-11N0OKMVG2DYY
kube-system     aws-node                        arn:aws:iam::065144736597:role/eksctl-myeks-custom-addon-iamserviceaccount-Role1-CLMK7A6K5NL3
kube-system     cluster-autoscaler              arn:aws:iam::065144736597:role/eksctl-myeks-custom-addon-iamserviceaccount-Role1-1S02W28MZOSL4
kube-system     ebs-csi-controller-sa           arn:aws:iam::065144736597:role/eksctl-myeks-custom-addon-iamserviceaccount-Role1-15HLE8HBOD9CN

eksctl create addon --name aws-ebs-csi-driver --cluster myeks-custom --service-account-role-arn  arn:aws:iam::065144736597:role/eksctl-myeks-custom-addon-iamserviceaccount-Role1-15HLE8HBOD9CN --force
(나에게 맞는 arn으로 변경)

kubectl get po -n kube-system

kubectl get sc

Metrics Server

https://docs.aws.amazon.com/ko_kr/eks/latest/userguide/metrics-server.html

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

kubectl get po -n kube-system

kubectl top nodes

Cluster Autoscaler

수동 스케일링

eksctl scale nodegroup --name myeks-ng1 --cluster myeks-custom --nodes 2

자동 스케일링

https://docs.aws.amazon.com/ko_kr/eks/latest/userguide/autoscaling.html

curl -o cluster-autoscaler-autodiscover.yaml https://raw.githubusercontent.com/kubernetes/autoscaler/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-autodiscover.yaml

cluster-autoscaler-autodiscover.yaml

163: - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/myeks-custom
(163번째 열에 클러스터 이름변경)

kubectl apply -f cluster-autoscaler-autodiscover.yaml

(적용이 된다면 사용)
kubectl patch deployment cluster-autoscaler -n kube-system -p '{"spec":{"template":{"metadata":{"annotations":{"cluster-autoscaler.kubernetes.io/safe-to-evict": "false"}}}}}'

kubectl -n kube-system edit deployment.apps/cluster-autoscaler

      - command:
        - ./cluster-autoscaler
        - --v=4
        - --stderrthreshold=info
        - --cloud-provider=aws
        - --skip-nodes-with-local-storage=false
        - --expander=least-waste
        - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/myeks-custom
        - --balance-similar-node-groups
        - --skip-nodes-with-system-pods=false
        image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.22.6

수정

• --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/myeks-custom
• --balance-similar-node-groups
• --skip-nodes-with-system-pods=false
• image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.22.2

kubectl set image deployment cluster-autoscaler -n kube-system cluster-autoscaler=k8s.gcr.io/autoscaling/cluster-autoscaler:v1.22.2

cluster autoscaler 로그 보기

kubectl -n kube-system logs -f deployment.apps/cluster-autoscaler

샘플 코드

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myweb-deploy
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: myweb
          image: ghcr.io/c1t1d0s7/go-myweb:alpine
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 200m
              memory: 200M
            limits:
              cpu: 200m
              memory: 200M

CloudWatch Container Insight

https://docs.aws.amazon.com/ko_kr/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-EKS-quickstart.html

https://github.com/git-for-windows/git/releases/download/v2.36.1.windows.1/Git-2.36.1-64-bit.exe
(window bash가 없어서 64비트로 받아준다)

ClusterName=myeks-custom
RegionName=ap-northeast-2
FluentBitHttpPort='2020'
FluentBitReadFromHead='Off'
[[ ${FluentBitReadFromHead} = 'On' ]] && FluentBitReadFromTail='Off'|| FluentBitReadFromTail='On'
[[ -z ${FluentBitHttpPort} ]] && FluentBitHttpServer='Off' || FluentBitHttpServer='On'
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluent-bit-quickstart.yaml | sed 's/{{cluster_name}}/'${ClusterName}'/;s/{{region_name}}/'${RegionName}'/;s/{{http_server_toggle}}/"'${FluentBitHttpServer}'"/;s/{{http_server_port}}/"'${FluentBitHttpPort}'"/;s/{{read_from_head}}/"'${FluentBitReadFromHead}'"/;s/{{read_from_tail}}/"'${FluentBitReadFromTail}'"/' | kubectl apply -f - 

Fargate

EC2 인스턴스 사용 , 파드를 실행

https://docs.aws.amazon.com/ko_kr/eks/latest/userguide/fargate.html

kubectl create ns dev

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myfg
  namespace: dev
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myfg
  template:
    metadata:
      labels:
        app: myfg
        env: fargate
    spec:
      containers:
      - name: myfg
        image: ghcr.io/c1t1d0s7/go-myweb
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - containerPort: 8080

apiVersion: v1
kind: Service
metadata:
  name: mysvc
  namespace: dev
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "external"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
spec:
  selector:
    app: myfg
  ports:
  - port: 80
    targetPort: 8080
  type: LoadBalancer

VPA

https://docs.aws.amazon.com/ko_kr/eks/latest/userguide/vertical-pod-autoscaler.html

사전 요구 사항

• openssl 1.1.1 이상
• metrics-server

git clone https://github.com/kubernetes/autoscaler.git

cd autoscaler/vertical-pod-autoscaler/

/hack/vpa-up.sh

kubectl get pods -n kube-system

VPA 예제

kubectl apply -f examples/hamster.yaml

클러스터 삭제

eksctl delete cluster -f .\myeks.yaml --force --disable-nodegroup-eviction

cloud watch 로그 리소스 삭제 해주기

---

Tip

lens

gui 방식으로 인터페이스 서비스 가능
참고링크: https://k8slens.dev/

choco install lens
(패키지 설치 시 에는 관리자 권한으로 접속)

k9s

tui 방식으로 인터페이스 서비스(text 형식)
참고링크: https://k9scli.io/

choco install k9s

k9s

visual studio code

extension(확장)
kubernetes 설치
(docker도 설치가능)

choco install kubernetes-helm

ctl + shift + p
명령줄에 바로 kubernetes create 가능(터미널 창에 치지 않아도 됨)

minikube

https://minikube.sigs.k8s.io/docs/start/

choco install minikube
(관리자 권한으로 설치)

choco install kubernetes-cli --version=1.22.4

클러스터 생성/실행

minikube start
(기본값으로 설치됨)

클러스터 중지

minikube stop

클러스터 상태

minikube status

VM 접속

minikube ssh

패키지 관리자 X
kubectl 명령 X
docker 명령 O

VM 내의 Docker Engine 사용

choco install docker-cli
(docker command 만 설치, 서버x)

minikube -p minikube docker-env --shell powershell | Invoke-Expression
(변수는 해당 터미널에서만 유효)

docker ps

클러스터 삭제

minikube delete

추가 옵션을 사용한 클러스터 생성/시작

minikube start --cpus 4 --memory 4G --disk-size 30G --driver virtualbox --kubernetes-version v1.22.9

노드 추가

minikube node list
minikube node add     #자동으로 join해줌

서비스 목록 확인

minikube service list

애드온

minikube addons list

minikube addons enable metrics-server
minikube addons enable ingress

minikube addons configure metallb

-- Enter Load Balancer Start IP: 192.168.X.200
-- Enter Load Balancer End IP: 192.168.X.209

클러스터 기본 옵션 지정

minikube config set cpus 2
minikube config set memory 4G
minikube config set driver virtualbox
minikube config set kubernetes-version v1.22.9
minikube config view