😡 VM [노드, 클러스터 구성하기, Cillium]

김성인·2023년 10월 10일

[DevOps] 🐳Docker & Kubernetes

목록 보기

14/62

노드 준비

https://kubernetes.io/ko/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

스왑 비활성화

# 현재 시스템에 적용(리부팅하면 재 설정 필요)
sudo swapoff -a
# 리부팅 필수
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

컨테이너 런타임 구성

https://www.itzgeek.com/how-tos/linux/ubuntu-how-tos/install-containerd-on-ubuntu-22-04.html#Using_Docker_Repository

# using Docker Repository
sudo apt update
sudo apt install -y ca-certificates curl gnupg lsb-release
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list

# install contaierd
sudo apt update
sudo apt install -y containerd.io
sudo systemctl status containerd

# Containerd configuration for Kubernetes
# 컨테이너d 런타임 등록
# SystemdCgroup: 도커네임스페이스를 쿠버네티스와 공유하기 위한 설정
cat <<EOF | sudo tee -a /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
EOF

sudo sed -i 's/^disabled_plugins \=/\#disabled_plugins \=/g' /etc/containerd/config.toml

sudo systemctl restart containerd

# 소켓이 있는지 확인
ls /var/run/containerd/containerd.sock

kubeadm 설치하기

https://kubernetes.io/ko/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#kubeadm-kubelet-%EB%B0%8F-kubectl-%EC%84%A4%EC%B9%98

apt 패키지 색인을 업데이트하고, 쿠버네티스 apt 리포지터리를 사용하는 데 필요한 패키지를 설치한다.
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl

구글 클라우드의 공개 사이닝 키를 다운로드 한다.
sudo curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

쿠버네티스 apt 리포지터리를 추가한다.

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

apt 패키지 색인을 업데이트하고, kubelet, kubeadm, kubectl을 설치하고 해당 버전을 고정한다.
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

고: Debian 12 및 Ubuntu 22.04 이전 릴리스에서는 /etc/apt/keyrings이 기본적으로 존재하지 않는다. 필요한 경우 이 디렉토리를 생성하여, 누구나 읽을 수 있지만 관리자만 쓸 수 있도록 만들 수 있다.

cat <<EOF > kube_install.sh
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl

sudo curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
EOF

sudo bash kube_install.sh

kubeadm version

참고 : https://www.inflearn.com/questions/888659/%EC%BF%A0%EB%B2%84%EB%84%A4%ED%8B%B0%EC%8A%A4-%EC%84%A4%EC%B9%98%ED%95%A0-%EB%95%8C-%EC%97%90%EB%9F%AC%EB%82%A9%EB%8B%88%EB%8B%A4-%ED%95%B4%EA%B2%B0-%EB%B0%A9%EB%B2%95-%EA%B3%B5%EC%9C%A0%ED%95%A9%EB%8B%88%EB%8B%A4-23-05-30-%EA%B8%B0%EC%A4%80

kubeadm 설치 오류

Hit:1 https://download.docker.com/linux/ubuntu focal InRelease
Hit:2 http://asia-northeast3.gce.archive.ubuntu.com/ubuntu focal InRelease 
Hit:3 http://asia-northeast3.gce.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:4 http://asia-northeast3.gce.archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:5 http://security.ubuntu.com/ubuntu focal-security InRelease           
Reading package lists... Done
Reading package lists... Done
Building dependency tree       
Reading state information... Done
ca-certificates is already the newest version (20230311ubuntu0.20.04.1).
curl is already the newest version (7.68.0-1ubuntu2.19).
The following packages were automatically installed and are no longer required:
  libatasmart4 libblockdev-fs2 libblockdev-loop2 libblockdev-part-err2 libblockdev-part2 libblockdev-swap2
  libblockdev-utils2 libblockdev2 libmbim-glib4 libmbim-proxy libmm-glib0 libnspr4 libnss3 libnuma1
  libparted-fs-resize0 libqmi-glib5 libqmi-proxy libudisks2-0 libxmlb2 usb-modeswitch usb-modeswitch-data
Use 'sudo apt autoremove' to remove them.
The following NEW packages will be installed:
  apt-transport-https
0 upgraded, 1 newly installed, 0 to remove and 21 not upgraded.
Need to get 1704 B of archives.
After this operation, 162 kB of additional disk space will be used.
Get:1 http://asia-northeast3.gce.archive.ubuntu.com/ubuntu focal-updates/universe amd64 apt-transport-https all 2.0.9 [1704 B]
Fetched 1704 B in 0s (21.0 kB/s)
Selecting previously unselected package apt-transport-https.
(Reading database ... 62245 files and directories currently installed.)
Preparing to unpack .../apt-transport-https_2.0.9_all.deb ...
Unpacking apt-transport-https (2.0.9) ...
Setting up apt-transport-https (2.0.9) ...
deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main
Hit:1 https://download.docker.com/linux/ubuntu focal InRelease
Hit:2 http://asia-northeast3.gce.archive.ubuntu.com/ubuntu focal InRelease                                    
Hit:3 http://asia-northeast3.gce.archive.ubuntu.com/ubuntu focal-updates InRelease                            
Hit:4 http://asia-northeast3.gce.archive.ubuntu.com/ubuntu focal-backports InRelease                          
Hit:6 http://security.ubuntu.com/ubuntu focal-security InRelease                                   
Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8993 B]  
Ign:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages
Ign:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages
Ign:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages
Err:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages
  404  Not Found [IP: 142.250.76.142 443]
Fetched 8993 B in 1s (6544 B/s)
Reading package lists... Done
E: Failed to fetch https://packages.cloud.google.com/apt/dists/kubernetes-xenial/main/binary-amd64/by-hash/SHA256/0d7ba2a5a480b51e90c78214afbd41cb4211c23cc52c2497ebcfc3f1bb8e7e31  404  Not Found [IP: 142.250.76.142 443]
E: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package kubelet
E: Unable to locate package kubeadm
E: Unable to locate package kubectl
E: Unable to locate package kubelet
E: Unable to locate package kubeadm
E: Unable to locate package kubectl
E: No packages found

🤗해결

echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

sudo apt-get update
sudo apt-get install kubelet kubeadm kubectl

sudo apt-get update

오류 한번 더 남

sudo apt-get update
Hit:1 https://download.docker.com/linux/ubuntu focal InRelease
Hit:2 http://asia-northeast3.gce.archive.ubuntu.com/ubuntu focal InRelease
Hit:3 http://asia-northeast3.gce.archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:4 http://asia-northeast3.gce.archive.ubuntu.com/ubuntu focal-backports InRelease
Hit:6 http://security.ubuntu.com/ubuntu focal-security InRelease
Get:5 https://packages.cloud.google.com/apt kubernetes-xenial InRelease [8993 B]
Ign:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages
Ign:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages
Ign:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages
Err:7 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 Packages
  404  Not Found [IP: 142.250.76.142 443]
Fetched 8993 B in 2s (5979 B/s)
Reading package lists... Done
E: Failed to fetch https://packages.cloud.google.com/apt/dists/kubernetes-xenial/main/binary-amd64/by-hash/SHA256/0d7ba2a5a480b51e90c78214afbd41cb4211c23cc52c2497ebcfc3f1bb8e7e31  404  Not Found [IP: 142.250.76.142 443]
E: Some index files failed to download. They have been ignored, or old ones used instead.

kubernets.list 파일 확인해보니 URL은 정상적으로 작성되어있다.. 그래서 다시 update를 하니까 이번엔 성공함..
그후 sudo apt-get install kubelet kubeadm kubectl 하니 성공

넷필터 브릿지 설정

네트워크 통신을 위한 설정

sudo -i

sudo modprobe br_netfilter
echo 1 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables

exit

클러스터 구성하기

마스터 노드 초기화

sudo kubeadm init: 옵션을 줘서 컨트롤 플레인을 여러개 만들수 있음. (마스터를 두대 이상)

유저 설정

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

파드 네트워크 설정

https://kubernetes.io/docs/concepts/cluster-administration/addons/

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

워커 노드 조인

워커 노드 조인방법

sudo kubeadm join 10.178.0.2:6443 --token 5mjpzx.trk8jvqeuyecyt0g \
        --discovery-token-ca-cert-hash sha256:efa7e790d83179e19dbc5635dc21573d2095c5e1fe61080b0cd32f67d86ded5d

Cilium 설치

https://kubernetes.io/docs/tasks/administer-cluster/network-policy-provider/cilium-network-policy/
컨테이너에 파드간 통신을 할때 쓰는 도구 세팅

curl -LO https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz
sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
rm cilium-linux-amd64.tar.gz
cilium install