Kamaji Control Plane + CAPO로 capi-quickstart.yaml 생성하기

사전 준비

1. 필수 도구 설치

# clusterctl 설치
curl -L https://github.com/kubernetes-sigs/cluster-api/releases/latest/download/clusterctl-linux-amd64 -o clusterctl
chmod +x ./clusterctl
sudo mv ./clusterctl /usr/local/bin/clusterctl

# kubectl 설치 (이미 설치되어 있다면 건너뛰기)
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
sudo mv kubectl /usr/local/bin/

2. Management Cluster 초기화

OpenStack Resource Controller (ORC) 설치:

# ORC 설치 (CAPO v0.12 이상에서 필요)
kubectl apply -f https://github.com/k-orc/openstack-resource-controller/releases/latest/download/install.yaml

Kamaji와 CAPO 초기화:

# Kamaji Control Plane Provider와 OpenStack Infrastructure Provider 초기화
clusterctl init --infrastructure openstack --control-plane kamaji

환경 변수 설정

OpenStack 인증 정보 설정

# OpenStack 환경 변수 설정
export OPENSTACK_CLOUD=openstack
export OPENSTACK_CLOUD_YAML_B64=$(base64 -w 0 /path/to/clouds.yaml)

# 클러스터 기본 설정
export CLUSTER_NAME=capi-quickstart
export CLUSTER_NAMESPACE=default
export KUBERNETES_VERSION=v1.28.0
export CONTROL_PLANE_MACHINE_COUNT=1
export WORKER_MACHINE_COUNT=2

# OpenStack 리소스 설정
export OPENSTACK_CONTROL_PLANE_MACHINE_FLAVOR=m1.medium
export OPENSTACK_NODE_MACHINE_FLAVOR=m1.medium
export OPENSTACK_IMAGE_NAME=ubuntu-2204-kube-v1.28.0
export OPENSTACK_SSH_KEY_NAME=your-ssh-key
export OPENSTACK_DNS_NAMESERVERS=8.8.8.8
export OPENSTACK_FAILURE_DOMAIN=nova

# 네트워크 설정
export PODS_CIDR=192.168.0.0/16
export SERVICES_CIDR=10.128.0.0/12

템플릿 파일 준비

Kamaji + OpenStack 템플릿 생성:

# kamaji-openstack-template.yaml
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: '${CLUSTER_NAME}'
  namespace: '${CLUSTER_NAMESPACE}'
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
        - '${PODS_CIDR}'
    services:
      cidrBlocks:
        - '${SERVICES_CIDR}'
    serviceDomain: cluster.local
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
    kind: KamajiControlPlane
    name: '${CLUSTER_NAME}-control-plane'
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: OpenStackCluster
    name: '${CLUSTER_NAME}'
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: OpenStackCluster
metadata:
  name: '${CLUSTER_NAME}'
  namespace: '${CLUSTER_NAMESPACE}'
spec:
  # Kamaji가 Control Plane 엔드포인트를 관리하므로 LoadBalancer 비활성화
  apiServerLoadBalancer:
    enabled: false
  disableAPIServerFloatingIP: true
  disableExternalNetwork: false
  cloudName: '${OPENSTACK_CLOUD}'
  identityRef:
    name: '${CLUSTER_NAME}-cloud-config'
    cloudName: '${OPENSTACK_CLOUD}'
  managedSecurityGroups:
    allowAllInClusterTraffic: false
    allNodesSecurityGroupRules:
      # Calico BGP 통신을 위한 규칙
      - remoteManagedGroups:
          - worker
        direction: ingress
        etherType: IPv4
        name: BGP
        portRangeMin: 179
        portRangeMax: 179
        protocol: "tcp"
        description: "Allow BGP among workers"
      # Calico IP-in-IP 통신을 위한 규칙
      - remoteManagedGroups:
          - worker
        direction: ingress
        etherType: IPv4
        name: IP-in-IP
        protocol: "4"
        description: "Allow IP-in-IP among workers"
---
apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
kind: KamajiControlPlane
metadata:
  name: '${CLUSTER_NAME}-control-plane'
  namespace: '${CLUSTER_NAMESPACE}'
spec:
  replicas: ${CONTROL_PLANE_MACHINE_COUNT}
  version: '${KUBERNETES_VERSION}'
  dataStoreName: default
  apiServer:
    extraArgs:
      - --cloud-provider=external
  controllerManager:
    extraArgs:
      - --cloud-provider=external
  addons:
    coreDNS: {}
    kubeProxy: {}
    konnectivity: {}
  kubelet:
    cgroupfs: systemd
    preferredAddressTypes:
      - InternalIP
  network:
    serviceType: LoadBalancer
---
apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: '${CLUSTER_NAME}-md-0'
  namespace: '${CLUSTER_NAMESPACE}'
spec:
  clusterName: '${CLUSTER_NAME}'
  replicas: ${WORKER_MACHINE_COUNT}
  selector:
    matchLabels:
      cluster.x-k8s.io/cluster-name: '${CLUSTER_NAME}'
      cluster.x-k8s.io/deployment-name: '${CLUSTER_NAME}-md-0'
  template:
    metadata:
      labels:
        cluster.x-k8s.io/cluster-name: '${CLUSTER_NAME}'
        cluster.x-k8s.io/deployment-name: '${CLUSTER_NAME}-md-0'
    spec:
      clusterName: '${CLUSTER_NAME}'
      version: '${KUBERNETES_VERSION}'
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          kind: KubeadmConfigTemplate
          name: '${CLUSTER_NAME}-md-0'
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: OpenStackMachineTemplate
        name: '${CLUSTER_NAME}-md-0'
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: OpenStackMachineTemplate
metadata:
  name: '${CLUSTER_NAME}-md-0'
  namespace: '${CLUSTER_NAMESPACE}'
spec:
  template:
    spec:
      cloudName: '${OPENSTACK_CLOUD}'
      flavor: '${OPENSTACK_NODE_MACHINE_FLAVOR}'
      image: '${OPENSTACK_IMAGE_NAME}'
      sshKeyName: '${OPENSTACK_SSH_KEY_NAME}'
      identityRef:
        name: '${CLUSTER_NAME}-cloud-config'
        cloudName: '${OPENSTACK_CLOUD}'
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
  name: '${CLUSTER_NAME}-md-0'
  namespace: '${CLUSTER_NAMESPACE}'
spec:
  template:
    spec:
      joinConfiguration:
        nodeRegistration:
          name: '{{ local_hostname }}'
          kubeletExtraArgs:
            cloud-provider: external
            provider-id: "openstack:///'{{ instance_id }}'"
      preKubeadmCommands:
        - hostname "{{ local_hostname }}"
        - echo "::1         ipv6-localhost ipv6-loopback" >/etc/hosts
        - echo "127.0.0.1   localhost" >>/etc/hosts
        - echo "127.0.0.1   {{ local_hostname }}" >>/etc/hosts
        - echo "{{ local_hostname }}" >/etc/hostname
---
apiVersion: v1
kind: Secret
metadata:
  name: '${CLUSTER_NAME}-cloud-config'
  namespace: '${CLUSTER_NAMESPACE}'
  labels:
    clusterctl.cluster.x-k8s.io/move: "true"
type: Opaque
data:
  clouds.yaml: '${OPENSTACK_CLOUD_YAML_B64}'
  cacert: '${OPENSTACK_CLOUD_CACERT_B64}'

capi-quickstart.yaml 생성

1. 템플릿을 사용하여 매니페스트 생성

# 필요한 변수들 확인
clusterctl generate cluster --from kamaji-openstack-template.yaml --list-variables

# capi-quickstart.yaml 생성
clusterctl generate cluster capi-quickstart \
  --from kamaji-openstack-template.yaml \
  --kubernetes-version v1.28.0 \
  --control-plane-machine-count=1 \
  --worker-machine-count=2 \
  > capi-quickstart.yaml

2. 또는 직접 변수 치환으로 생성

# 환경 변수를 사용하여 템플릿에서 직접 생성
envsubst < kamaji-openstack-template.yaml > capi-quickstart.yaml

클러스터 배포

1. 매니페스트 적용

# 생성된 매니페스트 적용
kubectl apply -f capi-quickstart.yaml

2. 클러스터 상태 확인

# 클러스터 상태 확인
kubectl get cluster capi-quickstart

# 상세 상태 확인
clusterctl describe cluster capi-quickstart

# Kamaji Control Plane 상태 확인
kubectl get kamajicontrolplane

# OpenStack 클러스터 상태 확인
kubectl get openstackcluster

3. 워커 클러스터 kubeconfig 획득

# kubeconfig 다운로드
clusterctl get kubeconfig capi-quickstart > capi-quickstart.kubeconfig

# 클러스터 접속 확인
KUBECONFIG=capi-quickstart.kubeconfig kubectl get nodes

주요 특징

Kamaji Control Plane 장점

• 비용 효율성: Control Plane이 컨테이너로 실행되어 별도 VM 불필요
• 빠른 프로비저닝: 약 16초 내에 Control Plane 준비 완료
• 간편한 업그레이드: 10초 내에 쿠버네티스 버전 업그레이드

OpenStack 통합

• 보안 그룹 자동 관리: CNI(Calico) 통신을 위한 규칙 자동 설정
• Load Balancer 비활성화: Kamaji가 Control Plane 엔드포인트 관리
• External Cloud Provider: OpenStack CCM 사용을 위한 설정

문제 해결

일반적인 이슈

1. ORC 미설치: CAPO v0.12+ 사용 시 반드시 ORC 설치 필요
2. 이미지 준비: cloud-init이 포함된 쿠버네티스 호환 이미지 필요
3. 네트워크 설정: OpenStack 네트워크 구성에 맞는 보안 그룹 규칙 설정

로그 확인

# Kamaji Control Plane Controller 로그
kubectl logs -n kamaji-system deployment/capi-kamaji-controller-manager

# CAPO Controller 로그
kubectl logs -n capo-system deployment/capo-controller-manager