[k8s] Helm Chart 기반 인프라 구성요소 설치

개요

• EKS 에서 운영에 필요한 인프라 구성요소를 Helm chart 로 설치하는 방법을 정리

  • 자체 앱은 Kustomize, 서드파티 인프라는 Helm 으로 역할 분리
  • 모든 chart 의 values.yaml 을 chart/ 디렉토리에서 Git 관리

• 설치 대상

구성요소	chart	namespace	용도
PostgreSQL	bitnami/postgresql	db	RDB
Redis	bitnami/redis	db	캐시/세션
OpenSearch	opensearch/opensearch	db	검색/분석
Prometheus + Grafana	kube-prometheus-stack	monitoring	모니터링
PushGateway	prometheus-pushgateway	monitoring	배치 메트릭 수집
Fluent-bit	fluent/fluent-bit	kube-system	로그 수집

---

공통 패턴: nodeSelector + tolerations

• 모든 인프라 구성요소에 Karpenter NodePool 에 맞는 nodeSelector/tolerations 적용
  • Karpenter 가 관리하는 노드에만 배포되도록 제한
  • values.yaml 에서 지정

nodeSelector:
  nodegroup: "<node_pool>"
tolerations:
  - key: "nodegroup"
    operator: "Equal"
    value: "<node_pool>"
    effect: "NoSchedule"

---

PostgreSQL

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install pg-dev bitnami/postgresql -n db -f values.yaml

• values.yaml 주요 설정

image:
  repository: bitnami/postgresql
  tag: latest

auth:
  username: admin
  password: "<password>"
  database: <db_name>

primary:
  nodeSelector:
    nodegroup: "<node_pool>"
  tolerations:
    - key: "nodegroup"
      operator: "Equal"
      value: "<node_pool>"
      effect: "NoSchedule"
  persistence:
    enabled: true
    storageClass: gp3
    size: 20Gi
  resources:
    requests:
      cpu: 200m
      memory: 512Mi
    limits:
      cpu: 500m
      memory: 1Gi

readReplicas:
  enabled: false

---

Redis

• standalone (싱글 노드) 과 replication (master/replica + sentinel failover) 중 선택

# standalone
helm install redis bitnami/redis -n db -f values.yaml

# cluster
helm install redis bitnami/redis -n db -f cluster_values.yaml

• standalone values.yaml 주요 설정

architecture: standalone

auth:
  enabled: false

master:
  nodeSelector:
    nodegroup: "<node_pool>"
  tolerations:
    - key: "nodegroup"
      operator: "Equal"
      value: "<node_pool>"
      effect: "NoSchedule"
  persistence:
    enabled: true
    storageClass: gp3
    size: 10Gi
  resources:
    requests:
      cpu: "2"
      memory: "2Gi"
    limits:
      cpu: "4"
      memory: "16Gi"

# Redis config override
redis:
  configmap: |-
    save 900 1
    save 300 10
    save 60 10000
    rdbcompression yes
    rdbchecksum yes
    maxmemory 16GB
    maxmemory-policy volatile-ttl
    appendonly no

---

OpenSearch

helm install opensearch-dev opensearch/opensearch -n db -f values.yaml

• values.yaml 주요 설정

clusterName: opensearch-cluster
replicas: 3

image:
  repository: opensearchproject/opensearch
  tag: "3.1.0"

# 인증 정보는 Secret 으로 분리
envFrom:
  - secretRef:
      name: opensearch-admin-password

persistence:
  enabled: true
  storageClass: gp3
  size: 50Gi

resources:
  requests:
    cpu: 500m
    memory: 2Gi
  limits:
    cpu: "1"
    memory: 4Gi

nodeSelector:
  nodegroup: "<node_pool>"
tolerations:
  - key: "nodegroup"
    operator: "Equal"
    value: "<node_pool>"
    effect: "NoSchedule"

---

Prometheus + Grafana (kube-prometheus-stack)

• kube-prometheus-stack 으로 Prometheus, Grafana, node-exporter, kube-state-metrics 를 한번에 설치

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

kubectl create namespace monitoring

helm upgrade --install kps prometheus-community/kube-prometheus-stack \
  -f values.yaml -n monitoring

• values.yaml 주요 설정

fullnameOverride: kps

prometheus:
  prometheusSpec:
    retention: 15d
    storageSpec:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          storageClassName: gp3
          resources:
            requests:
              storage: 50Gi
    scrapeInterval: 30s
    evaluationInterval: 30s
    # 모든 namespace 의 ServiceMonitor 를 수집
    serviceMonitorSelectorNilUsesHelmValues: false
    podMonitorSelectorNilUsesHelmValues: false

alertmanager:
  enabled: false

# 각 노드 OS 지표
prometheus-node-exporter:
  enabled: true

# K8s 오브젝트 상태 지표
kube-state-metrics:
  enabled: true
  nodeSelector:
    nodegroup: "<node_pool>"
  tolerations:
    - key: "nodegroup"
      operator: "Equal"
      value: "<node_pool>"
      effect: "NoSchedule"

• serviceMonitorSelectorNilUsesHelmValues: false
  • 이 값이 true (기본값) 이면 같은 Helm release 의 ServiceMonitor 만 수집
  • false 로 설정해야 모든 namespace 에서 생성한 ServiceMonitor 를 Prometheus 가 수집

Grafana 접근

• 실제 배포에서는 service.type: LoadBalancer 로 설정하여 ALB 로 직접 접근
• 로컬 테스트 시 포트포워딩도 가능

# LoadBalancer 주소 확인
kubectl get svc kps-grafana -n monitoring -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

# 또는 포트포워딩
kubectl port-forward --namespace monitoring svc/kps-grafana 3000:80

PushGateway 설치

• 배치 작업 등 pull 방식이 어려운 경우 push 로 메트릭을 전송하기 위해 사용

helm install pushgateway prometheus-community/prometheus-pushgateway \
  -n monitoring \
  --set serviceMonitor.enabled=true \
  --set serviceMonitor.namespace=monitoring \
  --set serviceMonitor.additionalLabels.release=prometheus-stack

• additionalLabels.release=prometheus-stack : Prometheus 가 이 ServiceMonitor 를 수집하도록 label 매칭

---

Fluent-bit (로그 수집)

• 컨테이너 로그를 CloudWatch Logs 로 전송
  • IRSA (IAM Roles for Service Accounts) 로 AWS 인증

IRSA 설정

1. EKS OIDC Provider 확인

aws eks describe-cluster \
  --name <cluster_name> \
  --query "cluster.identity.oidc.issuer" \
  --output text

2. IAM Role 생성 (trust-policy.json)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<account_id>:oidc-provider/<oidc_provider>"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "<oidc_provider>:sub": "system:serviceaccount:kube-system:fluent-bit"
        }
      }
    }
  ]
}

aws iam create-role \
  --role-name fluent-bit-iam-role \
  --assume-role-policy-document file://trust-policy.json

aws iam attach-role-policy \
  --role-name fluent-bit-iam-role \
  --policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy

3. ServiceAccount 생성

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: kube-system
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<account_id>:role/fluent-bit-iam-role

Helm 설치

helm repo add fluent https://fluent.github.io/helm-charts
helm upgrade --install fluent-bit fluent/fluent-bit \
  -n kube-system -f fluent-bit-values.yaml

• values.yaml 주요 설정

config:
  service: |
    [SERVICE]
        Flush 1
        Log_Level info
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020

  inputs: |
    [INPUT]
        Name tail
        Path /var/log/containers/*<app_name>*.log
        multiline.parser docker, cri
        Tag app.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

  filters: |
    [FILTER]
        Name kubernetes
        Match app.*
        Merge_Log On
        Keep_Log Off

  outputs: |
    [OUTPUT]
        Name cloudwatch_logs
        Match app.*
        region ap-northeast-1
        log_group_name /eks/app/<app_name>
        log_stream_prefix <app_name>-
        auto_create_group true

serviceAccount:
  create: false
  name: fluent-bit

tolerations:
  - key: "nodegroup"
    operator: "Equal"
    value: "infra"
    effect: "NoSchedule"

---

삭제 및 정리

# Helm release 삭제
helm uninstall <release_name> -n <namespace>

# PVC 도 정리할 경우 (데이터 삭제됨)
kubectl delete pvc -n <namespace> --all

---

reference

• Bitnami Helm Charts
• kube-prometheus-stack
• Fluent-bit Helm Chart
• AWS IRSA 공식 문서