Ingress(with 도메인, 단일 ALB 사용)에 PATH /mario 는 mario 게임 접속하게 설정하고, /tetris 는 tetris 게임에 접속하게 설정하고, SSL 적용 후 관련 스샷 올려주세요
테트리스 또는 테트리스/* 경로에 대해서 해당 대상그룹으로 포워딩
마리오 또는 마리오/* 경로에 대해서 해당 대상그룹으로 포워딩
요청에 응답이 없으면 404에러
호스트 Path(local-path-provisioner) 실습 및 문제점 확인과 성능 측정 후 관련 스샷 올려주세요
PVC가 특정 노드만 실행 중인 파드는 해당 노드에서만 실행이 가능합니다. 따라서 해당 노드의 문제가 발생하면 파드가 실행되지 않아 서비스 장애가 발생합니다.
정상
Every 2.0s: kubectl get pod,pv,pvc -owide Tue Jan 31 23:13:30 2023
NAME READY STATUS RESTARTS AGE IP NODE
NOMINATED NODE READINESS GATES
pod/date-pod-d95d6b8f-rb44q 1/1 Running 0 22s 172.30.58.14 i-00edbcc760f7b92c3 <none> <none>
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE VOLUMEMODE
persistentvolume/pvc-f133582f-15b7-4c05-a0cc-ba97ebc82f43 2Gi RWO Delete Bound default/localpath-claim local-path 18s Filesystem
NAME STATUS VOLUME
CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE
persistentvolumeclaim/localpath-claim Bound pvc-f133582f-15b7-4c05-a0cc-ba97ebc82f43
2Gi RWO local-path 22s Filesystem
drain 설정 이후 장애
Every 2.0s: kubectl get pod,pv,pvc -owide Tue Jan 31 23:14:10 2023
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/date-pod-d95d6b8f-njpl5 0/1 Pending 0 12s <none> <none> <none>
<none>
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE VOLUMEMODE
persistentvolume/pvc-f133582f-15b7-4c05-a0cc-ba97ebc82f43 2Gi RWO Delete Bound default/localpath-claim local-path 58s Filesystem
NAME STATUS VOLUME
CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE
persistentvolumeclaim/localpath-claim Bound pvc-f133582f-15b7-4c05-a0cc-ba97ebc82f43
2Gi RWO local-path 62s Filesystem
에러 로그
(repush:default) [root@kops-ec2 ~]# kubectl describe pod -l app=date | grep Events: -A5
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 38s default-scheduler 0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 1 node(s) had untolerated taint {node.kubernetes.io/unschedulable: }, 1 node(s) were unschedulable, 2 node(s) had volume node affinity conflict. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
kubectl describe pv
...
Node Affinity:
Required Terms:
Term 0: kubernetes.io/hostname in [i-00edbcc760f7b92c3]
...
로컬 호스트패스 스토리지 클래스는 다른 스토리지 솔루션에 비해 IOPS 성능이 뛰어나고 구성이 간편합니다. 하지만 특정 노드의 디스크를 사용하므로 해당 노드에 장애가 발생하면 서비스 장애가 발생합니다. 따라서 애플리케이션 레벨의 볼륨 복제 설정이 필요합니다.
W1PIP=3.39.6.65
W2PIP=13.209.43.177
(repush:default) [root@kops-ec2 ~]# kubestr fio -f fio-read.fio -s local-path --size 10G
PVC created kubestr-fio-pvc-9g6nq
Pod created kubestr-fio-pod-glttn
Running FIO test (fio-read.fio) on StorageClass (local-path) with a PVC of Size (10G)
Elapsed time- 2m32.640633959s
FIO test results:
FIO version - fio-3.30
Global options - ioengine=libaio verify= direct=1 gtod_reduce=
JobName:
blocksize= filesize= iodepth= rw=
read:
IOPS=3021.462891 BW(KiB/s)=12085
iops: min=2476 max=8992 avg=3022.627686
bw(KiB/s): min=9904 max=35968 avg=12090.510742
Disk stats (read/write):
nvme0n1: ios=362293/463 merge=0/153 ticks=6717336/9585 in_queue=6726920, util=99.954987%
AWS EBS를 PVC로 사용 후 온라인 볼륨 증가 후 관련 스샷 올려주세요
(repush:default) [root@kops-ec2 ~]# kubectl get pvc ebs-claim -o jsonpath={.spec.resources.requests.storage} ; echo
4Gi
(repush:default) [root@kops-ec2 ~]# kubectl get pvc ebs-claim -o jsonpath={.status.capacity.storage} ; echo
4Gi
(repush:default) [root@kops-ec2 ~]# kubectl exec -it app -- sh -c 'df -hT --type=ext4'
Filesystem Type Size Used Avail Use% Mounted on
/dev/nvme2n1 ext4 9.8G 28K 9.7G 1% /data
/dev/root ext4 124G 5.2G 119G 5% /etc/hosts
(repush:default) [root@kops-ec2 ~]# kubectl df-pv
PV NAME PVC NAME NAMESPACE NODE NAME POD NAME VOLUME MOUNT NAME SIZE USED AVAILABLE %USED IUSED IFREE %IUSED
pvc-84db7f09-0d5b-466e-9d4d-5f274c5a754c ebs-claim default i-032553c87eb1da454 app persistent-storage 9Gi 28Ki 9Gi 0.00 12 655348 0.00
AWS Volume SnapShots 실습 후 관련 스샷 올려주세요
파일 내용 추가 저장 확인
(repush:default) [root@kops-ec2 ~]# kubectl exec app -- tail -f /data/out.txt
Tue Jan 31 15:06:28 UTC 2023
Tue Jan 31 15:06:33 UTC 2023
Tue Jan 31 15:06:38 UTC 2023
Tue Jan 31 15:06:43 UTC 2023
VolumeSnapshot 확인
(repush:default) [root@kops-ec2 ~]# kubectl get volumesnapshot ebs-volume-snapshot -o jsonpath={.status.boundVolumeSnapshotContentName}
snapcontent-8ce8a40f-db02-4099-bd14-9c9bdddebaa5
AWS EBS 스냅샷 확인
(repush:default) [root@kops-ec2 ~]# aws ec2 describe-snapshots --owner-ids self --query 'Snapshots[]' --output table
----------------------------------------------------------------------------------------------------------------
| DescribeSnapshots |
+-----------------+--------------------------------------------------------------------------------------------+
| Description | Created by CreateImage(i-05ac6756498f91d2d) for ami-0a921992cb6eed04c |
| Encrypted | False |
| KmsKeyId | |
| OwnerId | 442156947938 |
| Progress | 100% |
| SnapshotId | snap-0733ac672a23a851b |
| StartTime | 2022-12-17T13:46:29.477000+00:00 |
| State | completed |
| StorageTier | standard |
| VolumeId | vol-007a6e562984be055 |
| VolumeSize | 8 |
+-----------------+--------------------------------------------------------------------------------------------+
| DescribeSnapshots |
+---------------+----------------------------------------------------------------------------------------------+
| Description | Created by AWS EBS CSI driver for volume vol-007f9ce3bc07b94a2 |
| Encrypted | True |
| KmsKeyId | arn:aws:kms:ap-northeast-2:442156947938:key/9fd996b1-affb-4042-9779-da9c68ec1ab5 |
| OwnerId | 442156947938 |
| Progress | 2% |
| SnapshotId | snap-0997a66f81a5473f2 |
| StartTime | 2023-01-31T15:01:01.865000+00:00 |
| State | pending |
| StorageTier | standard |
| VolumeId | vol-007f9ce3bc07b94a2 |
| VolumeSize | 4 |
+---------------+----------------------------------------------------------------------------------------------+
|| Tags ||
|+-------------------------------------+----------------------------------------------------------------------+|
|| Key | Value ||
|+-------------------------------------+----------------------------------------------------------------------+|
|| ebs.csi.aws.com/cluster | true ||
|| kubernetes.io/cluster/repush.click | owned ||
|| CSIVolumeSnapshotName | snapshot-8ce8a40f-db02-4099-bd14-9c9bdddebaa5 ||
|| Name | repush.click-dynamic-snapshot-8ce8a40f-db02-4099-bd14-9c9bdddebaa5 ||
|| KubernetesCluster | repush.click ||
|+-------------------------------------+----------------------------------------------------------------------+|
강제로 장애 재현
(repush:default) [root@kops-ec2 ~]# kubectl delete pod app && kubectl delete pvc ebs-claim
pod "app" deleted
persistentvolumeclaim "ebs-claim" deleted
(repush:default) [root@kops-ec2 ~]# kubectl exec app -- tail -f /data/out.txt
Tue Jan 31 15:06:28 UTC 2023
Tue Jan 31 15:06:33 UTC 2023
Tue Jan 31 15:06:38 UTC 2023
Tue Jan 31 15:06:43 UTC 2023 <<<
Tue Jan 31 15:09:23 UTC 2023
Tue Jan 31 15:09:28 UTC 2023
Tue Jan 31 15:09:33 UTC 2023
Tue Jan 31 15:09:38 UTC 2023