killercoda
Q1. Apiserver Troubleshooting
The Apiserver is not coming up, the manifest is misconfigured in 3 places. Fix it.
Solution
Log location
/var/log/pods
/var/log/containers
crictl ps + crictl logs
docker ps + docker logs (in case when Docker is used)
kubelet logs: /var/log/syslog or journalctlQ2. Multi-container Deployment
There is a multi-container Deployment in Namespace management which seems to have issues and is not getting ready.
Write the logs of all containers to /root/logs.log .
Can you see the reason for failure?
Solution
⚠️ >>을 통해 해당 파일에 내용을 추가할 수 있다
k -n management logs deploy/collect-data -c nginx >> /root/logs.log
k -n management logs deploy/collect-data -c httpd >> /root/logs.log🚨Q3. Nginx Ingress Controller
The Nginx Ingress Controller has been installed.
Create a new Ingress resource called world for domain name world.universe.mine . The domain points to the K8s Node IP via /etc/hosts .
The Ingress resource should have two routes pointing to the existing Services:
http://world.universe.mine:30080/europe/
and
http://world.universe.mine:30080/asia/
Solution
⚠️ IngressClass를 사용하여 Ingress를 생성해야 한다
해당 예제에서는 nginx를 사용하므로 ingressClassName를 nginx로 정의한다
k get ingressclass⚠️ ingress-nginx-controller가 port 30080에서 nodePort 형태로 서비스되고 있기 때문에 문제의 주소로 접근이 가능하다
controlplane $ k get svc -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.111.75.143 <none> 80:30080/TCP,443:30443/TCP 14m
ingress-nginx-controller-admission ClusterIP 10.108.13.210 <none> 443/TCP 14mapiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-wildcard-host
namespace: world
spec:
rules:
- host: "world.universe.mine"
http:
paths:
- pathType: Prefix
path: "/asia"
backend:
service:
name: asia
port:
number: 80
- pathType: Prefix
path: "/europe"
backend:
service:
name: europe
port:
number: 80Q4. Network Policy
There are existing Pods in Namespace space1 and space2 .
We need a new NetworkPolicy named np that restricts all Pods in Namespace space1 to only have outgoing traffic to Pods in Namespace space2 . Incoming traffic not affected.
We also need a new NetworkPolicy named np that restricts all Pods in Namespace space2 to only have incoming traffic from Pods in Namespace space1 . Outgoing traffic not affected.
The NetworkPolicies should still allow outgoing DNS traffic on port 53 TCP and UDP.
Solution
⚠️ kubernetes.io/metadata.name은 namespace에 관한 label이다
⚠️ Well known label을 통해 레이블이나 annotaion 정보를 불러올 수 있다.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: np
namespace: space1
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: space2
- to:
ports:
- protocol: TCP
port: 53
- protocol: UDP
port: 53
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: np
namespace: space2
spec:
podSelector: {}
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: space1🚨Q5. RBAC ServiceAccount Permissions
There are existing Namespaces ns1 and ns2 .
Create ServiceAccount pipeline in both Namespaces.
These SAs should be allowed to view almost everything in the whole cluster. You can use the default ClusterRole view for this.
These SAs should be allowed to create and delete Deployments in their Namespace.
Verify everything using kubectl auth can-i .
Solution
⚠️ ClusterRole로 생성되었다고 하더라도 namespace specific하면 ClusterRoleBinding이 아닌 RoleBinding으로 Bind 해주어야 한다
⚠️ kubectl api-resources로 apiGroups를 잘 확인하자
apiVersion: v1
kind: ServiceAccount
metadata:
name: pipeline
namespace: ns1
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: pipeline
namespace: ns2
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: deploy-clusterrole
rules:
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["create", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: deploy-clusterrole
namespace: ns1
subjects:
- kind: ServiceAccount
name: pipeline
namespace: ns1
roleRef:
kind: ClusterRole
name: deploy-clusterrole
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: deploy-clusterrole
namespace: ns2
subjects:
- kind: ServiceAccount
name: pipeline
namespace: ns2
roleRef:
kind: ClusterRole
name: deploy-clusterrole
apiGroup: rbac.authorization.k8s.io⚠️ Verify 방법을 잘 알아두어야 한다
# namespace ns1 deployment manager
k auth can-i delete deployments --as system:serviceaccount:ns1:pipeline -n ns1 # YES
k auth can-i create deployments --as system:serviceaccount:ns1:pipeline -n ns1 # YES
k auth can-i update deployments --as system:serviceaccount:ns1:pipeline -n ns1 # NO
k auth can-i update deployments --as system:serviceaccount:ns1:pipeline -n default # NO
# namespace ns2 deployment manager
k auth can-i delete deployments --as system:serviceaccount:ns2:pipeline -n ns2 # YES
k auth can-i create deployments --as system:serviceaccount:ns2:pipeline -n ns2 # YES
k auth can-i update deployments --as system:serviceaccount:ns2:pipeline -n ns2 # NO
k auth can-i update deployments --as system:serviceaccount:ns2:pipeline -n default # NO
# cluster wide view role
k auth can-i list deployments --as system:serviceaccount:ns1:pipeline -n ns1 # YES
k auth can-i list deployments --as system:serviceaccount:ns1:pipeline -A # YES
k auth can-i list pods --as system:serviceaccount:ns1:pipeline -A # YES
k auth can-i list pods --as system:serviceaccount:ns2:pipeline -A # YES
k auth can-i list secrets --as system:serviceaccount:ns2:pipeline -A # NO (default view-role doesn't allow)Q6. RBAC User Permissions
There is existing Namespace applications .
User smoke should be allowed to create and delete Pods, Deployments and StatefulSets in Namespace applications.
User smoke should have view permissions (like the permissions of the default ClusterRole named view ) in all Namespaces but not in kube-system .
Verify everything using kubectl auth can-i .
Solution
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: applications
name: app-role
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create","delete"]
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["create","delete"]
- apiGroups: ["apps"]
resources: ["statefulsets"]
verbs: ["create","delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: app-role
namespace: applications
subjects:
- kind: User
name: smoke
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: app-role
apiGroup: rbac.authorization.k8s.io⚠️ deny-RBAC는 만들 수 없으므로 view Cluster Role을 kube-system을 제외한 모든 namespace에 적용시켜주면 된다
k get ns # get all namespaces
k -n applications create rolebinding smoke-view --clusterrole view --user smoke
k -n default create rolebinding smoke-view --clusterrole view --user smoke
k -n kube-node-lease create rolebinding smoke-view --clusterrole view --user smoke
k -n kube-public create rolebinding smoke-view --clusterrole view --user smokekiller.sh
해당 포스트는 직접 문제를 풀어본 이후 참고할 수 있도록 작성하였다
TroubleShooting
apiVersion is not set if you choose to ignore these errors
해당 yaml 파일에서 apiVersion이 대소문자, 오탈자 등으로 잘못 입력되었을때 발생한다.
kubernetes.io의 Copy 기능이 정상적으로 작동하지 않을 수도 있다
apiVersion: v1Q1.
You have access to multiple clusters from your main terminal through kubectl contexts. Write all those context names into /opt/course/1/contexts.
Next write a command to display the current context into /opt/course/1/context_default_kubectl.sh, the command should use kubectl.
Finally write a second command doing the same thing into /opt/course/1/context_default_no_kubectl.sh, but without the use of kubectl.
Solution
⚠️ .sh 형태로 작성하는 문제에서는 alias k가 아닌 kubectl로 작성해야 한다.
sh /opt/course/1/context_default_kubectl.sh로 실행시켜볼 수 있다.
~/.kube/config에 config 파일이 위치하고 있다
k config view -o jsonpath="{.contexts[*].name}" | tr " " "\n"
kubectl config current-context
cat ~/.kube/config | grep currentQ2.
Use context: kubectl config use-context k8s-c1-H
Create a single Pod of image httpd:2.4.41-alpine in Namespace default. The Pod should be named pod1 and the container should be named pod1-container. This Pod should only be scheduled on controlplane nodes. Do not add new labels to any nodes.
Solution
⚠️nodeName이 아닌 tolerations을 사용해야 한다.
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: pod1
name: pod1
spec:
containers:
- image: httpd:2.4.41-alpine
name: pod1-container # change
resources: {}
dnsPolicy: ClusterFirst
restartPolicy: Always
tolerations: # add
- effect: NoSchedule # add
key: node-role.kubernetes.io/control-plane # add
nodeSelector: # add
node-role.kubernetes.io/control-plane: "" # add
status: {}🚨Q3.
Use context: kubectl config use-context k8s-c1-H
There are two Pods named o3db-* in Namespace project-c13. C13 management asked you to scale the Pods down to one replica to save resources.
Solution
Statefulset : 스테이트풀셋은 애플리케이션의 스테이트풀을 관리하는데 사용하는 워크로드 API 오브젝트이다.
파드 집합의 디플로이먼트와 스케일링을 관리하며, 파드들의 순서 및 고유성을 보장한다 .
⚠️kubectl get sts으로 STS 정보를 가져오고 이를 다운스케일 해야한다.
⚠️또한 해당 Pod가 Daemonset일 경우도 고려해야 한다.
이후 Pod의 Label을 확인하여 Statefulset임을 확실히 한다.
k -n project-c13 get deploy,ds,sts | grep o3db
k -n project-c13 get pod --show-labels | grep o3db
# edit하여 scale을 조정할 수 있다
k -n project-c13 edit sts o3dbQ6.
Use context: kubectl config use-context k8s-c1-H
Create a new PersistentVolume named safari-pv. It should have a capacity of 2Gi, accessMode ReadWriteOnce, hostPath /Volumes/Data and no storageClassName defined.
Next create a new PersistentVolumeClaim in Namespace project-tiger named safari-pvc . It should request 2Gi storage, accessMode ReadWriteOnce and should not define a storageClassName. The PVC should bound to the PV correctly.
Finally create a new Deployment safari in Namespace project-tiger which mounts that volume at /tmp/safari-data. The Pods of that Deployment should be of image httpd:2.4.41-alpine.
Solution
kind: PersistentVolume
apiVersion: v1
metadata:
name: safari-pv
spec:
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/Volumes/Data"kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: safari-pvc
namespace: project-tiger
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi⚠️Deployment yaml 파일의 들여쓰기를 주의하자
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
app: safari
name: safari
namespace: project-tiger
spec:
replicas: 1
selector:
matchLabels:
app: safari
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: safari
spec:
volumes: # add
- name: data # add
persistentVolumeClaim: # add
claimName: safari-pvc # add
containers:
- image: httpd:2.4.41-alpine
name: container
volumeMounts: # add
- name: data # add
mountPath: /tmp/safari-data # addQ7.
Use context: kubectl config use-context k8s-c1-H
The metrics-server has been installed in the cluster. Your college would like to know the kubectl commands to:
show Nodes resource usage
show Pods and their containers resource usage
Please write the commands into /opt/course/7/node.sh and /opt/course/7/pod.sh.
Solution
⚠️ .sh 형태로 작성하는 문제에서는 alias k가 아닌 kubectl로 작성해야 한다.
# -h를 통해 지원되는 옵션을 확인한다
k top pod -h
# /opt/course/7/node.sh
kubectl top node
# /opt/course/7/pod.sh
kubectl top pod --containers=true🚨Q8.
Use context: kubectl config use-context k8s-c1-H
Ssh into the controlplane node with ssh cluster1-controlplane1. Check how the controlplane components kubelet, kube-apiserver, kube-scheduler, kube-controller-manager and etcd are started/installed on the controlplane node. Also find out the name of the DNS application and how it's started/installed on the controlplane node.
Write your findings into file /opt/course/8/controlplane-components.txt. The file should be structured like:
kubelet: [TYPE]
kube-apiserver: [TYPE]
kube-scheduler: [TYPE]
kube-controller-manager: [TYPE]
etcd: [TYPE]
dns: [TYPE] [NAME]Choices of [TYPE] are: not-installed, process, static-pod, pod
Solution
ps aux | grep kubelet # shows kubelet process
➜ root@cluster1-controlplane1:~# find /etc/systemd/system/ | grep kube
/etc/systemd/system/kubelet.service.d
/etc/systemd/system/kubelet.service.d/10-kubeadm.conf
/etc/systemd/system/multi-user.target.wants/kubelet.service
➜ root@cluster1-controlplane1:~# find /etc/systemd/system/ | grep etcd
➜ root@cluster1-controlplane1:~# find /etc/kubernetes/manifests/
/etc/kubernetes/manifests/
/etc/kubernetes/manifests/kube-controller-manager.yaml
/etc/kubernetes/manifests/etcd.yaml
/etc/kubernetes/manifests/kube-apiserver.yaml
/etc/kubernetes/manifests/kube-scheduler.yamlkubelet: process
kube-apiserver: static-pod
kube-scheduler: static-pod
kube-controller-manager: static-pod
etcd: static-pod
dns: pod coredns🚨Q9.
Use context: kubectl config use-context k8s-c2-AC
Ssh into the controlplane node with ssh cluster2-controlplane1. Temporarily stop the kube-scheduler, this means in a way that you can start it again afterwards.
Create a single Pod named manual-schedule of image httpd:2.4-alpine, confirm it's created but not scheduled on any node.
Now you're the scheduler and have all its power, manually schedule that Pod on node cluster2-controlplane1. Make sure it's running.
Start the kube-scheduler again and confirm it's running correctly by creating a second Pod named manual-schedule2 of image httpd:2.4-alpine and check if it's running on cluster2-node1.
Solution
⚠️ Static Pod 형태인 kube-scheduler를 경로에서 제거하여 Temporarily하게 Scheduler를 제거한다.
cd /etc/kubernetes/manifests/
mv kube-scheduler.yaml ..⚠️ nadeName을 추가하여 Manually Scheduling한다
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2020-09-04T15:51:02Z"
labels:
run: manual-schedule
managedFields:
...
manager: kubectl-run
operation: Update
time: "2020-09-04T15:51:02Z"
name: manual-schedule
namespace: default
resourceVersion: "3515"
selfLink: /api/v1/namespaces/default/pods/manual-schedule
uid: 8e9d2532-4779-4e63-b5af-feb82c74a935
spec:
nodeName: cluster2-controlplane1 # add the controlplane node name
containers:
- image: httpd:2.4-alpine
imagePullPolicy: IfNotPresent
name: manual-schedule
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-nxnc7
readOnly: true
dnsPolicy: ClusterFirst🚨Q11.
Use context: kubectl config use-context k8s-c1-H
Use Namespace project-tiger for the following. Create a DaemonSet named ds-important with image httpd:2.4-alpine and labels id=ds-important and uuid=18426a0b-5f59-4e10-923f-c0e078e82462. The Pods it creates should request 10 millicore cpu and 10 mebibyte memory. The Pods of that DaemonSet should run on all nodes, also controlplanes.
Solution
⚠️Deployment를 생성한 이후 설정만 Daemonset으로 바꾸어도 된다
k -n project-tiger create deployment --image=httpd:2.4-alpine ds-important $do > 11.yamlapiVersion: apps/v1
kind: DaemonSet # change from Deployment to Daemonset
metadata:
creationTimestamp: null
labels: # add
id: ds-important # add
uuid: 18426a0b-5f59-4e10-923f-c0e078e82462 # add
name: ds-important
namespace: project-tiger # important
spec:
#replicas: 1 # remove
selector:
matchLabels:
id: ds-important # add
uuid: 18426a0b-5f59-4e10-923f-c0e078e82462 # add
#strategy: {} # remove
template:
metadata:
creationTimestamp: null
labels:
id: ds-important # add
uuid: 18426a0b-5f59-4e10-923f-c0e078e82462 # add
spec:
containers:
- image: httpd:2.4-alpine
name: ds-important
resources:
requests: # add
cpu: 10m # add
memory: 10Mi # add
tolerations: # add
- effect: NoSchedule # add
key: node-role.kubernetes.io/control-plane # add
#status: {} # removeQ12.
Use context: kubectl config use-context k8s-c1-H
Use Namespace project-tiger for the following. Create a Deployment named deploy-important with label id=very-important (the Pods should also have this label) and 3 replicas. It should contain two containers, the first named container1 with image nginx:1.17.6-alpine and the second one named container2 with image google/pause.
There should be only ever one Pod of that Deployment running on one worker node. We have two worker nodes: cluster1-node1 and cluster1-node2. Because the Deployment has three replicas the result should be that on both nodes one Pod is running. The third Pod won't be scheduled, unless a new worker node will be added.
In a way we kind of simulate the behaviour of a DaemonSet here, but using a Deployment and a fixed number of replicas.
Solution
⚠️ podAntiAffinity나 topologySpreadConstraint를 사용할 수 있다.
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
id: very-important # change
name: deploy-important
namespace: project-tiger # important
spec:
replicas: 3 # change
selector:
matchLabels:
id: very-important # change
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
id: very-important # change
spec:
containers:
- image: nginx:1.17.6-alpine
name: container1 # change
resources: {}
- image: google/pause # add
name: container2 # add
affinity: # add
podAntiAffinity: # add
requiredDuringSchedulingIgnoredDuringExecution: # add
- labelSelector: # add
matchExpressions: # add
- key: id # add
operator: In # add
values: # add
- very-important # add
topologyKey: kubernetes.io/hostname # add
status: {}apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
id: very-important # change
name: deploy-important
namespace: project-tiger # important
spec:
replicas: 3 # change
selector:
matchLabels:
id: very-important # change
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
id: very-important # change
spec:
containers:
- image: nginx:1.17.6-alpine
name: container1 # change
resources: {}
- image: google/pause # add
name: container2 # add
topologySpreadConstraints: # add
- maxSkew: 1 # add
topologyKey: kubernetes.io/hostname # add
whenUnsatisfiable: DoNotSchedule # add
labelSelector: # add
matchLabels: # add
id: very-important # add
status: {}⭐⭐Q13. Multi Container, Sidecar with Shared Volume
Use context: kubectl config use-context k8s-c1-H
Create a Pod named multi-container-playground in Namespace default with three containers, named c1, c2 and c3. There should be a volume attached to that Pod and mounted into every container, but the volume shouldn't be persisted or shared with other Pods.
Container c1 should be of image nginx:1.17.6-alpine and have the name of the node where its Pod is running available as environment variable MY_NODE_NAME.
Container c2 should be of image busybox:1.31.1 and write the output of the date command every second in the shared volume into file date.log. You can use while true; do date >> /your/vol/path/date.log; sleep 1; done for this.
Container c3 should be of image busybox:1.31.1 and constantly send the content of file date.log from the shared volume to stdout. You can use tail -f /your/vol/path/date.log for this.
Check the logs of container c3 to confirm correct setup.
Solution
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: multi-container-playground
name: multi-container-playground
spec:
containers:
- image: nginx:1.17.6-alpine
name: c1 # change
resources: {}
env: # add
- name: MY_NODE_NAME # add
valueFrom: # add
fieldRef: # add
fieldPath: spec.nodeName # add
volumeMounts: # add
- name: vol # add
mountPath: /vol # add
- image: busybox:1.31.1 # add
name: c2 # add
command: ["sh", "-c", "while true; do date >> /vol/date.log; sleep 1; done"] # add
volumeMounts: # add
- name: vol # add
mountPath: /vol # add
- image: busybox:1.31.1 # add
name: c3 # add
command: ["sh", "-c", "tail -f /vol/date.log"] # add
volumeMounts: # add
- name: vol # add
mountPath: /vol # add
dnsPolicy: ClusterFirst
restartPolicy: Always
volumes: # add
- name: vol # add
emptyDir: {} # add
status: {}k exec multi-container-playground -c c1 -- env | grep MY
k logs multi-container-playground -c c3Q14.
Use context: kubectl config use-context k8s-c1-H
You're ask to find out following information about the cluster k8s-c1-H:
How many controlplane nodes are available?
How many worker nodes are available?
What is the Service CIDR?
Which Networking (or CNI Plugin) is configured and where is its config file?
Which suffix will static pods have that run on cluster1-node1?
Write your answers into file /opt/course/14/cluster-info, structured like this:
Solution
➜ ssh cluster1-controlplane1
➜ root@cluster1-controlplane1:~# cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep range
- --service-cluster-ip-range=10.96.0.0/12
➜ root@cluster1-controlplane1:~# find /etc/cni/net.d/
/etc/cni/net.d/
/etc/cni/net.d/10-weave.conflist
➜ root@cluster1-controlplane1:~# cat /etc/cni/net.d/10-weave.conflist
{
"cniVersion": "0.3.0",
"name": "weave",
... Q16.
Use context: kubectl config use-context k8s-c1-H
Write the names of all namespaced Kubernetes resources (like Pod, Secret, ConfigMap...) into /opt/course/16/resources.txt.
Find the project-* Namespace with the highest number of Roles defined in it and write its name and amount of Roles into /opt/course/16/crowded-namespace.txt.
Solution
k api-resources --namespaced -o name > /opt/course/16/resources.txt
➜ k -n project-c13 get role --no-headers | wc -l
No resources found in project-c13 namespace.
0
➜ k -n project-c14 get role --no-headers | wc -l
300
➜ k -n project-hamster get role --no-headers | wc -l
No resources found in project-hamster namespace.
0
➜ k -n project-snake get role --no-headers | wc -l
No resources found in project-snake namespace.
0
➜ k -n project-tiger get role --no-headers | wc -l
No resources found in project-tiger namespace.
0⭐⭐Q18.
Use context: kubectl config use-context k8s-c3-CCC
There seems to be an issue with the kubelet not running on cluster3-node1. Fix it and confirm that cluster has node cluster3-node1 available in Ready state afterwards. You should be able to schedule a Pod on cluster3-node1 afterwards.
Write the reason of the issue into /opt/course/18/reason.txt.
Solution
First we check if the kubelet is running
➜ ssh cluster3-node1
➜ root@cluster3-node1:~# ps aux | grep kubelet
root 29294 0.0 0.2 14856 1016 pts/0 S+ 11:30 0:00 grep --color=auto kubeletNope, so we check if it's configured using systemd as service:
➜ root@cluster3-node1:~# service kubelet status
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: inactive (dead) since Sun 2019-12-08 11:30:06 UTC; 50min 52s ago
...Yes, it's configured as a service with config at /etc/systemd/system/kubelet.service.d/10-kubeadm.conf, but we see it's inactive. Let's try to start it:
➜ root@cluster3-node1:~# service kubelet start
➜ root@cluster3-node1:~# service kubelet status
...
Apr 30 22:03:10 cluster3-node1 systemd[5989]: kubelet.service: Failed at step EXEC spawning /usr/local/bin/kubelet: No such file or directoryWe see it's trying to execute /usr/local/bin/kubelet with some parameters defined in its service config file. A good way to find errors and get more logs is to run the command manually (usually also with its parameters).
➜ root@cluster3-node1:~# /usr/local/bin/kubelet
-bash: /usr/local/bin/kubelet: No such file or directory
➜ root@cluster3-node1:~# whereis kubelet
kubelet: /usr/bin/kubeletAnother way would be to see the extended logging of a service like using journalctl -u kubelet.
⚠️ Well, there we have it, wrong path specified. Correct the path in file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf and run:
vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf # fix
systemctl daemon-reload && systemctl restart kubelet
systemctl status kubelet # should now show running⭐Q20.
Use context: kubectl config use-context k8s-c3-CCC
Your coworker said node cluster3-node2 is running an older Kubernetes version and is not even part of the cluster. Update Kubernetes on that node to the exact version that's running on cluster3-controlplane1. Then add this node to the cluster. Use kubeadm for this.
Solution
kubeadm version
kubectl version --short
kubelet --version⚠️ controlplane으로 돌아와서 join 스크립트를 가져온다
kubeadm token create --print-join-commandQ24.
Use context: kubectl config use-context k8s-c1-H
There was a security incident where an intruder was able to access the whole cluster from a single hacked backend Pod.
To prevent this create a NetworkPolicy called np-backend in Namespace project-snake. It should allow the backend-* Pods only to:
connect to db1-* Pods on port 1111
connect to db2-* Pods on port 2222
Use the app label of Pods in your policy.
After implementation, connections from backend- Pods to vault- Pods on port 3333 should for example no longer work.
Solution
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: np-backend
namespace: project-snake
spec:
podSelector:
matchLabels:
app: backend
policyTypes:
- Egress # policy is only about Egress
egress:
- # first rule
to: # first condition "to"
- podSelector:
matchLabels:
app: db1
ports: # second condition "port"
- protocol: TCP
port: 1111
- # second rule
to: # first condition "to"
- podSelector:
matchLabels:
app: db2
ports: # second condition "port"
- protocol: TCP
port: 2222➜ k -n project-snake exec backend-0 -- curl -s 10.44.0.25:1111
database one
➜ k -n project-snake exec backend-0 -- curl -s 10.44.0.23:2222
database two
➜ k -n project-snake exec backend-0 -- curl -s 10.44.0.22:3333
...⭐Q25.
Use context: kubectl config use-context k8s-c3-CCC
Make a backup of etcd running on cluster3-controlplane1 and save it on the controlplane node at /tmp/etcd-backup.db.
Then create any kind of Pod in the cluster.
Finally restore the backup, confirm the cluster is still working and that the created Pod is no longer with us.
Solution
➜ root@cluster3-controlplane1:~# vim /etc/kubernetes/manifests/etcd.yaml⚠️ Dont use snapshot status because it can alter the snapshot file and render it invalid
Next we stop all controlplane components:
root@cluster3-controlplane1:~# cd /etc/kubernetes/manifests/
root@cluster3-controlplane1:/etc/kubernetes/manifests# mv * ..
root@cluster3-controlplane1:/etc/kubernetes/manifests# watch crictl ps⚠️ /var/lib/etcd-from-backup의 데이터를 참조하도록 변경해준다
➜ root@cluster3-controlplane1:~# vim /etc/kubernetes/etcd.yaml
...
name: etcd-certs
- hostPath:
path: /var/lib/etcd-from-backup # change
type: DirectoryOrCreate
name: etcd-data