1.

CORE-CONCEPT KNOWLEDGE

No Error - ONLY connected by kube-apiserver Debug step

1 etcd server -- validate
Has 2 ports type:
request
2 kube-apiserver - 127.0.0.1:2379:
retrived / update for kube-apiserver
data communicate (1)
=> ETCD servers
pod qua
(only node #)interact with ETCD)
componet
3 kube-controller-manager
- Step2: calculate & find the most suitable node
4 kube-scheduler
- Step3: send signal to KUBELET for pod creation
5 kubelet Contact points between worker nodes & control plan
6 kube-proxy - pods communication channel in same nodes / across nodes (NAT)

ReplicationController
7 ReplicaSet
Deployment

- can access to service clusterIP by ssh to worker nodes & curl cluster IP
8 Service Ex: kubectl get service | grep uac => clusterIP: 10.97.25.252:80
9 curl http://10.97.25.252/health => {OK}
10
11
 KNOWLEDGE

Debug step
kube-apiserver
(1) & (2)
Can
=> view be deployed as:
in /etc/kubernetes/manifests/etcd.yaml
kube-apiserver communicate (1)
a => ETCD servers - Pod
Can bein k8s
deployed as:
act with ETCD) -- systemd
Pod services
Can bein k8s
deployed as:
nd the most suitable node
-- Pod
systemd services
in k8s
o KUBELET for pod creation
en worker nodes & control plan - -deployed
systemd only
services
as systemd services
n channel in same nodes / across nodes (NAT) - deployed as daemonSet

Main difference (RC vs RS)

=> "Selector" to specify which Pod is included in RS

Useful when:
- Many pod with "labels" exists before of RS creation
=> create RS to include them all
ce clusterIP by ssh to worker nodes & curl cluster IP
Reason: because of docker networking
e | grep uac => clusterIP: 10.97.25.252:80
- docker0 as NAT ("veth" connect between docker virtual network & hos
5.252/health => {OK}
netes/manifests/etcd.yaml

ystemd services
nSet

vs RS)
ify which Pod is included in RS

ls" exists before of RS creation

de them all
docker networking
eth" connect between docker virtual network & host network)
 1.SCHEDULING KNOWLEDGE

No Error /etc/kubernetes/manifests Debug

2 dynamicPod vs staticPod => all manifests in /etc/kubernetes/manife
kubernetes.io/hostname
3 assign Pod to nodes (part1) failure-domain.beta.kubernetes.io/zone
beta.kubernetes.io/instance-type
4
- PreferNoSchedule: 50% sure for NOT assign new pod 3) Affinity & anti-affinity:
- Taints: used for NODES
- Toleration: use for PODs, only 80% guaran
- Affinity: use for PODs, 100% guaran
TAINTS / TOLERATION vs AFFINITY
5
Differentiate ??? Example for (*):
- Nodes: taints by "blue / red / green"
- Pods: tolerate also by "blue/red/green".
But there's case, pod still go to "OTHER"

5 Requests / Limits -- ifDynamic (with Scheduler)

specify "REQUESTS" = 64Mi, not limit
6 How many way to deploy & assign pod => use scheduler to determine worker nod
How many methods to schedule pod to nodes
- Manually -> using "nodeName" in pod m
7 kubectl get events - events for scheduling pod into workers
8 select pod based on "label" kp
B) --selector env=dev
If same <my-2nd-scheduler> on 2 masters
9 deploy "additional custom scheduler" on same cluster => use options:
--scheduler-name=<my-2nd-scheduler>
NG KNOWLEDGE

kubernetes/manifests Debug step

all manifests in /etc/kubernetes/manifests are created when kubelet starts
ernetes.io/hostname
ure-domain.beta.kubernetes.io/zone
a.kubernetes.io/instance-type
- matchExpressions: (A)
ffinity & anti-affinity:
- key: ....
nts: used for NODES
eration: use for PODs, only 80% guarantee pods go to correct node (*)
nity: use for PODs, 100% guarantee pods go to correct node (use with taints)

mple for (*):

des: taints by "blue / red / green"
ds: tolerate also by "blue/red/green".
there's case, pod still go to "OTHER"
- default:
amic (with Scheduler) : start
notbylimit
calling to API Server
ecify "REQUESTS" = 64Mi, spec: memory: 512Mi
use scheduler to determine worker node
containers:
nually -> using "nodeName" in pod manifest yaml
nts for scheduling pod into workers
selector env=dev
same <my-2nd-scheduler> on 2 masters, which will becomes main scheduler ??? name: mypod
se options: spec:
eduler-name=<my-2nd-scheduler> schedulerName: my-2nd-scheduler
 image: nginx:1.7.9
ons: (A)
ports:

2nd-scheduler
 LOGGING & MONITORING KNOWLEDGE

No Error Debug step

cAdvisor - containerAdvisor verify:

(subcomponent of kubelet) --------------
k top nodes
=> collect & report metrics of containers
to "metrics Servers" k top pods
 APPLICATION LIFE CYCLE

No Error Debug step

- Recreate: delete ALL, then start ALL
1 deployment "StrategyType"
-# Rolling update: update one-by-one
Get status
2 rolling update
kubectl rollout status deployment.v1.apps/nginx-
3 rollback
4 how many pods can be down at rollingUpd- use RollingUpdateStrategy: 25%maxUnavailable
pod specs: command & args in Pod Spec
4 command: overwrite ENTRYPOINT
args: overwrite CMD Secret / ConfigMap
echo -n "string_secret" | base64
5 create secret base64 -=>env (get each key "\n"
seperately)
remove newline character
6 Inject into Pods - envFrom (get all key at once)
- mountVolume
FE CYCLE

ug step Note
hen start ALL
one-by-one
# Scaling deployment
loyment.v1.apps/nginx- kubectl scale deployment.v1.apps/nginx-
egy: 25%maxUnavailable
n Pod Spec

gMap
base64
rately)
haracter
once)
 1.CLUSTER MAINTENANCE KNOWLEDGE

No Error Debug step

"--pod-eviction-timeout=5m0s" /
2 kpod eviction
drain time
node01 node01
OR
3 k cordon node01 -- master
markednodes"unscheduling"
are "tained"
4 kwhy
uncordon node01 - give the node back
master nodes can not be assigned po only pod with "toleration" to normal
couldstate (can beassign
be forceful
to master nodes cluster process
kubelet 1.Upgrade
/ kube-proxy: X-2
5 versions rule for cluster component kubectl:upgrade
Step2: X-1 => X+1workers
6 upgrade process + Options2A: all workers at a time => down time (faster)
kubeadm upgrade apply <version> -+kubeadm
Options2B:upgrade
rollingappy 1.12.0
node-by-node => no down time (slower)
7
--------------------------------------------------------------------
2.Backup & Restore Method (ETCD)
1. Query from API server & importer later on
8 Back up cluster
2. Backup & restore ETCD data
9
/etc/kubernetes/pki/etcd/ca.crt ca.crt -> cert file for CA whole etcd cluster
10
/etc/kubernetes/pki/etcd/server.crt server.crt -> cert file for server of etcd cluster
--key="/etc/kubernetes/pki/etcd/server.key" \
11 NOTE: should practice again
--cert="/etc/kubernetes/pki/etcd/server.crt"
--initial-cluster-token etcd-cluster-1 \ \
12 https://kodekloud.com/p/practice-test-
Restore ETCD data
--initial-advertise-peer-urls http://host1:2380 \
CE KNOWLEDGE

ug step Note
=5m0s" /

ed"
ormal
n" couldstate (can beassign
be forceful
2r process

s at a time => down time (faster)

yde-by-node
1.12.0 => no down timeWorker (slower)
node (only kubelet)
-----------------------------------
ethod (ETCD) - kubectl drain node01
& importer later on
D data
CA whole etcd cluster
server of etcd cluster
netes/pki/etcd/server.key" \ -> using etcdctl API version "3"
netes/pki/etcd/server.crt"
cd-cluster-1 \ \ ->
-> remember to API
using etcdctl use version "3"
urls http://host1:2380 \
 SECURITY KNOWLEDGE

=path_to_csvfile" in kube-apiserver.yaml Authentication & Certificates

- RBAC (Role based...)
1 / Certificates (TLS) - ABAC (Attribute based...)
/ LDAP - Node Authorization

- Private key (có chữ "key" trong filename)

2 TLS
- Public key (others: pem, crt)

=> /etc/kubernetes/pki/{ca.crt, ca.key}

3 2 CA (certificate Authority)
- For etcd cluster
"Certificate + key" (.crt + .key) cho từng:
1) services (client) / end-user
- admin
- kube-scheduler
- kube-controller-manager
- kube-proxy
=> talk to kube-apiserver

4 2) đầu server / client

Below talks to "Server"
- apiserver-kubelet (client => kubelet)
- apiserver-etcd (client => etcd)

Below server get requests from "client"

- kubelet (server)
- etcd cluster (server)
- api-server (server) (/etc/k8s/pki/{ca.crt,ca.key}
5 Quy trình tạo cert cho new user
- create role + rolebinding with specific RBAC (only get 2. pod,
Tạo "private
...) key" cho new user
- verify permission "baohuynh-test" with "can-i" plugin 3. Tạo "certificate signing request"
- expire day (*)
6 Có thể có 2 CA cấp)
(certificate Authorization): openssl x509 -noout -text -in apiserver.crt
- issuer (người
7 -Debug
[1] CAincho K8S
case service (kube-apiserver,
kube-apiserver failed kube-scheduler,...)
[1]
=> -check
/etc/kubernetes/pki/
wrong path_cert {ca.crt, ca.key}
8 - [2] CA cho etcd cluster [2] - /etc/kubernetes/pki/etcd/
B) " docker ps -a | grep kube-api {ca.crt, ca.key} logs
" =>"docker
Example test: https://kodekloud.com/p/practice- <..>"
kube cluster
9
Thay vì phải tự tạo file .crt = cách lấy 2 file {ca.crt, ca.key}
3/As an admin, Appove/deny pending csr
- Role / Clusterrole RBAC Authorization
10 RBAC - Rolebinding / Clusterrolebinding
(for user/group/serviceAccount)

ResourceNames (1) giới hạn quyền, chỉ truy cập vào pod tên "BaoHuyn

NOTICE
 list all of apiGroups & related resources
=> needed "config" info for role / clusterrole
NOTICE
+ Pod, deployment, svc, secret,... => apiGroups: ["
kubectl api-resources (2) + persistentvolumes/... => apiGroups:
["storage.k8s.io"]

(check by "kubectl api-resrouce | grep <keywrord>

11 Inspect "authorization mode" -> view config in manifest of API-server
Aggregated ClusterRoles
12
Ex:

Image security
1. Create secret (stores docker login credentials)
--> see next cell
2. Using imagePullSecrets (built-in) in .yaml
apiVersion: v1
kind: Pod
Pull from private repository: ................
13
spec:
containers:
- name: private-reg-container
image: <your-private-image>
imagePullSecrets:
- name: private-reg-cred
Security Context
runAsGroup: 3000 runAsUser: 1000
- capabilities => some priviledges in Linux runAsGroup: 3000
- from Network Policies
user: alice
- to
podSelector:
+ ipBlock
ntication
..) & Certificates
ed...)

"key" trong filename)

pem, crt)

pki/{ca.crt, ca.key}

a.key}
-subj "/O=$GROUP/CN=$USER"
ho new user
4) openssl x509 -req -days 3650 -sha256 -in $CLIENTCERTCSR \
ning request"
ut -text -in apiserver.crt
pki/
_cert{ca.crt, ca.key}
pki/etcd/
ep kube-api{ca.crt, ca.key} logs
" =>"docker C) Cert expires
=> regenerate from .csr + .key
spec:
ve/deny pending csr request: $(cat server.csr | base64 | tr -d '\n')
BAC Authorization Impersonate & check access
rrolebinding - kubectl auth can-i <user-name> get pods
ceAccount) -> yes / no

uy cập vào pod tên "BaoHuynh" apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole
metadata:
name: storage-admin
rules:
- apiGroups: ["storage.k8s.io"] <= (1)
resources: ["storageclasses", "persistentvolumes"]
resourceNames: ["BaoHuynh"] <= (2)
verbs: ["*"]
 apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
related resources metadata:
fo for role / clusterrole name: storage-admin
rules:
vc, secret,... => apiGroups: [""] - apiGroups: ["storage.k8s.io"] <= (1)
.. => apiGroups: resources: ["storageclasses", "persistentvolumes"]
resourceNames: ["BaoHuynh"] <= (2)
verbs: ["*"]
i-resrouce | grep <keywrord>)
ifest of API-server name: monitoring
aggregationRule:
clusterRoleSelectors:
Image security
es docker login credentials)

rets (built-in) in .yaml kubectl create secret docker-registry \

private-reg-cred \
--docker-username=dock_user \
--docker-password=dock_password \
[email protected] \
--docker-server=myprivateregistry.com:5000
container
te-image>
cred
Security Context

Network Policies matchLabels:

role: client
- ipBlock:
 labels:
rbac.example.com/aggregate-to-monitoring: "true"
# These rules will be added to the "monitoring" role.
 STORAGE

No Error delete) Debug step

1 emptyDir - Can be mounted at different path of multi-container
EBS
inside/ AzureDisk
a Pod <----> PV <--------> PVC <-------> Pod Volume Declare
3 -Flow
PVCVolume Usage - NFS (ngag hàg với Container trong
2 - Mount -directly
....... into Pod
Declare
  Volume
5 + Create a volume for PVC / Direct hostPath
Volume Mount (inside Pod) storage
4 PV reclaiming policies + Create MountPath for using that volume
deleting will make PVC in state Root cause:on cloud provider still exist EBS, Azure
PVC is "in-use" by pod
6 emtyDir:
"Terminating" forever => couldatnot "terminated"
7 emtyDir vs hostPath - Define "Pod"
8 - Exist during Pod life-cycle
9
 Note

------> Pod Volume Declare <----> Pod VolumeMount (with Path)

gag hàg với Container trong yaml)
apiVersion: v1 task-pv-claim
claimName: accessModes:
kind: PersistentVolume
container:
- Delete: disappear with PVC
- Recycle: clear old data + available for
hostPath:
- Define at "PV"
- Exists outside pod's life-cycle
 claimName: task-pv-claim
containers:
- name: task-pv-container
 NETWORKING KNOWLEDGE
No Error Debug step
- standards cho việc thiết lập các
1
route/forward-traffic between host & - kube-apiserver: public (0.0.0.0:6443)
2 Cluster networking exposure - kubelet: public (0.0.0.0:10250)
- etcd: 1.CNI / Overlay
public (0.0.0.0:2379)
network ( weave )
inpsect which network plugin is used => --cni-bin-dir=/opt/cni/bin \ => (list plugins carried by weave)
4 weave plugin (support CNI) =>
by kubelet - (assign IP & create NAT table) for (configuration
--cni-conf-dir=/etc/cni/net.d \ => traffic for weave)
5 daemonset:
docker0 vs weave (10.140.0.0/12) internal IP -> external IP
6 pod get IP from weave subnet
PODs
(10.140.0.0/12)
8
- Define by weave add-on (daemonset)
2.kube-proxy (Service network)
- accessible cluster wide (from any places in
10 Service ClusterIP
cluster)
kube-apiserver:
11 Where defines service IP ranges
kube-proxy --service-cluster-ip-range=10.96.0.0/12
- represent the "Service" + convey traffic
12 => define IP of Pod & Service
from/to "service"
13 CNI (weave) vs kube-proxy
2/ kube-proxy 3.DNS in K8S
curl http://sticky-ads.default.svc.cluster.local
14 DNS record hold by coredns
- Full-qualified name for Pod (MUST)

- configmap - coredns => main configure for

CoreDNS
15
- daemonset + service - coredns (use above
cm as Config File)
where to config root domain name
server (cluster.local) ???

$ cat /var/lib/kubelet/config.yaml:
..................
clusterDNS:
16
- 10.96.0.10 => IP for coreDNS service
clusterDomain: cluster.local
----------
17 examine root domain for each pod search default.svc.nextgen.cloud \
+ subdomain: name of svc name: default-subdomain
svc.nextgen.cloud \
18
spec:
- ClusterFirst: match "*.cluster.local" then using
19 Pod DNS Policies
DNS by coredns, other will use DNS resolve
4.Ingress
 Service ( Ingress ) (nodePort / LB)
||
V
Services (wear svc / video svc)
||
20 Ingress overview
V
Backend Pods

- Ingress Controller
- Ingress Resource (/trogiup, /vuitrangram)

Host configure on the ingress-resource

HOST:
EDGE

- kube-scheduler: localhost:10251
- kube-controller-manager: localhost:10252
weave )
list plugins carried by weave)
configuration for weave)
+ role + clusterrole
+ rolebinding + clusterrolebinding
{ "dst": "0.0.0.0" }
]
work) }

- without kube-proxy, no routing table => packet don't

configmap - coredns
=> mount to /etc/coredns/Corefile in Pod deploy network policies
----
.:53 {
errors health
kubernetes nextgen.cloud in-addr.arpa ip6.arpa
{
pods insecure
upstream fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
proxy . /etc/resolv.conf
cache 30
reload
}
For every pod started by Kubelet, it will inherit configure
from this:
/var/lib/kubelet/config.yaml

lookup svc IP in different namespaces:

kubectl exec -it myweb-xyz nslookup mysql.payroll
>hostname: busybox-1
nslookup.out
subdomain: default-subdomain nameservers:
- None: ignore k8s cluster DNS configure, use it owns - 1.2.3.4
"dnsConfig" on pod specs
4.Ingress searches:
Architect

Ingress Controller architecture:

-----------------------------------------
- configmap:
+ SSL, logpath, header,.....

- Ingress controller (deployment):

+ container: ....
+ args: --configmap=$(POD_NS)/nginx-configmap
+ env: ....
+ ports: http(80) & https (443)

- Service
+ serviceType: nodePort / LB

- Authentication:
-+ ServiceAccount
with + clusterrolebinding
"Host": only for that domain
(ex: trogiup.chotot.com)
=> networking.k8s.io/v1beta1 => extensions/v1beta1 check by "kubectl api-resources"
i-resources"
 TROUBLESHOOTING KNOWLEDGE

No Error Debug
- check application configure (DB_user, step
DB_pass,...)
1 Application failure + kube-replication-controller-manager
2 ControlPlan failure Verify application working:
+ kube-scheduler
+ kube-scheduler
3 node failure
4 networking failure ???
 NOWLEDGE

Debug
onfigure (DB_user, step
DB_pass,...)
ontroller-manager
king:
 OTHER KNOWLEDGE

No Useful when (wildcart): Error Debug step

kn k8s-master-v3-77 -o jsonpath
1 - getting all of nodes's status
-json
getting - if [ ] :=> list / array
2 withcurrent restart
root format (.) count....
new line {"\n"} -Ex:
if {} :=> dictionary
3
tab {"\t"} kn -o jsonpath='{.items[*].metadata.name}

--custom- Ex: sort pv by their capacity with custom column fir

4 columns=HEADER1:.metada.name,HEADER2:.spe k get pv -o custom-
c.capacity.storage columns=NAME:.metadata.name,CAPACITY:.spec
by='{.spec.capacity.storage}' > pv-and-capacity-sor
5 --sort-by='{spec.capacity.storage}'
**** IF conditon **** NOTE: --custom-columns + --sort-by DONT use .ite
6 k config view --kubeconfig=/root/my-kube-config -o
Ex: select the context (name) for user "aws-user" by
- jsonpath='{}' (@.context.user == "aws-user")]
='{.items[*]}'
Ex: kn -o jsonpath='{range .items
='{range .items[*]}'
.status.conditions[*]}{.type}{.status
='{range .items[*]}{.metadata.name}:'
='{range .items[*]}{.metadata.name}:{.spec.taints[*]}'
='{range .items[*]}{.metadata.name}:{range .spec.taints[*]}{.effect}'
='{range .items[*]}{.metadata.name}:{range .spec.taints[*]}{.effect}{.key}'
='{range .items[*]}{"\n"}{.metadata.name}:{range .spec.taints[*]}{.effect}{.key}'
E

Debug step
-o jsonpath='{.metadata.name}' (syntax like awk)

s[*].metadata.name}{"\n"}{.items[*].status.capacity.cpu}'
apacity with custom column first

adata.name,CAPACITY:.spec.capacity.storage --sort-
torage}' > pv-and-capacity-sorted.txt

mns + --sort-by DONT use .item[*] in query

onfig=/root/my-kube-config -o jsonpath='{.contexts[?
kn -o jsonpath='{range
aws-user")].name}' .items[*]}{"\n"}{.metadata.name}:{range .spec.taints[*]}--
> /opt/outputs/aws-context-name
{.effect}'
kn -o jsonpath='{range .items[*]}{"\n"}{.metadata.name}:{range .status.conditions[*]}
kn -o jsonpath='{range .items[*]}{"\n"}{.metadata.name}:{.spec.taints[*].effect}:{range
{.type}:{.status}**'
us.conditions[*]}{.type}{.status}--'
 Kubernetes The Hard Way

No Error option: Debug step

1 - High availability kube-controller-manager
kube-scheduler ETCD HA cluster
Leader election - Raft algorithm (first node appear when
2
mechanism create cluster)
ALWAYS choose "ODD number"

=> 1. For Quorum with "node down" Tolerance & g

using Quorum = N/2 + 1
=> 2. In case of network split into 2 separate ones
3 Faiure Tolerance -> If 3 nodes, 2 write ok -> successful
suppose cluster got 6 nodes
write to cluster
each splitted cluster have 3 nodes
(accept 1 down)
-> don't have enough QUORUM (4 nodes) for s
-> corrupted
(see below example)

5
6
7
8
9
se "ODD number"

um with "node down" Tolerance & get less VMs setup

f network split into 2 separate ones

uster got 6 nodes
d cluster have 3 nodes
e enough QUORUM (4 nodes) for successful write
d
example)
 OTHER KNOWLEDGE

No Error Ex: Debug step

1 kubectl proxy POD_NAME=$(kubectl
- kubectl get pods -o json | jq
run --generator=deployment/v1beta1 -r '.items[].m
nginx --image=
2 - Using kubectl to create pod curl http://localhost:8001/api/v1/namespaces/default/pod
--dry-run -o yaml
k run nginx --image=nginx --restart=Never
3 run pod without create Deployment
k run nginx --image=nginx --restart=Never -o yaml > pod.yam
kubectl config view --kubeconfig=/root/my-kube-config
EDGE

Debug step
ubectl get pods -o json | jq
tor=deployment/v1beta1 -r '.items[].metadata.name'
nginx )
--image=nginx --replicas=4
st:8001/api/v1/namespaces/default/pods/$POD_NAME/proxy/
nginx --restart=Never
nginx --restart=Never -o yaml > pod.yaml
kubeconfig=/root/my-kube-config