17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming!

· GitHub

Instantly share code, notes, and snippets.

tuannvm / helm-cheatsheet.md
Last active 4 hours ago

Embed <script src="https://gist. Download ZIP

#Helm #Kubernetes #cheatsheet, happy helming!

helm-cheatsheet.md

Helm cheatsheet
Helm cheatsheet
Get started
Struture
General Usage
Template
Hooks
Chart repository
Signing
Test
Flow Control
If/Else

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 1/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

With
Range
Variables
Named Templates
Files inside Templates
Glob-patterns & encoding
YAML reference

Get started
https://deis.com/blog/2016/getting-started-authoring-helm-charts/
https://docs.bitnami.com/kubernetes/how-to/
https://github.com/kubernetes/helm/blob/master/docs/charts.md
https://docs.helm.sh/chart-template-guide/
http://helm.readthedocs.io/en/latest/architecture/

Struture

.
├── Chart.yaml --> metadata info
├── README.md
├── requirements.yaml --> define dependencies
├── templates
│ ├── spark-master-deployment.yaml --> configuration with template supported
│ ├── spark-worker-deployment.yaml
│ └── spark-zeppelin-deployment.yaml
│ └── NOTES.txt --> display when run "helm chart"

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 2/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

│ └── _helpers.tpl --> template handler

└── values.yaml --> variable list, will be interpolated on templates file during deployment
│
└── charts
├── apache/
├── Chart.yaml

Chart.yaml

name: The name of the chart (required)

version: A SemVer 2 version (required)
description: A single-sentence description of this project (optional)
keywords:
- A list of keywords about this project (optional)
home: The URL of this project's home page (optional)
sources:
- A list of URLs to source code for this project (optional)
maintainers: # (optional)
- name: The maintainer's name (required for each maintainer)
email: The maintainer's email (optional for each maintainer)
engine: gotpl # The name of the template engine (optional, defaults to gotpl)
icon: A URL to an SVG or PNG image to be used as an icon (optional).
appVersion: The version of the app that this contains (optional). This needn't be SemVer.
deprecated: Whether or not this chart is deprecated (optional, boolean)
tillerVersion: The version of Tiller that this chart requires. This should be expressed as a SemVer range:

requirements.yaml

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 3/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

Adding an alias for a dependency chart would put a chart in dependencies using alias as name of new
dependency. Condition - The condition field holds one or more YAML paths (delimited by commas). If this
path exists in the top parent's values and resolves to a boolean value, the chart will be enabled or disabled
based on that boolean value. Only the first valid path found in the list is evaluated and if no paths exist then the
condition has no effect. Tags - The tags field is a YAML list of labels to associate with this chart. In the top
parent's values, all charts with tags can be enabled or disabled by specifying the tag and a boolean value.
Conditions (when set in values) always override tags

dependencies:
- name: apache
version: 1.2.3
repository: http://example.com/charts
alias: new-subchart-1
condition: subchart1.enabled, global.subchart1.enabled
tags:
- front-end
- subchart1

- name: mysql
version: 3.2.1
repository: http://another.example.com/charts
alias: new-subchart-2
condition: subchart2.enabled,global.subchart2.enabled
tags:
- back-end
- subchart1

General Usage

helm list --all

helm repo (list|add|update)
helm search

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 4/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

helm inspect <chart-name>

hem install --set a=b -f config.yaml <chart-name> -n <release-name> # --set take precedented, merge into -
helm status <deployment-name>
helm delete <deployment-name>
helm inspect values <chart-name>
helm upgrade -f config.yaml <deployment-name> <chart-name>
helm rollback <deployment-name> <version>

helm create <chart-name>

helm package <chart-name>
helm lint <chart-name>

helm dep up <chart-name> # update dependency

helm get manifest <deployment-name> # prints out all of the Kubernetes resources that were uploaded to the
helm install --debug --dry-run <deployment-name> # it will return the rendered template to you so you can

--set outer.inner=value is translated into this:

outer:
inner: value

--set servers[0].port=80,servers[0].host=example:

servers:
- port: 80
host: example

--set name={a, b, c} translates to:

name:
- a

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 5/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

- b
- c

--set name=value1,value2:

name: "value1,value2"

--set nodeSelector."kubernetes.io/role"=master

nodeSelector:
kubernetes.io/role: master

--set livenessProbe.exec.command=[cat,docroot/CHANGELOG.txt] --set livenessProbe.httpGet=null

livenessProbe:
- httpGet:
- path: /user/login
- port: http
initialDelaySeconds: 120
+ exec:
+ command:
+ - cat
+ - docroot/CHANGELOG.txt

--timeout
--wait
--no-hooks
--recreate-pods

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 6/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

Template
Values that are supplied via a values.yaml file (or via the --set flag) are accessible from the .Values object in
a template

Release.Name:
Release.Time:
Release.Namespace: The namespace the chart was released to.
Release.Service: The service that conducted the release. Usually this is Tiller.
Release.IsUpgrade: This is set to true if the current operation is an upgrade or rollback.
Release.IsInstall: This is set to true if the current operation is an install.
Release.Revision: The revision number. It begins at 1, and increments with each helm upgrade.
Chart: The contents of the Chart.yaml. Thus, the chart version is obtainable as "Chart.Version" and the maint
Files: Files can be accessed using {{index .Files "file.name"}} or using the "{{.Files.Get name}}" or "{{.Fi
Capabilities: "({{.Capabilities.KubeVersion}}", Tiller "({{.Capabilities.TillerVersion}}", and the supported

{{.Files.Get config.ini}}
{{.Files.GetBytes}} useful for things like images

{{.Template.Name}}
{{.Template.BasePath}}

default value

{{default "minio" .Values.storage}}

//same
{{ .Values.storage | default "minio" }}

put a quote outside

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 7/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

heritage: {{.Release.Service | quote }}

# same result
heritage: {{ quote .Release.Service }}

global variable

global:
app: MyWordPress

// could be access as "{{.Values.global.app}}"

Includes a template called mytpl.tpl , then lowercases the result, then wraps that in double quotes

value: {{include "mytpl.tpl" . | lower | quote}}

required function declares an entry for .Values.who is required, and will print an error message when that
entry is missing

value: {{required "A valid .Values.who entry required!" .Values.who }}

The sha256sum function can be used together with the include function to ensure a deployments template
section is updated if another spec changes

kind: Deployment
spec:
template:
metadata:
annotations:

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 8/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

checksum/config: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }}

[...]

The annotation "helm.sh/resource-policy": keep instructs Tiller to skip this resource during a helm delete
operation

In the templates/ directory, any file that begins with an underscore(_) is not expected to output a
Kubernetes manifest file. So by convention, helper templates and partials are placed in a _helpers.tpl file.

Hooks
Read more

include these annotation inside hook yaml file, for e.g templates/post-install-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
annotations:
# This is what defines this resource as a hook. Without this line, the
# job is considered part of the release.
"helm.sh/hook": post-install, post-upgrade
"helm.sh/hook-weight": "-5"

Chart repository
Read more

Signing

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 9/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

Read more

Test
Read more

Flow Control

If/Else

{{ if PIPELINE }}
# Do something
{{ else if OTHER PIPELINE }}
# Do something else
{{ else }}
# Default case
{{ end }}

data:
myvalue: "Hello World"
drink: {{ .Values.favorite.drink | default "tea" | quote }}
food: {{ .Values.favorite.food | upper | quote }}
{{- if eq .Values.favorite.drink "lemonade" }}
mug: true
{{- end }} # notice the "-" in the left, if will help eliminate newline before variable

With
with can allow you to set the current scope (.) to a particular object

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 10/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

data:
myvalue: "Hello World"
{{- with .Values.favorite }}
drink: {{ .drink | default "tea" | quote }}
food: {{ .food | upper | quote }}
{{- end }} # instead of writing ".Values.favorite.drink"

Inside of the restricted scope , you will not be able to access the other objects from the parent scope

Range

# predefined variable
pizzaToppings:
- mushrooms
- cheese
- peppers
- onions

toppings: |-
{{- range $i, $val := .Values.pizzaTopping }}
- {{ . | title | quote }} # upper first character, then quote
{{- end }}

sizes: |-
{{- range tuple "small" "medium" "large" }}
- {{ . }}
{{- end }} # make a quick list

Variables
It follows the form $name . Variables are assigned with a special assignment operator: :=

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 11/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

data:
myvalue: "Hello World"
{{- $relname := .Release.Name -}}
{{- with .Values.favorite }}
drink: {{ .drink | default "tea" | quote }}
food: {{ .food | upper | quote }}
release: {{ $relname }}
{{- end }}

# use variable in range

toppings: |-
{{- range $index, $topping := .Values.pizzaToppings }}
{{ $index }}: {{ $topping }}
{{- end }}

#toppings: |-
# 0: mushrooms
# 1: cheese
# 2: peppers
# 3: onions

{{- range $key,$value := .Values.favorite }}

{{ $key }}: {{ $value }}
{{- end }} # instead of specify the key, we can actually loop through the values.yaml file and print value

There is one variable that is always global - $ - this variable will always point to the root context

...
labels:
# Many helm templates would use `.` below, but that will not work,
# however `$` will work here
app: {{ template "fullname" $ }}
# I cannot reference .Chart.Name, but I can do $.Chart.Name

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 12/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

chart: "{{ $.Chart.Name }}-{{ $.Chart.Version }}"

release: "{{ $.Release.Name }}"
heritage: "{{ $.Release.Service }}"
...

Named Templates
template names are global

# _helpers.tpl
{{/* Generate basic labels */}}
{{- define "my_labels" }}
labels:
generator: helm
date: {{ now | htmlDate }}
version: {{ .Chart.Version }}
name: {{ .Chart.Name }}
{{- end }}

When a named template (created with define) is rendered, it will receive the scope passed in by the template
call.

# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Release.Name }}-configmap
{{- template "my_labels" . }} # Notice the final dot, it will pass the global scope inside template file. W
{{- include "my_labels" . | indent 2 }} # similar to "template" directive, have the ability to control ind

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 13/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

referable to use include over template . Because template is an action , and not a function , there is no
way to pass the output of a template call to other functions; the data is simply inserted inline .

Files inside Templates

# file located at parent folder

# config1.toml: |-
# message = config 1 here
# config2.toml: |-
# message = config 2 here
# config3.toml: |-
# message = config 3 here

data:
{{- $file := .Files }} # set variable
{{- range tuple "config1.toml" "config2.toml" "config3.toml" }} # create list
{{ . }}: |- # config file name
{{ $file.Get . }} # get file's content
{{- end }}

Glob-patterns & encoding

apiVersion: v1
kind: ConfigMap
metadata:
name: conf
data:
+{{ (.Files.Glob "foo/*").AsConfig | indent 2 }}
---
apiVersion: v1
kind: Secret
metadata:
name: very-secret

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 14/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

type: Opaque
data:
+{{ (.Files.Glob "bar/*").AsSecrets | indent 2 }}

+token: |-
+ {{ .Files.Get "config1.toml" | b64enc }}

YAML reference

# Force type
age: !!str 21
port: !!int "80"

# Fake first line to preserve integrity

coffee: | # �no strip
# Commented first line
Latte
Cappuccino
Espresso

coffee: |- # strip off trailing newline

Latte
Cappuccino
Espresso

coffee: |+ # preserve trailing newline

Latte
Cappuccino
Espresso

another: value

myfile: | # insert static file

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 15/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

{{ .Files.Get "myfile.txt" | indent 2 }}

coffee: > �# treat as one long line

Latte
Cappuccino
Espresso

kubernetes-cheatsheet.md

Kubernetes cheatsheet
Kubernetes cheatsheet
Getting Started
Sample yaml
Workflow
Physical components
Master
Node
Everything is an object - persistent entities
Namespaces
Labels
ClusterIP
Controller manager

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 16/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

Kube-scheduler
Pod
Status
Probe
Pod priorities
Multi-Container Pods
Init containers
Lifecycle hooks
Quality of Service (QoS)
PodPreset
ReplicaSet
Deployments
ReplicationController
DaemonSet
StatefulSet
Job (batch/v1)
Cronjob
Horizontal pod autoscaler
Services
Volumes
Persistent volumes
Role-Based Access Control (RBAC)
Notes
Basic commands
jsonpath
Resource limit
https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 17/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

CPU
Memory
Chapter 13. Integrating storage solutions and Kubernetes
Downward API
Labs
Guaranteed Scheduling For Critical Add-On Pods
Set command or arguments via env

Getting Started
Fault tolerance
Rollback
Auto-healing
Auto-scaling
Load-balancing
Isolation (sandbox)

Sample yaml

apiVersion: <>
kind: <>
metadata:
name: <>
labels:
...
annotations:
...
spec:
containers:
https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 18/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

...
initContainers:
...
priorityClassName: <>

Workflow
(kube-scheduler, controller-manager, etcd) --443--> API Server

API Server --10055--> kubelet

non-verified certificate
MITM
Solution:
set kubelet-certificate-authority
ssh tunneling

API server --> (nodes, pods, services)

Plain HTTP (unsafe)

Physical components

Master
API Server (443)
kube-scheduler
controller-manager
cloud-controller-manager

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 19/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

kube-controller-manager
etcd

Other components talk to API server, no direct communication

Node
Kubelet

Container Engine

CRI
The protocol which used to connect between Kubelet & container engine

Kube-proxy

Everything is an object - persistent entities

maintained in etcd, identified using

names: client-given
UIDs: system-generated

Both need to be unique

three management methods

Imperative commands (kubectl)

Imperative object configuration (kubectl + yaml)
repeatable
observable

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 20/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

auditable
Declarative object configuration (yaml + config files)
Live object configuration
Current object configuration file
Last-applied object configuration file

Node Capacity
---------------------------
| kube-reserved |
|-------------------------|
| system-reserved |
|-------------------------|
| eviction-threshold |
|-------------------------|
| |
| allocatable |
| (available for pods) |
| |
| |
---------------------------

Namespaces
Three pre-defined

default
kube-system
kube-public: auto-readable by all users

Objects without namespaces

Nodes
https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 21/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

PersistentVolumes
Namespaces

Labels
key / value
loose coupling via selectors
need not be unique

ClusterIP

Independent of lifespan of any backend pod

Service object has a static port assigned to it

Controller manager
ReplicaSet, deployment, daemonset, statefulSet
Actual state <-> desired state
reconciliation loop

Kube-scheduler
nodeSelector
Affinity & Anti-Affinity
Node
Steer pod to node
Pod
Steer pod towards or away from pods
Taints & tolerations (anti-affinity between node and pod!)

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 22/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

Base on predefined configuration (env=dev:NoSchedule)

...
tolerations:
- key: "dev"
operator: "equal"
value: "env"
effect: NoSchedule
...

Base on node condition (alpha in v1.8)

taints added by node controller

Pod

kubectl run name --image=<image>

What's available inside the container?

File system
Image
Associated Volumes
ordinary
persistent
Container
Hostname
Pod
Pod name
User-defined envs
Services
https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 23/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

List of all services

Access with:

Symlink (important):

/etc/podinfo/labels
/etc/podinfo/annotations

Or:

volumes:
- name: podinfo
downwardAPI:
items:
- path: "labels"
fieldRef:
fieldPath: metadata.labels
- path: "annotations"
fieldRef:
fieldPath: metadata.annotations

Status

Pending
Running
Succeeded
Failed
Unknown

Probe

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 24/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

Liveness
Failed? Restart policy applied
Readiness
Failed? Removed from service

Pod priorities

available since 1.8

PriorityClass object
Affect scheduling order
High priority pods could jump the queue
Preemption
Low priority pods could be pre-empted to make way for higher one (if no node is available for high priority)
These preempted pods would have a graceful termination period

Multi-Container Pods

Share access to memory space

Connect to each other using localhost
Share access to the same volume
entire pod is host on the same node
all in or nothing
no auto healing or scaling

Init containers

run before app containers

always run to completion
run serially
https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 25/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

Lifecycle hooks

PostStart
PreStop (blocking)

Handlers:

Exec
HTTP

...
spec:
containers:
lifecycle:
postStart:
exec:
command: <>
preStop:
http:
...

Could invoke multiple times

Quality of Service (QoS)

When Kubernetes creates a Pod it assigns one of these QoS classes to the Pod:

Guaranteed (all containers have limits == requests)

If a Container specifies its own memory limit, but does not specify a memory request, Kubernetes automatically
assigns a memory request that matches the limit. Similarly, if a Container specifies its own cpu limit, but does
not specify a cpu request, Kubernetes automatically assigns a cpu request that matches the limit.

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 26/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

Burstable (at least 1 has limits or requests)

BestEffort (no limits or requests)

PodPreset

You can use a podpreset object to inject information like secrets, volume mounts, and environment variables etc into
pods at creation time. This task shows some examples on using the PodPreset resource

apiVersion: settings.k8s.io/v1alpha1
kind: PodPreset
metadata:
name: allow-database
spec:
selector:
matchLabels:
role: frontend
env:
- name: DB_PORT
value: "6379"
volumeMounts:
- mountPath: /cache
name: cache-volume
volumes:
- name: cache-volume
emptyDir: {}

ReplicaSet
Features:

Scaling and healing

Pod template
number of replicas
https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 27/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

Components:

Pod template

Pod selector (could use matchExpressions)

Label of replicaSet

Number of replica

Could delete replicaSet without its pods using --cascade =false

Isolating pods from replicaSet by changing its labels

Deployments
versioning and rollback

Contains spec of replicaSet within it

advanced deployment

blue-green

canary

Update containers --> new replicaSet & new pods created --> old RS still exists --> reduced to zero

Every change is tracked

Append --record in kubectl to keep history

Update strategy

Recreate
https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 28/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

Old pods would be killed before new pods come up

RollingUpdate
progressDeadlineSeconds
minReadySeconds
rollbackTo
revisionHistoryLimit
paused
spec.Paused

kubectl rollout undo deployment/<> --to-revision=<>

kubectl rollout statua deployment/<>

kubectl set image deployment/<> <>=<>:<>

kubectl rollout resume/pause <>

ReplicationController
RC = ( RS + deployment ) before
Obsolete

DaemonSet
Ensure all nodes run a copy of pod
Cluster storage, log collection, node monitor ...

StatefulSet
Maintains a sticky identity

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 29/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

Not interchangeable
Identifier maintains across any rescheduling

Limitation

volumes must be pre-provisioned

Deleting / Scaling will not delete associated volumes

Flow

Deployed 0 --> (n-1)

Deleted (n-1) --> 0 (successor must be completely shutdown before proceed)
Must be all ready and running before scaling happens

Job (batch/v1)
Non-parallel jobs
Parallel jobs
Fixed completion count
job completes when number of completions reaches target
With work queue
requires coordination
Use spec.activeDeadlineSeconds to prevent infinite loop

Cronjob
Job should be idempotent

Horizontal pod autoscaler

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 30/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

Targets: replicaControllers, deployments, replicaSets

CPU or custom metrics
Won't work with non-scaling objects: daemonSets
Prevent thrashing (upscale/downscale-delay)

Services
Logical set of backend pods + frontend

Frontend: static IP + port + dns name

Backend: set of backend pods (via selector)

Static IP and networking.

Kube-proxy route traffic to VIP.

Automatically create endpoint based on selector.

CluterIP

NodePort

external --> NodeIP + NodePort --> kube-proxy --> ClusterIP

LoadBalancer

Need to have cloud-controller-manager

Node controller
Route controller
Service controller
Volume controller

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 31/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

external --> LB --> NodeIP + NodePort --> kube-proxy --> ClusterIP

ExternalName

Can only resolve with kube-dns

No selector

Service discovery

SRV record for named port

port-name.port-protocol.service-name.namespace.svc.cluster.local
Pod domain
pod-ip-address.namespace.pod.cluster.local
hostname is metadata.name

spec.dnsPolicy

default
inherit node's name resolution
ClusterFirst
Any DNS query that does not match the configured cluster domain suffix, such as “www.kubernetes.io”, is
forwarded to the upstream nameserver inherited from the node
ClusterFirstWithHostNet
if host network = true
None (since k8s 1.9)
Allow custom dns server usage

Headless service

with selector? --> associate with pods in cluster

without selector? --> forward to externalName
https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 32/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

Could specify externalIP to service

Volumes
Lifetime longer than any containers inside a pod.

4 types:

configMap

emptyDir

share space / state across containers in same pod

containers can mount at different times
pod crash --> data lost
container crash --> ok

gitRepo

secret

store on RAM

hostPath

Persistent volumes

Role-Based Access Control (RBAC)

Role
Apply on namespace resources
ClusterRole

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 33/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

cluster-scoped resources (nodes,...)

non-resources endpoint (/healthz)
namespace resources across all namespaces

Custom Resource Definitions

CustomResourceDefinitions themselves are non-namespaced and are available to all namespaces.

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
# name must match the spec fields below, and be in the form: <plural>.<group>
name: crontabs.stable.example.com
spec:
# group name to use for REST API: /apis/<group>/<version>
group: stable.example.com
# version name to use for REST API: /apis/<group>/<version>
version: v1
# either Namespaced or Cluster
scope: Namespaced
names:
# plural name to be used in the URL: /apis/<group>/<version>/<plural>
plural: crontabs
# singular name to be used as an alias on the CLI and for display
singular: crontab
# kind is normally the CamelCased singular type. Your resource manifests use this.
kind: CronTab
# shortNames allow shorter string to match your resource on the CLI
shortNames:
- ct
# categories is a list of grouped resources the custom resource belongs to.
categories:
- all
validation:
# openAPIV3Schema is the schema for validating custom objects.
https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 34/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

openAPIV3Schema:
properties:
spec:
properties:
cronSpec:
type: string
pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
replicas:
type: integer
minimum: 1
maximum: 10
# subresources describes the subresources for custom resources.
subresources:
# status enables the status subresource.
status: {}
# scale enables the scale subresource.
scale:
# specReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Spec.Rep
specReplicasPath: .spec.replicas
# statusReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status
statusReplicasPath: .status.replicas
# labelSelectorPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.
labelSelectorPath: .status.labelSelector

Notes

Basic commands

# show current context

kubectl config current-context

# get specific resource

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 35/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

kubectl get (pod|svc|deployment|ingress) <resource-name>

# Get pod logs

kubectl logs -f <pod-name>

# Get nodes list

kubectl get no -o custom-columns=NAME:.metadata.name,AWS-INSTANCE:.spec.externalID,AGE:.metadata.creationTim

# Run specific command | Drop to shell

kubectl exec -it <pod-name> <command>

# Describe specific resource

kubectl describe (pod|svc|deployment|ingress) <resource-name>

# Set context
kubectl config set-context $(kubectl config current-context) --namespace=<namespace-name>

# Run a test pod

kubectl run -it --rm --generator=run-pod/v1 --image=alpine:3.6 tuan-shell -- sh

from @so0k link

access dashboard

# bash
kubectl -n kube-system port-forward $(kubectl get pods -n kube-system -o wide | grep dashboard | awk '{print

# fish
kubectl -n kube-system port-forward (kubectl get pods -n kube-system -o wide | grep dashboard | awk '{print

jsonpath
https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 36/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

From link

{
"kind": "List",
"items":[
{
"kind":"None",
"metadata":{"name":"127.0.0.1"},
"status":{
"capacity":{"cpu":"4"},
"addresses":[{"type": "LegacyHostIP", "address":"127.0.0.1"}]
}
},
{
"kind":"None",
"metadata":{"name":"127.0.0.2"},
"status":{
"capacity":{"cpu":"8"},
"addresses":[
{"type": "LegacyHostIP", "address":"127.0.0.2"},
{"type": "another", "address":"127.0.0.3"}
]
}
}
],
"users":[
{
"name": "myself",
"user": {}
},
{
"name": "e2e",
"user": {"username": "admin", "password": "secret"}
}
]
}

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 37/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

Function Description Example Result

text the plain text kind is {.kind} kind is List

the current
@ {@} the same as input
object

. or [] child operator {.kind} or {['kind']} List

recursive
.. {..name} 127.0.0.1 127.0.0.2 myself e2e
descent

wildcard. Get all

* {.items[*].metadata.name} [127.0.0.1 127.0.0.2]
objects

[start:end subscript
{.users[0].name} myself
:step] operator

{.items[*]['metadata.name', 127.0.0.1 127.0.0.2 map[cpu:4]

[,] union operator
'status.capacity']} map[cpu:8]

{.users[?
?() filter secret
(@.name=="e2e")].user.password}

range, {range .items[*]}[{.metadata.name}, [127.0.0.1, map[cpu:4]]

iterate list
end {.status.capacity}] {end} [127.0.0.2, map[cpu:8]]

quote
{range .items[*]}{.metadata.name}{'\t'}
'' interpreted 127.0.0.1 127.0.0.2
{end}
string

Below are some examples using jsonpath:

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 38/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

$ kubectl get pods -o json

$ kubectl get pods -o=jsonpath='{@}'
$ kubectl get pods -o=jsonpath='{.items[0]}'
$ kubectl get pods -o=jsonpath='{.items[0].metadata.name}'
$ kubectl get pods -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.startTime}{"\n"}{end}'

Resource limit

CPU

The CPU resource is measured in cpu units. One cpu, in Kubernetes, is equivalent to:

1 AWS vCPU
1 GCP Core
1 Azure vCore
1 Hyperthread on a bare-metal Intel processor with Hyperthreading

Memory

The memory resource is measured in bytes. You can express memory as a plain integer or a fixed-point integer with
one of these suffixes: E, P, T, G, M, K, Ei, Pi, Ti, Gi, Mi, Ki. For example, the following represent approximately the same
value:

128974848, 129e6, 129M , 123Mi

Chapter 13. Integrating storage solutions and Kubernetes

External service without selector (access with external-database.svc.default.cluster endpoint)

kind: Service
apiVersion: v1
https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 39/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

metadata:
name: external-database
spec:
type: ExternalName
externalName: "database.company.com

external service with IP only

kind: Service
apiVersion: v1
metadata:
name: external-ip-database
---
kind: Endpoints
apiVersion: v1
metadata:
name: external-ip-database
subsets:
- addresses:
- ip: 192.168.0.1
ports:
- port: 3306

Downward API

The following information is available to containers through environment variables and downwardAPI volumes:

Information available via fieldRef:

spec.nodeName - the node’s name

status.hostIP - the node’s IP
metadata.name - the pod’s name
metadata.namespace - the pod’s namespace
https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 40/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

status.podIP - the pod’s IP address

spec.serviceAccountName - the pod’s service account name
metadata.uid - the pod’s UID
metadata.labels[''] - the value of the pod’s label (for example, metadata.labels['mylabel']); available in
Kubernetes 1.9+
metadata.annotations[''] - the value of the pod’s annotation (for example,
metadata.annotations['myannotation']); available in Kubernetes 1.9+
Information available via resourceFieldRef:
A Container’s CPU limit
A Container’s CPU request
A Container’s memory limit
A Container’s memory request

In addition, the following information is available through downwardAPI volume fieldRef:

metadata.labels - all of the pod’s labels, formatted as label-key="escaped-label-value" with one label per line
metadata.annotations - all of the pod’s annotations, formatted as annotation-key="escaped-annotation-value"
with one annotation per line

Labs

Guaranteed Scheduling For Critical Add-On Pods

See link

Marking pod as critical when using Rescheduler. To be considered critical, the pod has to:
Run in the kube-system namespace (configurable via flag)
Have the scheduler.alpha.kubernetes.io/critical-pod annotation set to empty string

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 41/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

Have the PodSpec’s tolerations field set to [{"key":"CriticalAddonsOnly", "operator":"Exists"}] .

The first one marks a pod a critical. The second one is required by Rescheduler algorithm.

Marking pod as critical when priorites are enabled. To be considered critical, the pod has to:
Run in the kube-system namespace (configurable via flag)
Have the priorityClass set as system-cluster-critical or system-node-critical , the latter being the
highest for entire cluster
scheduler.alpha.kubernetes.io/critical-pod annotation set to empty string(This will be deprecated too).

Set command or arguments via env

env:
- name: MESSAGE
value: "hello world"
command: ["/bin/echo"]
args: ["$(MESSAGE)"]

tools.md

Helm
helm chart unit test https://github.com/xchapter7x/hcunit?utm_sq=g92df5t58c

Container
https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 42/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

container test: https://github.com/GoogleContainerTools/container-structure-test

AWS
SSO login: https://github.com/wnkz/aws-sso/blob/master/README.md

tuannvm commented on 12 Jul 2017 • edited Author Owner

Internal ELB configuration for Service :

kind: Service
apiVersion: v1
metadata:
name: someService
annotations:
service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0

AWS Service annotations

service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval (in minutes)

service.beta.kubernetes.io/aws-load-balancer-access-log-enabled (true|false)

service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name

service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix

service.beta.kubernetes.io/aws-load-balancer-backend-protocol (http|https|ssl|tcp)

service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled (true|false)

service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout (in seconds)

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 43/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout (in seconds, default 60)

service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled (true|false)

service.beta.kubernetes.io/aws-load-balancer-internal: '0.0.0.0/0'

service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'

service.beta.kubernetes.io/aws-load-balancer-ssl-cert (IAM or ACM ARN)

service.beta.kubernetes.io/aws-load-balancer-ssl-ports (default '*')

tuannvm commented on 17 Jul 2017 • edited Author Owner

Copy object (secret/configmap) to different namespace:

kubectl get secret gitlab-registry --namespace=revsys-com --export -o yaml |\

kubectl apply --namespace=devspectrum-dev -f -

check logs of multiple pods:

for i in (kpods | grep drone-agent | awk '{print $2}' ); echo $i ; kubectl logs --tail=10 $i; end

Get token from service account:

kubectl get sa <service-account-name> -o json | jq -r ".secrets[0].name" | xargs kubectl get secret -o json | jq -r ".data.token" | ba

Get Certificate authority:

# See https://github.com/stedolan/jq/issues/204#issuecomment-27089261
kubectl get sa <service-account-name> -o json | jq -r ".secrets[0].name" | xargs kubectl get secret -o json | jq '.data["ca.crt"]' | p

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 44/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

Start minikube

minikube start --kubernetes-version v1.9.11 --cpus 4 --memory 8092

Access service from different namespace :

<service-name>.<namespace>.svc.cluster.local

Check all supported api versions:

kubectl api-versions

Convenient one-line script to check reachability:

for from in "foo" "bar" "legacy";

do for to in "foo" "bar" "legacy";
do kubectl exec $(kubectl get pod -l app=sleep -n ${from} -o jsonpath={.items..metadata.name}) -c sleep -n ${from} -- curl
"http://httpbin.${to}:8000/ip" -s -o /dev/null -w "sleep.${from} to httpbin.${to}: %{http_code}\n";
done;
done

Get default token:

TOKEN=$(kubectl describe secret $(kubectl get secrets | grep default-token | cut -f1 -d ' ' | head -1) | grep -E '^token' | cut -f2
-d':' | tr -d '\t')

Get raw http response:

kubectl get --raw /apis/batch/v1

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 45/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

tuannvm commented on 18 Jul 2017 Author Owner

secrets.yaml

apiVersion: v1
kind: Secret
metadata:
name: "{{ template "fullname" . }}"
labels:
heritage: {{ .Release.Service | quote }}
release: {{ .Release.Name | quote }}
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
app: "{{ template "fullname" . }}"
type: Opaque
data:
{{- range $key, $value := .Values.secrets }}
{{ $key }}: {{ $value | b64enc | quote}}
{{- end }}

values.yaml:

secrets:
aws-access-key-id: ""
aws-secret-access-key: ""

inside the templates:

env:
{{- range $key, $value := .Values.config }}
- name: {{ $key | upper | replace "-" "_" }}
value: {{ $value | quote }}
{{- end}}

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 46/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

{{- range $key, $value := .Values.secrets }}

- name: {{ $key | upper | replace "-" "_" }}
valueFrom:
secretKeyRef:
name: {{ template "fullname" $ }}
key: {{ $key }}
{{- end }}

tuannvm commented on 24 Jul 2017 Author Owner

Upgrade with setting new value and retain the old one:

helm upgrade <relase-name> <chart-path>/ --reuse-values --set <key>=<value>

tuannvm commented on 27 Jul 2017 • edited Author Owner

Add checksum annotation to allow deployment upgrade while configmap/secret changed:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: {{ template "fullname" . }}
labels:
heritage: {{ .Release.Service | quote }}
release: {{ .Release.Name | quote }}
chart: "{{.Chart.Name}}-{{.Chart.Version}}"
app: "{{ template "fullname" . }}"
annotations:
checksum/config-map: {{ include (print $.Chart.Name "/templates/secret.yaml") . | sha256sum }}

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 47/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

tuannvm commented on 31 Jul 2017 • edited Author Owner

Drain kubernetes node:

kubectl drain <node-name> --ignore-daemonsets --delete-local-data --force

tuannvm commented on 31 Jul 2017 • edited Author Owner

Create kubernetes user link

Delete evicted pod:

kubectl get po -a --all-namespaces -o json | \

jq '.items[] | select(.status.reason!=null) | select(.status.reason | contains("Evicted")) |
"kubectl delete po \(.metadata.name) -n \(.metadata.namespace)"' | xargs -n 1 bash -c

check init container logs:

kubectl logs <pod-name> -c <init-container-name>

tuannvm commented on 22 Sep 2017 Author Owner

run in non-exit shell to debug:

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 48/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

command: [ "/bin/sh", "-c", "--" ]

args: [ "while true; do sleep 30; done;" ]

tuannvm commented on 16 Nov 2017 Author Owner

Useful functions for interpolation: https://godoc.org/github.com/Masterminds/sprig

tuannvm commented on 13 Dec 2017 Author Owner

jq with dash - : stedolan/jq#38 (comment)

tuannvm commented on 11 Jan 2018 Author Owner

sample deployment:

apiVersion: apps/v1beta2 # for versions before 1.8.0 use apps/v1beta1

kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 2 # tells deployment to run 2 pods matching the template
template: # create pods using pod definition in this template
metadata:
# unlike pod-nginx.yaml, the name is not included in the meta data as a unique name is
# generated from the deployment name
labels:

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 49/50
17/04/2020 #Helm #Kubernetes #cheatsheet, happy helming! · GitHub

app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80

bikranz4u commented on 7 Dec 2018

Thanks for the Cheat Sheet.

nalinguptalinux commented on 21 Aug 2019

Thanks

prashanth-sams commented on 29 Jan

Nice! If you wish to see few more in details...

https://devopsqa.wordpress.com/2020/01/29/helm-cli-cheatsheet/

https://gist.github.com/tuannvm/4e1bcc993f683ee275ed36e67c30ac49 50/50