Extending Kubernetes 101

Transcript

1. Extending Kubernetes 101
 Michael Hausenblas @mhausenblas
 Developer Advocate, Red Hat


   2018-11-15, ContainerConf, Mannheim

2. Hit me up on Twitter: @mhausenblas 2 • Developer Advocate

   @ Red Hat (Go, Kubernetes, OpenShift) • Developer Advocate @ Mesosphere (Mesos, DC/OS, Kubernetes) • Chief Data Engineer @ MapR (HDFS, HBase, Drill, etc.) • Applied research (4y in Ireland, 7y in Austria) • Nowadays mainly developing tools in Go (Python, Node, Java, C++) • Kinda developer turned ops (aka appops) $ whois mhausenblas

3. Hit me up on Twitter: @mhausenblas 3 admin SRE developer

   infosec architect PM PHB

4. Kubernetes 101

5. Hit me up on Twitter: @mhausenblas 5

6. Hit me up on Twitter: @mhausenblas 6 Kubernetes kubernetes.io •

   Container lifecycle management • Declarative API + control loops • Robust, flexible, scalable • Extensible

7. Hit me up on Twitter: @mhausenblas 7 • infrastructure admin

   • namespace admin • developer Roles and responsibilities

8. How can I customize Kubernetes?

9. Hit me up on Twitter: @mhausenblas 9 • in-tree (upstream)

   via SIG or direct PR • maintain your own fork • built-in customization approaches Customization options in principle

10. Hit me up on Twitter: @mhausenblas 10 • configuration files

    and flags (kubelet, kube-apiserver, etc.) • extension points • cloud providers • kubelet (plugins for network/devices/storage and container runtimes) • kubectl plugins • access extensions in the API server • custom resources/controllers • extension API servers • scheduler extensions Customization approaches I I A A A I I A I infrastructure API

11. Hit me up on Twitter: @mhausenblas 11 Extension patterns Example:

    manage a custom resource Example: authn/authz Example: network, storage, kubectl

12. Hit me up on Twitter: @mhausenblas 12 Cloud providers github.com/kubernetes

    • in-tree libraries/controller manager • interfaces for things like: • load balancers • network routes • nodes/VMs I

13. Hit me up on Twitter: @mhausenblas 13 kubelet: network/device/storage plugins

    • Network—standard: CNI
 github.com/containernetworking/cni 
 kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins • Devices—GPUs, FPGAs, etc.
 kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins • Storage—20+ in-tree, up-and-coming standard: CSI
 kubernetes.io/docs/concepts/storage/volumes/#types-of-volumes 
 kubernetes.io/blog/2018/04/10/container-storage-interface-beta I

14. Hit me up on Twitter: @mhausenblas 14 kubelet: container runtimes

    • Container runtime—standard: CRI (since Kubernetes 1.5)
 kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes • Nowadays multiple options: • runc • containerd • Kata containers • gVisor • hyper.sh cri-o.io I

15. Hit me up on Twitter: @mhausenblas 15 kubectl plugins •

    Extend the set of commands
 kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins • Write in any programming language (note: these are binary extensions) • Examples: context control, service catalog, user verification I

16. as simple plugin in action: kubectl inspect

17. Extending the Kubernetes API

18. Hit me up on Twitter: @mhausenblas 18 Quick control plane

    refresher

19. Hit me up on Twitter: @mhausenblas 19 The life of

    an API request Flow diagram is based on Extensible Admission is Beta and Kubernetes deep dive: API Server – part 1. persisting to etcd API HTTP handler authn & authz mutating admission object schema validation validating admission mutating webhooks validating webhooks

20. Hit me up on Twitter: @mhausenblas 20 What are (in-tree)

    core resources? A kubernetes.io/docs/reference/generated/kubernetes-api/v1.12/

21. Hit me up on Twitter: @mhausenblas 21 Access extensions in

    the API server • Admission controllers (in-tree, via configuration of the API server)
 https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/ • Dynamic Admission Control
 https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/ • Admission Webhooks (beta) • Initializers (alpha)
 A

22. Hit me up on Twitter: @mhausenblas 22 Custom resources •

    Support for “known” resources beyond core resources
 kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources
 blog.openshift.com/kubernetes-deep-dive-api-server-part-3a • Use the API server to manage custom resources in etcd for you • Custom resource definition (CRD) and instances • Use the CLI to interact with custom resources in the usual way,
 for example: kubectl get mycustomresource A

23. Hit me up on Twitter: @mhausenblas 23 Custom resource—example A

24. Hit me up on Twitter: @mhausenblas 24 Custom controller •

    Implement control loops beyond what thee (in-tree)
 controller manager supports • Custom controller • dealing with core resources
 github.com/kelseyhightower/secrets-controller • dealing with custom resources (aka operator)
 github.com/kubernetes/sample-controller A

25. Hit me up on Twitter: @mhausenblas 25 Custom resources and

    controllers A resource controller core custom in-tree custom Kubernetes control plane operator simple controller X X X X X X

26. Operators

27. Hit me up on Twitter: @mhausenblas 27 Operators operator =

    custom resource + custom controller • Motivation: application lifecycle management • Use one of 30+ available operators or write your own with: • Kubebuilder • Kubernetes Operator Kit • kutil • Metacontroller • Operator SDK A

28. Hit me up on Twitter: @mhausenblas 28 Operator use cases

    • zero-downtime upgrades of the app the operator supervises • workflow automations • policy enforcement • managing stateful workloads • resizing of followers in a distributed datastore • backup & restore of a database • re-balancing of a distributed message queue A

29. Hit me up on Twitter: @mhausenblas 29 Operator examples •

    etcd • Prometheus • Postgres • Vitess MySQL • MongoDB • Couchbase • Kafka A

30. a simple operator in action: NoDefaultsPolicy github.com/mhausenblas/operator-101

31. $ operator-sdk new nodefpol-operator

32. $ operator-sdk add api --api-version=nodefpol.k8space.io/v1alpha1 --kind=NoDefaultsPolicy

33. $ operator-sdk add controller --api-version=nodefpol.k8space.io/v1alpha1 --kind=NoDefaultsPolicy

34. $ kubectl -n ndp-demo apply -f deploy/crds/nodefpol_v1alpha1_nodefaultspolicy_crd.yaml $ OPERATOR_NAME=nodefpol-operator operator-sdk

    up local --namespace "ndp-demo"

35. grep ‘//TODO(user)’

36. Hit me up on Twitter: @mhausenblas 36 Extension API servers

    • Full control but a lot of effort and responsibility
 kubernetes.io/docs/tasks/access-kubernetes-api/setup-extension-api-server • Typically more LOC than an controller or operator • You might end up to manage storage in etcd yourself • And beyond: the Open Service Broker API and the service catalog
 kubernetes.io/docs/concepts/extend-kubernetes/service-catalog
 openservicebrokerapi.org A

37. Hit me up on Twitter: @mhausenblas 37 Scheduler extensions A

    scheduler selects a node to run your pods on, based on resource requirements, QoS, affinity, etc.
 jvns.ca/blog/2017/07/27/how-does-the-kubernetes-scheduler-work • You can modify policies or run multiple schedulers (with pod opt-in)
 kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers
 embano1.github.io/post/sched-reconcile • You can use a Webhook 
 github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/scheduler_extender.md I

38. Hit me up on Twitter: @mhausenblas 38 Other stuff you

    can customize in Kubernetes • Monitoring & alerting (Prometheus/Grafana), logging (ELK/EFK stack) • Secret management (encryption at rest, Vault) • Ingress
 kubernetes.io/docs/concepts/services-networking/ingress • DNS
 kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers • kube-proxy
 kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep-dive

39. Resources

40. Hit me up on Twitter: @mhausenblas 40

41. Hit me up on Twitter: @mhausenblas 41 • Tim Hockin—Kubernetes

    Extensibility
 speakerdeck.com/thockin/kubernetes-extensibility • Jonathan Berkhahn & Carolyn Van Slyck—Kubectl Plugins 101
 kccnceu18.sched.com/event/DqwJ/kubectl-plugins-101-jonathan-berkhahn-ibm-carolyn-van-slyck- microsoft-intermediate-skill-level-slides-attached • Adrien Trouillaud—Kubernetes Custom Resource, Controller & Operator Development Tools
 admiralty.io/kubernetes-custom-resource-controller-and-operator-development-tools.html • Toader Sebastian—A complete guide to Kubernetes Operator SDK
 banzaicloud.com/blog/operator-sdk/ • Rob Szumski—Building an Kubernetes Operator for Prometheus and Thanos
 robszumski.com/building-an-operator/ Articles and slide decks

42. Hit me up on Twitter: @mhausenblas 42 • github.com/kubernetes/kubectl/tree/master/pkg/pluginutils •

    github.com/carolynvs/kubectl-flags-plugin • github.com/jordanwilson230/kubectl-plugins • github.com/kelseyhightower/denyenv-validating-admission-webhook • github.com/kubernetes-sigs/controller-tools • github.com/kubernetes-sigs/kubebuilder • metacontroller.app • github.com/yaronha/kube-crd • github.com/operator-framework/operator-sdk • github.com/operator-framework/awesome-operators • reactiveops.github.io/rbac-manager Repos, examples, tooling

43. Hit me up on Twitter: @mhausenblas 43 • kubernetes.io/docs/concepts/extend-kubernetes/extend-cluster/ •

    kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/ • kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/ • kubernetes.io/docs/tasks/access-kubernetes-api/setup-extension-api-server/ • kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/ • kubernetes.io/docs/reference/access-authn-authz/webhook/ • kubernetes.io/docs/setup/scratch/#cloud-provider • kubernetes.io/blog/2018/01/extensible-admission-is-beta/ Kubernetes docs and blog posts

44. Hit me up on Twitter: @mhausenblas 44 • Tim Hockin

    & Michael Rubin—Kubernetes Distributions and ‘Kernels'
 https://www.youtube.com/watch?v=fXBjA2hH-CQ • Stefan Schimanski: • Kubernetes as a API driven platform, Reykjavík Kubernetes Meetup
 https://www.youtube.com/watch?v=BiE7oKeEzDU • SIG API Machinery Deep Dive
 https://www.youtube.com/watch?v=XsFH7OEIIvI • James Munnelly—Extending the Kubernetes API: What the Docs Don't Tell You
 https://www.youtube.com/watch?v=PYLFZVv68lM 
 Videos

45. plus.google.com/+RedHat linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHatNews learn.openshift.com