and flags (kubelet, kube-apiserver, etc.) • extension points • cloud providers • kubelet (plugins for network/devices/storage and container runtimes) • kubectl plugins • access extensions in the API server • custom resources/controllers • extension API servers • scheduler extensions Customization approaches I I A A A I I A I infrastructure API
Extend the set of commands kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins • Write in any programming language (note: these are binary extensions) • Examples: context control, service catalog, user verification I
an API request Flow diagram is based on Extensible Admission is Beta and Kubernetes deep dive: API Server – part 1. persisting to etcd API HTTP handler authn & authz mutating admission object schema validation validating admission mutating webhooks validating webhooks
the API server • Admission controllers (in-tree, via configuration of the API server) https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/ • Dynamic Admission Control https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/ • Admission Webhooks (beta) • Initializers (alpha) A
Support for “known” resources beyond core resources kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources blog.openshift.com/kubernetes-deep-dive-api-server-part-3a • Use the API server to manage custom resources in etcd for you • Custom resource definition (CRD) and instances • Use the CLI to interact with custom resources in the usual way, for example: kubectl get mycustomresource A
custom resource + custom controller • Motivation: application lifecycle management • Use one of 30+ available operators or write your own with: • Kubebuilder • Kubernetes Operator Kit • kutil • Metacontroller • Operator SDK A
• zero-downtime upgrades of the app the operator supervises • workflow automations • policy enforcement • managing stateful workloads • resizing of followers in a distributed datastore • backup & restore of a database • re-balancing of a distributed message queue A
• Full control but a lot of effort and responsibility kubernetes.io/docs/tasks/access-kubernetes-api/setup-extension-api-server • Typically more LOC than an controller or operator • You might end up to manage storage in etcd yourself • And beyond: the Open Service Broker API and the service catalog kubernetes.io/docs/concepts/extend-kubernetes/service-catalog openservicebrokerapi.org A
scheduler selects a node to run your pods on, based on resource requirements, QoS, affinity, etc. jvns.ca/blog/2017/07/27/how-does-the-kubernetes-scheduler-work • You can modify policies or run multiple schedulers (with pod opt-in) kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers embano1.github.io/post/sched-reconcile • You can use a Webhook github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/scheduler_extender.md I
& Michael Rubin—Kubernetes Distributions and ‘Kernels' https://www.youtube.com/watch?v=fXBjA2hH-CQ • Stefan Schimanski: • Kubernetes as a API driven platform, Reykjavík Kubernetes Meetup https://www.youtube.com/watch?v=BiE7oKeEzDU • SIG API Machinery Deep Dive https://www.youtube.com/watch?v=XsFH7OEIIvI • James Munnelly—Extending the Kubernetes API: What the Docs Don't Tell You https://www.youtube.com/watch?v=PYLFZVv68lM Videos