Kubernetes on Azure

Learn about Kubernetes benefits, challenges, and enhancements made

possible from a managed platform. Get the most out of Azure Kubernetes
Service (AKS) with top scenarios, Azure capabilities, and tools.
Table of contents
03 About Kubernetes
09 Infrastructure automation
19 End-to-end developer experience
29 Balancing agility and security
37 Secure cluster setup
49 Network segmentation
57 Top AKS scenarios
69 Additional resources
How Kubernetes works
1. Kubernetes users communicate with API server and apply desired state
2. Master nodes actively enforce desired state on worker nodes
3. Worker nodes support communication between containers
4. Worker nodes support communication from the Internet

03
 Worker node
Kubernetes
control Internet
kubelet kube-proxy

4
2
3
Docker

1
Master node Prod Prod

Containers Containers
API server

Worker node
-controller-
manager -scheduler Internet
kubelet kube-proxy

4
2
3
replication, namespace,
serviceaccounts, etc. etcd
Docker

Prod Prod

Containers Containers

04
But Kubernetes on its
own is not enough
• Save time from infrastructure management and roll out updates faster
without compromising security
• Unlock the agility for containerized applications using:
• Infrastructure automation that simplifies provisioning, patching,
and upgrading
• Tools for containerized app development and CI/CD workflows
• Services that support security, governance, and identity and access
management

Learn more at
aka.ms/k8slearning

05
 Development Platform

IDE container
support
Security Governance Identity

Source code
<\> repository

Registry
supporting
Kubernetes
Helm

CI/CD Infrastructure automation

Monitoring Virtual machines Networking

Microservice
debugging Storage Data

06
What’s behind the
Kubernetes growth?
The perceived developer benefits of Kubernetes

42% 45% 50%

portability scalability agility
07
 Azure Kubernetes
Service (AKS) momentum
AKS usage grew 30x since it was made generally available in June 2018

Trusted by thousands of customers

08
Infrastructure
automation

09
 Kubernetes gives you the knobs to schedule and deploy containers
across clusters, scale to your desired state, and manage the
Kubernetes lifecycle to keep your apps up and running.

As your applications move to production, they often span multiple

containers, deployed across a cluster of servers—increasing the
complexity of operating the knobs and taking up time you could
be spending delivering value to your customers.

A fully managed Kubernetes service, like Azure Kubernetes Service

(AKS), automates provisioning, upgrading, monitoring, and scaling
for compute resources.

10
 Manage Kubernetes
with ease
• Automated provisioning, upgrades, and patches
• High reliability and availability
• Serverless scaling
• API server monitoring
Kubernetes on Azure

• Delivered at no charge

11
 User

Self-managed master node(s)

App/workload Kubernetes etcd
definition API endpoint API server Store

Controller Cloud
Scheduler Manager Controller

Customer VMs

Docker Docker Docker Docker Docker

Pods Pods Pods Pods Pods

12
 User Azure Monitor

Azure managed control plane

Azure Container
App/workload Kubernetes Instances (ACI)
definition API endpoint
Pods

Virtual node

Schedule pods over

private tunnel
Availability Reliability
Customer VMs

Docker Docker Docker Docker Docker

Pods Pods Pods Pods Pods Auto

scaling

13
 “Thanks to AKS, we can now spin up new
demo environments in 10 minutes instead
of 24 hours. Moving Docushare Flex from
virtual machines to containers in Azure
allows us to provision environments
faster, empowering our sales and partner
network.”
— Robert Bingham, Director of DocuShare Cloud Operation
Xerox

14
 Virtual node
• Elastically provision capacity in seconds
• No infrastructure to manage
• Built on open-sourced Virtual Kubelet
technology, a sandbox project from CNCF
Capability

Learn more at
aka.ms/aksbook/virtualnode

15
 VM VM
Pods Pods

Azure Container Instances (ACI)

Application Pods
Architect
Kubernetes
control plane

Infrastructure
Architect
Deployment/ Virtual node
tasks

VM VM
Pods Pods

16
 Auto scale
• Efficiently scale and run apps without
downtime—all out of the box
• Automatically add or remove instances
based on resource utilization
Capability

Learn more at
aka.ms/aksbook/autoscale

17
 Out-of-the-box cluster autoscaling

Pod
Node
Cluster
Container
Nodes
Containers Pods

Automatically
Pods Containers Clusters
spin up more…

= exhausted

18
End-to-end
developer
experience

19
 Kubernetes API itself doesn’t include development tools. To run an application
in a Kubernetes cluster, a developer may use a code editor to write code and
perhaps a source code control repository to manage it; a Docker client to help
with containerization; Helm for packaging; and kubectl, or a YAML configuration
to deploy containers to Kubernetes.

In a real-world scenario, the picture becomes much more complicated. As

containers, environments, and the teams that work with them multiply, release
frequency increases—along with developmental and operational complexity. For
example, the need to merge code effectively with the option to rollback; testing
the application in a way that mimics the production environment but doesn’t
impact the production environment; and, quickly identifying and addressing any
issues without downtime. The last thing you want to have on top of this
complexity is a fragmented tool chain.

A managed Kubernetes platform designed for developers can integrate

seamlessly with your favorite IDE, CI/CD, and monitoring tools and automate
these workflows to support your Kubernetes app development. An IDE that
directly supports Kubernetes deployment can help you set up the most complex
microservices development environment and connect with your private container
registry. With built-in CI/CD and a pre-configured deployment strategy, you can
accelerate the move from code to container to Kubernetes cluster in minutes by
automating those tasks. Finally, a complete view from container health monitoring
to centralized logging can be auto-configured with your developer portal to
prevent resource bottlenecks, trace malicious requests, and keep your Kubernetes
applications healthy.

20
 Accelerate containerized
development
Develop
• Native containers and Kubernetes support in IDE
• Remote debugging and iteration for multi-containers
• Effective code merge
Automatic containerization
Kubernetes on Azure

Deliver
• CI/CD pipeline with automated tasks in a few clicks
• Pre-configured canary deployment strategy
• In depth build and delivery process review and integration testing
• Private registry with both container image and Helm chart management

Operate
• Out-of-box control plane telemetry, log aggregation, and container health
• Declarative resource management
• Auto scaling

21
 Kubernetes cluster
Source
Code editor code control YAML
Scale

Container
image

22
 Develop Deliver Operate

Azure AKS
Inner loop Source Container production
code control Registry cluster
Scale
Azure AKS dev
DevSpaces cluster
Test

Container
Debug image
Azure
Monitor

Azure Pipelines
Helm chart Terraform

23
 “We are building our
own new applications using
microservices—and AKS is our
choice for orchestrating their
workloads.”
— Ståle Heitmann, Chief Technology Officer
Hafslund Nett

24
 Azure Dev Spaces
1. “Integration” dev space runs full baseline version of app
2. John and Sanjay collaborate on FeatureX
3. Code committed to the master source control
4. CI/CD pipeline triggered to deploy into “Integration
5. Helm assets used in later environments by CD system

Dev Spaces is enabled per Kubernetes namespaces and can be defined as

anything. Any namespace in which Dev Spaces is not enabled runs unaffected.
Capability

Learn more at
aka.ms/aksbook/devspaces

25
 Source CI/CD
control pipeline
4
5
git commit
git push

3
helm upgrade helm upgrade
Container --install --install
registry values.test.yaml values.prod.yaml
AKS cluster

Lisa
Lisa
'up' or F5 debug namespace
1
values.dev.yaml
Integration Production
John namespace namespace

John namespace FeatureX

2
namespace

Sanjay Sanjay
namespace

Dev Spaces enabled

26
 Azure Pipelines for AKS
• Add a full CI/CD pipeline to your AKS cluster with automated routine tasks
and multiple deployment strategies—all set up in just a few clicks
• Detect failures early and optimize your pipelines in a heartbeat with deep
traceability into your deployments and source code
Capability

Learn more at
aka.ms/aksbook/pipelines

27
 Developer

Deep traceability

Source Container
Repository image Pod

Source Azure Pipelines Azure Pipelines AKS

code Build Release cluster
Continuous Continuous Deploy
Integration Delivery strategies
</>

Azure Monitor

Iterate Monitor

28
Balancing agility
and security

29
 Kubernetes provides built-in capabilities like namespaces and
admission controller to help with isolation and privilege
management for your Kubernetes resources. But to achieve
hardened security and meet compliance requirements, your
applications need more in-depth defense and dynamic control
that goes beyond Kubernetes itself.

As container images become the new deployment format, the

ecosystem around their security and controls is starting to emerge.
Still, enforcing security and compliance without hindering agility is
challenging and prone to error. The complexity lies in both
development and infrastructure operations. For example, how do
you embed policy requirements of your organization while the
images are getting built and deployed as part of the CI/CD
workflows?

An enterprise-grade platform designed for developers can

provide cloud services that offer deep, real-time observability for
your build and release pipelines, and apply compliance audit and
reconfigurations easily—all as part of the DevOps workflow.

30
 Put guardrails around the
development process
1. Auto-build with continuous security: Enforce pre-defined policies to build your pipeline
2. Least privilege principle: Only build pipelines that have the key/permission to push image
into registry
3. Governance: Add policy audit to your pipeline—non-compliant releases will be flagged
Kubernetes on Azure

for review and action

4. Least privilege principle: Only the release pipeline has permission to create new pods or
new applications in your Kubernetes environment
5. Open Policy Agent: Only images from trusted registries will get deployed and executed in
the cluster

31
 Kubernetes cluster
Source Container
code control image YAML

32
 Private AKS
Source Container container 5 production
code control image registry cluster
Open Policy
Agent

2
Least privilege
principle

Azure
Pipelines

Auto-build with
1 continuous security

4
3 Least privilege
principle
Governance

33
 “Using Kubernetes on Azure
satisfies our objectives for efficient
software development. It aligns
well with our digital plans and our
choice of open-source solutions
for specific programming
languages.”
— Rasmus Hald, Head of Cloud Architecture
A.P. Moller - Maersk

34
 Azure Pipelines to deliver;
Azure Policy to enforce
1. Cloud architect assigns a policy across clusters; policy can be set to block non-compliance
(deny) or generate non-compliance warnings (audit)
2. Developer makes code change that kicks off an Azure Pipelines build
3. Azure Pipelines evaluates the request for policy compliance
4. If policy is set to deny, Azure Pipelines rejects the build attempt if any non-compliance is
identified
5. If policy is set to audit, a non-compliance event is logged and the build is allowed to
proceed
Capability

Learn more at
aka.ms/aksbook/policy

35
 Cloud Azure
Architect Policy

Azure Pipelines
4
Deny policy
Yes </> No
Fail 5
Developer
Compliance check
AKS

2 </> 3
Cluster-1 Cluster-2 Cluster-3

Pass

36
Secure cluster
setup

37
 As a cloud-native container orchestration tool, Kubernetes provides
various access points to its users. These include the API server and
kubectl to access it via the command line, kubelet for interacting
with the container runtime, and etcd storage for state and cluster
information, just to name a few.

Malicious access to any of the above can be severely problematic.

While you can use Kubernetes settings and associated best practices
to manage security, production systems demand hardened security
that goes beyond configurations and settings.

To secure your cluster, you want to build on a secure, enterprise-

grade platform that can easily incorporate solutions for identity
and access management (IAM), secrets management, and policy
enforcement without introducing a steep learning curve for your
team. For example, you can use Azure Active Directory to get fine-
grained identity and access control to Kubernetes resources from
cluster to containers, while Azure Policy can provide rules
enforcement across multiple clusters.

38
 Hardened security for
Kubernetes resources
• Get secure login and fine-grained identity and access control to Kubernetes
resources from cluster to containers
• Securely store and centrally manage secrets outside the cluster using Azure
Key Vault Flex Volumes
Kubernetes on Azure

• Validate requests to pods and define conditions required for pods to run in
cluster using Pod Security Policy
• Enforce and synchronize access control with other services required for the
application with identity for Kubernetes pods in the same IAM solution
• Record, monitor, investigate API calls for suspicious activities using audit
logging
• Audit and enforce rules defined in Azure Policy across multiple clusters in
real-time—powered by Open Policy Agent

39
 Application

Authorization and
authentication
Clusters

Master Node Worker Nodes

Namespace
Control plane
components etcd kubelet Pods
Resource
isolation and
governance

Secrets and
Request to pods Service-to-service identity
configuration access

Control plane access:

Kubernetes API traffic audit

40
Application

Secure user login: IAM solution:

Kubernetes API audit logging Azure Active Directory for RBAC
Clusters

Master Node Worker Nodes

Namespace
Control plane
components etcd kubelet Pods
Governance:
Azure Policy

Service-to-service access:
pod identity using AAD
Secrets management:
Pod security policy
Azure Key Vault Azure
Storage Database

Secure user login:

Kubernetes API audit logging

41
 “Using Azure Kubernetes Service puts
us into a position to not only deploy
our business logic in Docker containers,
including the orchestration, but also…
to easily manage the exposure and
control and meter the access.”
— Thomas Gossler, Lead Architect, Digital Ecosystem Platform
Siemens Healthineers

42
 Pod identity
1. Kubernetes operator defines an identity
map for K8s service accounts
2. Node Managed Identity (NMI) watches for
mapping reaction and syncs to Managed
Service Identify (MSI)
3. Developer creates a pod with
a service account, and pod uses standard
Azure SDK to fetch a token bound to MSI
4. Pod uses access token to consume other
Azure services; services validate token
Capability

Learn more at
aka.ms/aksbook/podidentity

43
 Developer

<\>

3
Kubernetes

Kubernetes Azure
controller Identity Azure SQL
Binding Pod Server
1

Active Token
Directory
Pod Identity Azure MSI

NMI + EMSI

44
 Identity and access
management through
AAD and RBAC
1. A developer authenticates to the AAD token issuance endpoint and requests
an access token
2. The AAD token issuance endpoint issues the access token
3. The access token is used to authenticate to the secured resource
4. Data from the secured resource is returned to the web application
Capability

Learn more at
aka.ms/aksbook/aad

45
 Azure Active
Directory

Developer
Token 2
AKS

Token

46
 Azure Policy for
Kubernetes clusters
1. Cloud architect assigns a deployment policy across cluster(s)
2. Developer uses standard Kubernetes API to deploy to the cluster
3. Real-time deployment enforcement (acceptance/denial) provided
to developer based on policy
4. Cloud architect obtains compliance report for the entire
environment and can drill down to individual pod level
Capability

Learn more at
aka.ms/aksbook/policy

47
 Cloud Azure
Architect Policy

Compliance reports

Cluster-1 Cluster-2 Cluster-3

4


AKS
Developer
3 Cluster-1 Cluster-2 Cluster-3

2

48
Network
segmentation

49
 As you decompose your application into microservices, the
complexity of management and networking—both within the
cluster and to external services—increases.

Kubernetes applications are distributed by nature. Native

components such as ingress, kube-proxy, and namespaces assist
service discovery, load balancing, and segmentation, but are
insufficient for the secure communication paths required by your
workloads in production.

Capabilities like virtual networks, network policy, and application

gateways can help set up a solid foundation for secure networking.
An enterprise-grade platform can also offer you hybrid networking
capabilities that help utilize your existing technology investment.

50
 Secure communication
path
• Create an isolated environment using Azure Virtual Network to allow only
authenticated IPs to access your network
• Protect against threats and intrusions using App Gateway with WAF
• Secure communication paths between namespaces (and nodes) using
Kubernetes on Azure

network policy
• Connect to on-premises infrastructure using Azure Express Route
• Secure connection between VNets with VNet peering

51
 Kubernetes cluster

Control plane

Worker Node Worker Node

…
kubelet Pods Pods kubelet

Containers Containers

Namespace

52
 Other peered VNets App Enterprise system
Gateway Azure
VNet
Express
peering
Azure VNet Route

Kubernetes cluster

Internal Load External DNS

Balancer

Control plane Ingress controller

Worker Node Worker Node

…
kubelet Pods Pods kubelet

Containers Containers

Namespace

53
 “Azure support for Docker,
Kubernetes, Puppet, Terraform,
Cassandra, and other open source
tools has become very important
to us and has really accelerated
our move into Azure.”
— Robert Rudduck, Director of Architecture and DevOps,
Ambit Energy

54
 Network policy
• Secure communication paths between namespaces
and nodes
• Better controls with user-defined network policy
• All powered by Calico, an open source project
Capability

Learn more at
aka.ms/aksbook/networkpolicy

55
 Project

Cluster

Node Node

Namespace Namespace

Pod

Containers

56
Top AKS scenarios
Lift and shift Machine
Microservices IoT DevSecOps
to containers learning

Cost saving Agility Performance Portability Security

Without refactoring Faster application Low latency Build once, Deliver code faster
your app development processing run anywhere and securely at scale

57
 App modernization
without code changes
Capabilities
Lift and shift to containers

1. Use Azure Container Registry to store container images and Helm charts for your
modernized applications, replicated globally for low latency image serving
2. Integrate AKS with Azure Pipelines or other Kubernetes ecosystem tooling to enable
continuous integration/continuous delivery (CI/CD)
3. Enhance security with Azure Active Directory and RBAC to control access to AKS
resources

Learn more at
aka.ms/aksbook/liftandshift

58
 Virtual network

Active
Directory

Azure
Existing Container Azure
application Registry Pipelines Database
AKS for MySQL

59
 Microservices for faster
app development
Capabilities
1. Use Azure Dev Spaces to iteratively develop, test, and debug microservices
targeted for AKS clusters.
2. Azure DevOps has native integration with Helm and helps simplifying continuous
integration/continuous delivery (CI/CD)
3. Virtual node—a Virtual Kubelet implementation—allows fast scaling of services for
unpredictable traffic.
Microservices

4. Azure Monitor provides a single pane of glass for monitoring app telemetry,
cluster-to-container level health analytics.

Learn more at
aka.ms/aksbook/microservices

60
 Azure
AKS production cluster
Inner loop Source Container Azure
code control Registry Pods Monitor
Azure AKS dev
DevSpaces cluster
Test

Debug
Auto-build

Azure Pipeline/
DevOps Project

Container instances
Pods

CI/CD

61
 Data scientist in a box
Capabilities
1. Package ML model into a container and publish to Azure Container Registry
2. Azure Blob Storage hosts training data sets and trained model
3. Use Kubeflow to deploy training job to AKS, distributed training job to AKS
includes Parameter servers and Worker nodes
4. Serve production model using Kubeflow, promoting a consistent environment
across test, control and production
5. AKS supports GPU-enabled VM
Machine learning

6. Developer can build features querying the model running in AKS cluster

Learn more at
aka.ms/aksbook/ml

62
 App
developer

Query the model for AI

features in app
AKS

ML model in
containers

Azure
Data Container
scientist Registry
Serve the model
in production
Kubeflow
Parameter Worker GPU-enabled
server nods nodes VMS

Azure Blob
Storage

63
 Scalable Internet of
Things solutions
Capabilities
1. Azure IoT Edge encrypts data and send to Azure, which then
decrypts the data and sends to storage
2. Virtual node, an implementation of Virtual Kubelet, serves as the
translator between cloud and Edge
3. IoT Edge Provider in virtual node redirects containers to IoT Edge
and extend AKS cluster to target millions of edge devices
4. Consistent update, management, and monitoring as one unit in AKS
using single pod definition

Learn more at
IoT

aka.ms/aksbook/iot

64
 Decrypt
Decompress Compress
Send to Storage Encrypt
Send to Cloud

Azure Azure IoT Edge

Kubernetes cluster

Node Node Virtual node

Docker Docker Docker Docker Docker IoT Edge Docker
container container container container container Provider containers

65
 DevSecOps
Capabilities
1. Developers rapidly iterate, test, and debug different parts of an application together in
the same Kubernetes cluster
2. Code is merged into a GitHub repository, after which automated builds and tests are run
by Azure Pipelines
3. Container image is pushed to the Azure Container Registry
4. Kubernetes clusters are provisioned using tools like Terraform; Helm charts, installed by
Terraform, define the desired state of app resources and configurations
5. Operators enforce policies to govern deployments to the AKS cluster
6. Release pipeline automatically executes pre-defined deployment strategy with each code
change
7. Policy enforcement and auditing is added to CI/CD pipeline using Azure Policy
8. App telemetry, container health monitoring, and real-time log analytics are obtained
DevSecOps

using Azure Monitor

9. Insights used to address issues and fed into next sprint plans

Learn more at
aka.ms/aksbook/devsecops
66
Inner loop
Azure
Azure AKS dev Monitor
DevSpaces cluster
Test

Debug App Container Real-time

telemetry health log analytics

Azure AKS
Source Container production
code control Registry cluster Azure
Policy

Container
image
 v1

v2

Azure
Pipelines Helm chart Terraform

67
 Microsoft’s contributions
to the community
Microsoft brings knowledge from working with diverse customers to the Kubernetes
community, giving developers access to the latest Microsoft learnings and technologies,
and making Kubernetes itself enterprise-friendly and easier to use.

Helm Hub CNAB

Packaging &
distribution Helm Duffle

Scalability & Virtual Open Policy

governance Kubelet Agent

Kubernetes Draft VS Code Kubernetes

Extensions
developer tooling

68 Brigade
Best support for your needs

Learning path What is Kubernetes Hear from experts

aka.ms/LearnKubernetes aka.ms/aks/k8sLearning aka.ms/aks/videos

Case studies Azure Kubernetes Try for free

aka.ms/aks/casestudy aka.ms/aks/page aka.ms/aks/trial

69
© 2019 Microsoft Corporation. All rights reserved. This document is for informational purposes only.
Microsoft makes no warranties, express or implied, with respect to the information presented here.