The NetApp Guide to Kubernetes:

Persistent Volumes, Dynamic

Provisioning, Cloud Storage
1 The NetApp Guide to Kubernetes
 Introduction

Kubernetes is today’s most widely-used platform for container

and microservices orchestration and provides the scalability and
flexibility required for deploying enterprise applications and services.

Managing storage in a Kubernetes cluster with dynamic persistent

storage provisioning massively reduces the manual administration
required for allocating cloud storage to pods and containers.

This ebook covers everything you need to know about Kubernetes,

from the technology behind its use, how it works, and how NetApp’s
Trident provisioner can be used to dynamically provision persistent
volumes in the cloud via NetApp® Cloud Volumes ONTAP®.

2 The NetApp Guide to Kubernetes

Table of Contents
Introduction..................................................................................................................................................................2

Table of Content..........................................................................................................................................................3

PART ONE: What Is Kubernetes?.........................................................................................................................4

1.1 What Makes Kubernetes So Important..............................................................................................5

1.2 Provisioning Persistent Volumes for Kubernetes..........................................................................9

1.3 Static Provisioning Vs. Dynamic Provisioning................................................................................11

1.4 Managing Stateful Applications in Kubernetes............................................................................ 14

PART TWO: NetApp Solutions and Benefits for Kubernetes.................................................................. 16

2.1 Data Protection for Persistent Data Storage in Kubernetes Applications..........................17

2.2 Storage Efficiency for Improving Persistent Volume Storage Costs.................................. 19

2.3 NFS Files Shares and Kubernetes.................................................................................................... 22

2.4 Cloning Kubernetes Persistent Volumes with FlexClone....................................................... 23

PART THREE: How to Use Trident and Cloud Volumes ONTAP with Kubernetes........................ 24

3.1 Using Cloud Manager for Kubernetes Deployment with NetApp Trident........................ 24

3.2 Cloning Persistent Volumes............................................................................................................... 26

3.3 How to Set Up MySQL Kubernetes Deployments with Cloud Volumes ONTAP...........27

3.4 How to Use NFS File Services with Kubernetes........................................................................30

Conclusion.................................................................................................................................................................. 32

3 The NetApp Guide to Kubernetes

PART ONE

What Is Kubernetes?
TO UNDERSTAND KUBERNETES, YOU FIRST HAVE TO UNDERSTAND THE CORE
COMPONENT OF KUBERNETES ARCHITECTURE: CONTAINERS.
Containers are lightweight, independent units that software developers and DevOps engineers use to encapsulate applications.
On deployment, a container provides process and file system separation in much the same way as a virtual machine, but with
considerable improvements in server efficiency. That efficiency allows a much greater density of containers to be co-located on the
same host.

LIGHTWEIGHT, PROCESS AND SYSTEM BETTER SERVER

CO-LOCATING
INDEPENDENT SEPARATION EFFICIENCY

While container technology has been part of Unix-like operating systems since the turn of the century, it was only with the advent of
Docker that containers really came into the mainstream.

Docker has succeeded by bringing both standardization to container runtimes, for example, through the Open Container Initiative,
and by creating a complete container management system around the raw technology, simplifying the process of creating and
deploying containers for end users. Docker, however, can only be used to execute a container on a single host machine. That’s where
Kubernetes stepped in.

Docker Kubernetes
vs
Single host machine Cluster of machines

4 The NetApp Guide to Kubernetes

1.1 WHAT MAKES KUBERNETES SO IMPORTANT

KUBERNETES TAKES CONTAINER DEPLOYMENT TO A WHOLE NEW LEVEL BY

PROVIDING A ROBUST SOLUTION FOR MANAGING CONTAINERS, AND CONTAINERIZED
WORKLOADS ACROSS A CLUSTER OF MACHINES.

Kubernetes makes it possible to execute multiple instances And Kubernetes implementation is growing. A recent
of a container across a number of machines and achieve bi-annual survey of over 2000 IT professionals from North
fault tolerance and horizontal scale-out at the same time. America and Europe by the Cloud Native Computing
Kubernetes was created by Google after over a decade Foundation found that 75% of respondents were using
of using container orchestration internally to operate containers in production today, with the remaining
their public services. Google had been using containers number planning to use them in the future. Kubernetes
for a long time and developed their own proprietary usage has remained very strong with 83%, up from 77%,
solutions for data center container deployment and scaling. using the platform and 58% using it in production.
Kubernetes builds on those solutions as Google had been
using containers to enable the world-wide community of
software developers to grow the platform.

Multiple More Fault Open

Machines Scalable tolerance Source

5 The NetApp Guide to Kubernetes

AT THE HEART OF KUBERNETES IS CONTAINERIZATION.

The ability to manage applications changes in demand. The flexibility of this service has driven
independently of infrastructure holds Kubernetes introduction and adaptation
For deployed applications, Kubernetes
great value for cloud deployments. We across the cloud, with all major cloud
offers many benefits, such as service
can build out a cluster of machines in vendors offering a native Kubernetes
discovery, load balancing, rolling updates,
the cloud that provides the compute service, for example Amazon Elastic
and much more. Kubernetes acts as an
and storage resources for all of our Kubernetes Service (Amazon EKS) and
application server that is used to run all
applications, and then let Kubernetes Google Kubernetes Engine. Kubernetes
of the services, message queues, batch
ensure we get the best resource is also the foundation of other container
processes, database systems, caching
utilization. Kubernetes can also be orchestration platforms, such as Red Hat
services, etc. that make up an enterprise
configured to automatically scale the OpenShift.
application deployment.
cluster up and down in response to

6 The NetApp Guide to Kubernetes

IMPORTANT KUBERNETES TERMINOLOGY
In this section, we’ll give you an introduction to Kubernetes terminology that describes the main moving parts that make up
the service.

1 Cluster 4 Stateful Sets

1 Cluster
The collective set of compute nodes and For certain types of applications, such as
storage resources that form a Kubernetes database systems, it is crucial to maintain
2 Pod environment. Each cluster has at least the relationship between pods and data
one master, which is responsible for storage volumes. Stateful sets provide
overall management of the cluster, and an alternative model to Kubernetes
3 Deployment a number of nodes on which containers Deployments, and give each pod a unique
will be scheduled to execute. Each node and durable identity.
must have a container runtime installed,
4 Stateful Sets which is usually Docker, but may be an 5 Stateless Applications
alternative, such as rkt.
Stateless applications do not keep

5
Stateless 2 Pod a private record of client session
Applications information, which allows any running
In the Kubernetes architecture, a set of instance of the same application to
containers may be deployed and scaled process incoming requests. Applications
6 Services together. This is achieved by using deployed to Kubernetes or to containers
pods, which are the minimum unit of in general are typically stateless, and so
deployment in a Kubernetes cluster, and are easier to scale out horizontally across
7 Volume allow more than one container to share the cluster.
the same resources, such as IP address,
file systems, etc.
8 Persistent Volume 6 Services
3 Deployment When multiple, interchangeable pod

9
Persistent Volume A deployment is used to control replicas are active at the same time,
Claim Kubernetes pod creation, updates, clients need a simple way to find any
and scaling within the cluster, and is active pod they can send requests to.
normally used for stateless applications. Services solve this problem by acting as
10 Storage Class
A stateless application does not depend gateway to a set of pods, which may even
on maintaining its own client session exist in different Kubernetes cluster.
11
Dynamic Storage information, allowing any instance of
Provisioning the application to be equally capable of
serving client requests.
12 Provisioner

7 The NetApp Guide to Kubernetes

The following terms relate to storage provisioning in a Kubernetes cluster:

7 Volume 9 Persistent Volume Claim 11 Dynamic Storage

A storage area provisioned directly to a A claim for a persistent volume, which
Provisioning
pod. Kubernetes supports a wide variety acts as the link between a pod and a When Kubernetes administrators are
of volume types, including Amazon persistent volume. A persistent volume required to manually set up persistent
Elastic Block Storage (EBS). Volumes claim is used by Kubernetes to search for volumes ahead of time, this is known as
enable the containers within a pod to suitable persistent volumes that can fulfil static provisioning. By contrast, dynamic
share information and are destroyed the request. This search is based on the provisioning is used to automatically
when their parent pod is deleted. size of the required storage, the access allocate persistent volumes based on the
mode, a selector definition used to match persistent volume claims that are received
against labels on the persistent volume, by the cluster. The type of storage to
8 Persistent Volume
and, optionally, a storage class name. allocate is determined by the storage
A volume that exists independently class specified in the claim.
of any specific pod and with its own
lifetime. Persistent volumes can be
10 Storage Class
used to support stateful applications, Storage classes add a further level of
12 Provisioner
such as database services, enabling all abstraction to storage provisioning by When using dynamic storage
components of an enterprise solution allowing persistent volume claims to only provisioning, each storage class will
to be deployed and managed by specify the type of storage they require. stipulate the provisioner that should be
Kubernetes. Another major advantage For example, slow, fast, and shared may used to create new persistent volumes.
of using persistent volumes is that they all be valid user-defined storage classes. Internal provisioners are provided by
insulate the developers creating pods Storage class also encapsulates the Kubernetes for a wide range of storage
from the lower-level implementation details of the provisioner to be used, the options, such as Amazon EBS, however,
details of the storage they are accessing. type of volume to create, as well as other it’s also possible to specify external
This allows a persistent volume to provisioner-specific settings. This gives provisioners, such as NetApp Trident.
be provisioned using a local disk, for much greater control to the cluster over
example, on a developer workstation, how the storage is provisioned, which is
and using Cloud Volumes ONTAP to why storage classes are generally used
manage applications in production. with dynamic storage provisioning.

8 The NetApp Guide to Kubernetes

1.2 PROVISIONING PERSISTENT VOLUMES FOR KUBERNETES

PERSISTENT VOLUMES PROVIDE STORAGE RESOURCES TO PODS IN THE SAME WAY

THAT NODES PROVIDE COMPUTE.
Storage is required by containers and pods for a variety of purposes, from caching data to building stateful applications, such as
database services. Kubernetes uses persistent volumes to create a separation between the application developers that simply
require storage for their pods and the lower level details of how that storage is provisioned. This layer of abstraction allows the
storage implementation to be controlled independently of requesting applications. That means that, for example, storage can be
provisioned using local storage in a development cluster and with a more robust solution in production.

Each persistent volume is created by a provisioner that uses a plugin to interface with different types of backend storage, with
support for Amazon EBS. The lifetime of a persistent volume is determined by its reclaim policy, which controls the action the cluster
will take when a pod releases its ownership of the storage.

Volume: Storage provisioned directly to a pod that enable the containers within a pod to share information.
Destroyed when their parent pod is deleted.

Persistent Volume: A volume that exists independently of any specific pod and with its own lifetime.
Can be used to support stateful applications, such as database services, enabling all components of an
enterprise solution to be deployed and managed by Kubernetes.

9 The NetApp Guide to Kubernetes

KUBERNETES USES PERSISTENT VOLUME CLAIMS TO PROVIDE CLUSTERS
WITH STORAGE.
This is how it works: Each pod uses a persistent volume claim to bind to storage from the cluster of a particular size, access mode,
and volume mode. The claim can also use a selector to only match volumes with a specific set of labels. On receiving a claim, the
cluster will search for an existing persistent volume with which to fulfill the request.

Persistent Volume Claim:

A claim for a persistent volume, which acts as the link between a pod and a persistent volume. Used by
Kubernetes to search for suitable persistent volumes that can fulfill the request.

10 The NetApp Guide to Kubernetes

1.3 STATIC PROVISIONING VS. DYNAMIC PROVISIONING

THERE ARE TWO CHOICES FOR HOW PERSISTENT VOLUMES ARE PROVISIONED BY
KUBERNETES: STATIC AND DYNAMIC.
Cluster administrators can pre-allocate persistent volumes for the cluster, known as static provisioning, however, this requires
prior knowledge of storage requirements as a whole. Dynamic volume provisioning is an alternative model for managing storage
provisioning in Kubernetes, and is used to automatically deploy persistent volumes based on the claims received by the cluster.

Static Provisioning Dynamic Provisioning

• Set amount of storage before use • Created more organically

vs
• Controlled by administrators • Automatic creation process
• Pro: Storage spend is fixed • Pro: Able to provision volumes as needed
• Con: Can’t scale storage if needed • Con: No set limit on storage spend

11 The NetApp Guide to Kubernetes

USING DYNAMIC PROVISIONING IN A KUBERNETES CLUSTER REDUCES A LOT
OF THE ADMINISTRATIVE OVERHEAD INVOLVED IN MANUALLY CREATING
PERSISTENT VOLUMES.
Automatically allocating and deallocating persistent volumes in response to persistent volume claims can also help to reduce wasted
storage that is allocated but never used. Persistent volume claims that are fulfilled dynamically make use of a storage class attribute
to specify the type of storage required, which allows for data to be tiered by I/O performance profile.

Storage Class:
Storage classes add a further level of abstraction to storage provisioning by allowing persistent volume claims
to only specify the type of storage they require. Encapsulate the details of the provisioner to be used, the type
of volume to create, etc. Generally used with dynamic storage provisioning.

12 The NetApp Guide to Kubernetes

Kubernetes comes with a variety
of internal provisioners that
dynamically allocate storage using
Amazon EBS. However, that’s
where the support for storage ends:
storage management features, such
as creating and restoring backups,
ensuring high availability, and
provisioning test copies of persistent
volumes are all out of scope for
Kubernetes and must be catered
for by cluster administrators. In a
hybrid or multicloud environment,
this can lead to a multitude of
different approaches that increase
the complexity of managing storage
across the enterprise.

What can you do to avoid that?

Let’s explore what NetApp Trident
and Cloud Volumes ONTAP can do
for you.

13 The NetApp Guide to Kubernetes

1.4 MANAGING STATEFUL APPLICATIONS IN KUBERNETES

Stateful applications, such as database services and message brokers, record and manage the information generated within an
enterprise platform. Though Kubernetes has always supported stateless applications—which are horizontally scalable due to the
interchangeability of each pod—stateful applications require stronger guarantees for the storage they use.

Why the need for extra guarantees when it comes to stateful vs stateless? Whereas the storage used by stateless containerized
applications can simply be re-initialized when a pod is rescheduled to different node in the cluster, stateful applications are recording
business-critical information that must be preserved at all costs—that requires persistent storage with an independent lifetime.

Stateful Sets: For certain types of applications, such as database systems, it is crucial to maintain the
relationship between pods and data storage volumes. Stateful sets provide an alternative model to Kubernetes
deployments, and give each pod a unique and durable identity.

Stateless Applications: Stateless applications do not keep a private record of client session
information, which allows any running instance of the same application to process incoming requests.
Applications deployed to Kubernetes or to containers in general are typically stateless, and so are easier to
scale out horizontally across the cluster.

14 The NetApp Guide to Kubernetes

Kubernetes persistent volumes are Take a look at this example use case Trident provisions new persistent
used to create a layer of abstraction on GitHub, deploying MySQL with container storage that benefit from
between pods and their provisioned WordPress where persistent volumes all of NetApp’s data management
storage, allowing each to be managed were used in this way to deploy a capabilities. Using stateful sets simplifies
separately. Pods use a persistent database server. the deployment of all stateful applications
volume claim as a request for storage by automating much of the required
In both static and dynamic provisioning
that is matched to a persistent volume provisioning and administrative activity.
scenarios, stateful applications have
by the Kubernetes cluster. If a pod in a stateful set goes down,
a very real need for reliable storage
Kubernetes will automatically bind a
Using a reclaim policy of “retain”, a that is resilient against failure, and data
new instantiation of the pod to the
persistent volume will not be deleted protection features, such as backup and
dynamically provisioned persistent
or cleared down after the persistent restore. These facilities are not provided
volume it was previously using.
volume claim is released. Because of by Kubernetes, but are instead
this, in the event of a failure or the delegated to the provisioner used and
rescheduling of the pod onto a different the storage backend. Users have to
node, the relationship between pod find outside solutions, such as NetApp
and storage can be re-established. Trident and Cloud Volumes ONTAP.

15 The NetApp Guide to Kubernetes

PART TWO

NetApp Solutions and Benefits

for Kubernetes
NETAPP TRIDENT AND CLOUD VOLUMES ONTAP WORK TOGETHER TO DYNAMICALLY
PROVISION VOLUMES FOR KUBERNETES.
NetApp Trident is a dynamic storage provisioner for Kubernetes that fulfils persistent volume claims using storage managed by
Cloud Volumes ONTAP leveraging AWS, or on-premises ONTAP appliances as the storage back end for Kubernetes persistent
volumes. This enables Kubernetes clusters to take advantage of the power of NetApp storage management for persistent volume
provisioning in on-premises, hybrid cloud, or multicloud environments.

Cloud Volumes ONTAP is deployable on AWS and delivers:

1 High Availability 3 File Services 5 Data Tiering

Cloud Volumes ONTAP HA provides A flexible, scalable, performant, and Cloud Volumes ONTAP can easily tier
storage failover capability that works secure solution for deploying file shares cold data to Amazon Simple Storage
across Availability Zones, and guarantees and file services with multiprotocol Service (S3) save on storage costs.
high availability, RPO=0, and access for SMB / CIFS and NFS access
RTO < 60 seconds. for both Windows and Linux-based file
workloads.

2 Data Protection 4 Storage efficiencies 6 Storage cloning

NetApp Snapshots™ are used to create Volumes created using Cloud Volumes Existing storage volumes can be instantly
instant, space efficient, backups of a ONTAP are able to use built-in and space-efficiently cloned and made
storage volume of any size that can be technologies such as thin provisioning, available for read/write use using NetApp
instantly restored. NetApp SnapMirror® data deduplication, and data FlexClone® technology. NetApp Trident
technology uses snapshots as the basis compression making it possible to reduce integrates with cloning to allow persistent
for incrementally replicating a volume cloud storage footprint and operational volume claims to use a clone, rather than
to another instance of Cloud Volumes costs by 50-70%. always provisioning new storage.
ONTAP, which may reside in another
region as a disaster recovery (DR) or
backup secondary copy.

16 The NetApp Guide to Kubernetes

2.1 DATA PROTECTION FOR PERSISTENT DATA STORAGE IN
KUBERNETES APPLICATIONS
Enterprise workloads typically have a strong requirement for reliable data storage. Kubernetes persistent volumes can be
provisioned using a variety of solutions. However, ensuring that the data is easy to backup and restore, always available,
consistent, and durable in a Kubernetes workload disaster recovery situation or any other failure is the responsibility of end
users and administrators.

Stateful applications in production environments, such as database services, require access to redundant and highly available
data storage. Most stateless applications make use of stateful services in order to fulfill client requests, and therefore have
an indirect dependency on robust data storage services as well. Kubernetes provides a lot of flexibility when it comes to
persistent data storage provisioning, however, each solution uses its own specific mechanisms for protecting data, which may
also have limitations.

A SOLUTION FOR PERSISTENT DATA NEEDS

Kubernetes caters for persistent this is not sufficient protection for all ability to create and restore backups.
data storage through persistent organizations where end users are Examples of why you’d take regular
volumes, which have a life-cycle that is expected to build their own solutions to backups include ensuring that previous
independent of any particular container protect data across Availability Zones, versions of the data are available in
and that can be provisioned using a or across regions. When it comes to case of user error and providing your
diverse range of storage platforms. Kubernetes workload DR requirements, deployment in Kubernetes security
investing in this type of data protection against malicious access, such as
How do different storage solutions
is not only mandatory for business ransomware attacks. Due to the large
protect that data differently? For
continuity and regulatory requirements, it size of production datasets, an efficient
example, persistent volumes can be
also pays huge dividends in the long run. procedure is required not only to create
provisioned using Amazon EBS, which
backups, but also to restore them
provides some level of data redundancy Another important requirement for
consistently.
within an Availability Zone; however, protecting persistent data storage is the

17 The NetApp Guide to Kubernetes

YOU CAN GET BETTER DATA PROTECTION USING CLOUD VOLUMES ONTAP.
Cloud Volumes ONTAP makes use of a number of features to better protect your data in Kubernetes

ONTAP Snapshots Cloud Volumes ONTAP HA Data replication

can be used to instantly create space- is a high availability solution that of a Kubernetes workload DR site
efficient backups for Kubernetes storage automatically mirrors storage volumes to in another region can be easily
of any size. Using NetApp SnapCenter®, a secondary instance of Cloud Volumes accomplished using NetApp SnapMirror,
application-aware snapshots can be ONTAP, which may be placed in a which provides efficient, block-level data
created by temporarily freezing I/O write different Availability Zone or Availability replication between ONTAP storage
operations in order to guarantee that Set. These two instances can be deployed environments. After creating an initial
the data is in a consistent state before in either an active-active or active- baseline copy, all further synchronization
a snapshot is taken. Snapshots can also passive configuration, with the ability to occurs on an incremental basis, copying
be instantly restored back to the original failover and failback between the nodes over only the data that has changed.
source volume or to a new volume. Users without affecting client applications Cloud Volumes ONTAP can also be used
can even access the snapshot directly as that are actively using the storage. to failover storage to the destination
a read-only view of the source data at the Should a planned or unplanned failover volume, and efficiently re-synchronize in
point in time the snapshot was created. be required, Cloud Volumes ONTAP HA the reverse direction in order to failback.
provides an RPO=0, i.e. zero data loss, As well as mirroring storage for DR
and an RTO of less than 60 seconds. purposes, NetApp SnapMirror can also be
used to create a repository for long-term
backups and data archiving.

18 The NetApp Guide to Kubernetes

 2.2 STORAGE EFFICIENCY FOR IMPROVING
PERSISTENT VOLUME STORAGE COSTS
In both static and dynamic provisioning, it can be a challenge to ensure
storage efficiency, as application developers must prepare for storage usage
peaks and can often overestimate the amount of storage they need. In a cloud
environment, this leads to an unnecessary increase in cloud storage costs.

Many times, large allocations of storage made for an application are never
used, and in other cases the data stored within a persistent volume is never
compressed, leading to extra storage space being used up unnecessarily. These
types of issues can be resolved using the storage efficiency technologies that
are part of Cloud Volumes ONTAP, which the NetApp Trident provisioner makes
available to Kubernetes.

There may also be situations where a large part of the production dataset
is cold, or infrequently accessed, but which cannot be moved to more
cost-effective storage without introducing a lot of complexity to the ways
application services rely on that data. These application services require a
uniform view of their data, with fast access to hot data and on-demand access
to cold data. This can again be a difficult problem to solve, resulting in large
allocations of high performance and costly storage.

Another scenario where transparently-applied storage efficiency would

hold great value is where the data being stored contains a high degree of
redundancy, which could be compressed in order to reduce the overall amount
of storage required. Moving the onus for compression to the storage layer
simplifies application software development and also helps to reduce the cloud
storage footprint of legacy and third-party application services, where changes
to the systems cannot be made.

Though storage efficiency is always important in the cloud, it can be even more
so in Kubernetes environments due to the inherent scalability of containers.
Spinning up new pods to deal with an increase in workload, or to provide
greater redundancy and availability, also requires allocating new persistent
volumes. The storage overhead for these persistent volumes can be brought
under control through efficient data storage.

19 The NetApp Guide to Kubernetes

YOU CAN REDUCE KUBERNETES STORAGE COSTS WITH CLOUD VOLUMES ONTAP
STORAGE EFFICIENCIES.

Let’s take a look at the Cloud Volumes ONTAP efficiency technologies you can leverage via NetApp Trident to reduce the
storage required for persistent volumes in Kubernetes.

Thin Provisioning
Thin provisioning makes it possible to create persistent volumes that appear to pods as having the size they requested through a
persistent volume claim, but without needing to allocate all of that storage in advance. Cloud Volumes ONTAP will automatically
add storage capacity to the persistent volume as and when it is required, and also return back freed up storage to the common pool
when data is deleted. This ensures that storage space is only allocated when it’s actually needed, which reduce storage usage costs
and drives up storage space utilization. Thin provisioning also makes it much easier to plan for the future storage requirements of
the cluster.

Data Deduplication
Cloud Volumes ONTAP is able to transparently apply transformations, which help to reduce storage space usage, to the data it
stores. Cloud Volumes ONTAP employs data deduplication, which collapses identical copies of a block into a single block, with
reference pointers inserted into every place the block is used. This dramatically reduces storage space requirements, with some
companies reporting savings of up to 70%. A small amount of storage space is consumed in order to maintain the metadata
required to support the block mappings.

20 The NetApp Guide to Kubernetes

 Data Compression

Data in an ONTAP storage volume can be transparently compressed without

requiring any changes to client applications and services. This compression is
applied to groups of consecutive blocks, as opposed to entire files, which makes
reading and updating highly optimal. Data compression can also be used in
conjunction with data deduplication.

Data Compaction
After applying inline data deduplication and compression, multiple blocks that
are not completely filled are combined together, removing the unused of space
that would have otherwise been left in each block.

Storage Tiering
A major advantage of using Cloud Volumes ONTAP is the ability to automatically
balance data between a capacity storage tier for colder data and a performance
tier for fast access. Cloud Volumes ONTAP will use Amazon S3 for the capacity
tier for large amounts of cold or infrequently accessed data that must still be
available on-demand. When the data is required, it is quickly moved to the
performance tier, and will age back out to the capacity tier when it is no longer in
active use. This saves a significant amount on storage costs.

21 The NetApp Guide to Kubernetes

2.3 NFS FILE SHARES AND KUBERNETES

The NFS (Network File System) protocol Unless containers are deployed to
is widely used in sharing files for the same pod, sharing data between
enterprise environments, allowing many containers can also be difficult. Deploying
users to access the same files at the same containers in this way, i.e. to the same
time. This makes sharing data a lot easier. pod, should only be done when it makes
NFS can be used for a wide range of use sense for the application, such as when
cases, such as for creating data lakes for the containers work together directly to
building analytics solutions, database fulfill some function. Putting containers
services, data archives, etc. If you’re into the same pod when they simply
using Kubernetes, NFS can be used with need to share data can lead to scalability
Kubernetes pods to manage persistent problems as a pod will only be scheduled
storage requirements and share data and to a single node.
files between containers and other pods.
NFS solves these problems by allowing
There are several reasons why NFS shared many hosts to mount the same file
storage is preferable for Kubernetes system at the same time, and for all hosts
deployments. For starters, there’s an to access files concurrently. With NFS,
advantage to NFS over iSCSI because users don’t need to format the storage
managing a large number of individual volume using an Operating System file
iSCSI storage allocations can add to system, such as ext4; the storage can
administrative overhead. All those block- simply be mounted and used straight
level persistent volume allocations can away. This makes it much easier to
make storage management more difficult, attach storage to pods and reduce the
especially when Kubernetes persistent administrative overhead of working
volumes are statically allocated. In these with persistent storage. NFS storage
cases, storage utilization may be lower, volumes in Kubernetes can also be
as a persistent volume is used only for a easily expanded without any client-side
single pod and that pod may not make changes using Trident, which we’ll discuss
use of all of its storage. Also, sharing data in more detail in the next section.
between pods can be difficult when using
block-level persistent volumes.

22 The NetApp Guide to Kubernetes

 2.4 CLONING KUBERNETES PERSISTENT
VOLUMES WITH FLEXCLONE

Creating test copies of production data is very often a requirement for DevOps
engineers implementing CI/CD pipelines or setting up staging clusters for pre-
production testing. Automated software test suites will often mutate the data
that they access, which means that a fresh copy of the data is required in order
to repeat the testing, with test cycles sometimes executed hundreds of times.
Ensuring faster TTM (Time To Market) and the delivery of high quality software
relies upon developers working in parallel and executing tests as often
as required.

Restoring data backups is not a scalable solution for creating test data sets:
with the size of the source data involved it just takes too much time to perform
a restore. This leads to reduced developer productivity when multiple, up-to-
date copies of persistent volumes are required, which also have to be refreshed
frequently. Using this approach also means that the amount of storage used to
develop and support an application will be many times greater than the storage
requirements of the production environment, causing a significant increase in
cloud storage costs. That’s not something any developer wants to get blamed for.

The time taken to perform a restore and the storage space it uses can both be
very wasteful, as usually most of the restored data remains unaffected by the
software testing and must be recreated simply to bring the data to a consistent
point or to get an up-to-date copy. In contrast, not only is NetApp FlexClone able
to instantly clone a source volume of any size, it also does so with zero storage
penalty for any storage size required. NetApp Trident helps you take advantage
of this for persistent volumes in a Kubernetes cluster.

In the third part of this ebook we’ll walk you through the steps of cloning your
persistent volumes in order to facilitate test environments and reduce your
storage footprints.

23 The NetApp Guide to Kubernetes

PART THREE

How to Use NetApp Trident and Cloud

Volumes ONTAP with Kubernetes
In this section we’ll see a few coding examples for how users can make Trident and Cloud Volumes ONTAP work with their
deployments on Kubernetes.

USING CLOUD CLONING MYSQL KUBERNETES

DEPLOYMENTS NFS WITH
MANAGER PERSISTENT
WITH CLOUD KUBERNETES
WITH TRIDENT VOLUMES
VOLUMES ONTAP

3.1 USING CLOUD MANAGER FOR KUBERNETES DEPLOYMENT WITH

NETAPP TRIDENT

Cloud Manager is the central platform from which you can deploy and manage instances of Cloud Volumes ONTAP for both large
and small environments. The graphical, web-based user interface makes it easy to setup Cloud Volumes ONTAP storage services
and organize them across multiple tenants for better overall manageability. On-premises systems and AWS deployments can all be
controlled from a single dashboard, and NetApp SnapMirror replication relationships created between them with a simple drag-and-
drop.

Cloud Manager allows us to deploy NetApp Trident to a Kubernetes cluster and then relate the cluster to Cloud Volumes
ONTAP instances.

PREREQUISITES
Before you can start you should make sure that you have network access between Cloud Manager, the instances of Cloud Volumes
ONTAP to be used, and the actual Kubernetes cluster. Cloud Manager will also require internet access in order to download the
latest deployment packages for NetApp Trident.

24 The NetApp Guide to Kubernetes

DEPLOYING TRIDENT

To deploy NetApp Trident, Cloud Manager

must be provided with the kubeconfig
file for the target Kubernetes cluster,
which provides all the necessary details
for making a connection and performing
cluster operations. After uploading the
file, Cloud Manager takes care of the rest
of the deployment.

Setting up access to a Kubernetes cluster from Cloud Manager.

ASSOCIATING THE KUBERNETES CLUSTER WITH CLOUD VOLUMES ONTAP

Now that we have NetApp Trident

installed, our Kubernetes cluster is able
to interface with Cloud Volumes ONTAP,
however, we still need to associate the
cluster with the specific instance of Cloud
Volumes ONTAP that it should use. This
is achieved by clicking the Kubernetes
icon on the details page for that instance,
and confirming that we wish to proceed.
Cloud Manager will then complete the
configuration process and allow us to
start provisioning persistent storage
for Kubernetes.
Associating a deployment of Cloud Volumes ONTAP with the Kubernetes cluster

When integrating with NFS Kubernetes deployments, Trident is also able to create the storage class and the NFS mount points.
Other benefits include cloning Kubernetes persistent volumes with FlexClone, using storage efficiencies to cut persistent volume
footprint and costs, storage tiering to Amazon S3 and data protection.

25 The NetApp Guide to Kubernetes

 3.2 CLONING PERSISTENT VOLUMES
Data cloning can help speed up TTM and reduce storage footprint and costs.
Instead of always provisioning new persistent volumes in response to persistent
volume claims, using Trident the user can clone existing volumes. To instruct
Trident to create a clone, simply give it the name of the persistent volume claim
that was used to provision the original persistent volume, which must also have
been allocated through Trident. This information is provided through a metadata
annotation on the persistent volume claim that is requesting the volume clone.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: basicclone
annotations:
trident.netapp.io/cloneFromPVC: basic
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: basic

In the background, Trident will find the ONTAP volume that was used to provision
storage for the original persistent volume claim and then use NetApp FlexClone
to perform a data cloning operation. Clones are created by using a snapshot
on the parent volume. Both the clone and parent volumes are free to accept
changes to their data, which are written to new blocks using a redirect-on-write
mechanism that protects blocks locked by a snapshot from being overwritten.
Using this technique, the clone requires negligible storage space when it is first
created, and only grows with the changes that are made to it.

By using the default reclaim policy of delete, the clone will automatically be
deleted when the pod using it is destroyed. This can be used to conveniently
clean up any allocated storage at the end of a round of testing, for example.
Trident also supports an optional annotation to split the clone from the parent
volume it is associated with. This turns the clone into an independent copy of
the data, which means the space efficiencies discussed previously will be lost,
however, this can be more suitable in scenarios when the clone will undergo a lot
of changes, or when the state of a volume is needed to provide seed data for a
new requirement.

26 The NetApp Guide to Kubernetes

3.3 HOW TO SET UP MYSQL KUBERNETES
DEPLOYMENTS WITH CLOUD VOLUMES ONTAP
MySQL is a popular flavor of SQL database. The following provides an example of
deploying a single Kubernetes MySQL instance using NetApp Trident.

First, we will create a persistent volume claim for the storage we need. As Trident
uses dynamic provisioning, we will specify a storage class, which must have been
setup prior to executing this manifest. Each storage class defines the provisioner
to be used, along with any other provisioner-specific settings that will determine
how the storage is provisioned.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: mysql-pvc
annotations:
trident.netapp.io/reclaimPolicy: “Retain”
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
storageClassName: high-performance

When this claim is used by a pod, Trident will automatically create a 100 GiB
high-performance storage volume in Cloud Volumes ONTAP to fulfil the request.
This storage class may have been implemented to make use of Amazon EBS
Provisioned IOPS disks, for example. A reclaim policy of “retain” has also been
specified, which will prevent the persistent volume from being deleted if the pod
releases the claim.

27 The NetApp Guide to Kubernetes

 We can now bind a pod to this storage by referencing it in our MySQL
deployment manifest. As shown below, we simply need to reference
our previously defined persistent volume claim and specify where
the bound persistent volume should be mounted within our MySQL
Kubernetes container.

apiVersion: apps/v1
kind: Deployment
metadata:
name: mysql-server
labels:
app: mysql
spec:
selector:
matchLabels:
app: mysql
tier: database
strategy:
type: Recreate
template:
metadata:
labels:
app: mysql
tier: database
spec:
containers:
- image: mysql:5.6
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-pass
key: password
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-pv
mountPath: /var/lib/mysql
volumes:
- name: mysql-pv
persistentVolumeClaim:
claimName: mysql-pvc

28 The NetApp Guide to Kubernetes

We can additionally set up test instances of our Kubernetes MySQL database using Trident’s integration with NetApp FlexClone. The
following persistent volume claim manifest demonstrates how this can be done.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: mysql-pvc-clone
annotations:
trident.netapp.io/cloneFromPVC: mysql-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
storageClassName: high-performance

We simply annotate our claim with the name of the persistent volume claim to clone from, and Trident takes care of the rest by
automatically finding the source storage volume in Cloud Volumes ONTAP and performing a NetApp FlexClone operation. The
default reclaim policy of delete will ensure that the clone volume is automatically deleted when it is no longer required.

29 The NetApp Guide to Kubernetes

3.4 HOW TO USE NFS FILE SERVICES WITH KUBERNETES
As a Kubernetes NFS provisioner, the benefits of using Trident include mounting persistent volumes as Read/Write Many,
dynamically resizing NFS persistent volumes, and creating separate storage classes for different mount parameters and
other requirements.

MOUNTING PERSISTENT VOLUMES AS READ/WRITE MANY

Persistent Volumes using NFS can be mounted as Read/Write Many, which allows multiple pods to mount the same volumes. Using
native Kubernetes constructs means that storage can be allocated without the need for custom procedures or processes. Using the
Read/Write Many access mode allows many persistent volumes claims to access the same persistent volume.

With block-level Kubernetes persistent volumes, Read/Write Once must be used, which means that there is a 1-to-1 relationship
between persistent volume claims and persistent volumes.

Here’s a persistent volume claim (PVC) for a Kubernetes NFS volume example:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: nfs_share
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 100Gi
storageClassName: silver

30 The NetApp Guide to Kubernetes

DYNAMICALLY RESIZING NFS PERSISTENT VOLUMES

New to provisioning in Kubernetes is the when it comes to NFS. With Trident, NFS when the underlying storage is expanded.
support for editing persistent volumes persistent volumes can be dynamically Kubernetes will take care of this, but it
claims to request more storage space. resized and all the back-end changes will also require that the pod using the
With this new capability, persistent required taken care of. This makes Trident storage to be restarted. That will cause
volumes can be expanded without a major value add. downtime. With NFS, no file system
needing to re-create the pod that uses expansion is required, and the pod can
By extending the support for resizing
the storage. In order for that to happen, continue to work without interruption.
persistent volumes to NFS, Trident is a
the underlying storage provisioner must
major value add. You can read more about resizing NFS
support resize. The commonly used
volumes with Trident on GitHub.
provisioners, such as those for Amazon Block-level storage, such as Amazon EBS,
EBS storage support this—except not needs to have the file system extended

31 The NetApp Guide to Kubernetes

CREATING SEPARATE STORAGE CLASSES

Separate storage classes can be created for different mount parameters and other requirements, for example using separate storage
classes for NFSv3 and NFSv4.

Storage classes allow users to control the type and configuration of the storage they are using when provisioning in Kubernetes.
That extends to the Kubernetes dynamic provisioning NFS storage Trident provides.

Take this Kubernetes NFS storage class example. Here is the storage class manifest:

The allowVolumeExpansion attribute controls whether the

apiVersion: storage.k8s.io/v1 resizing of persistent volumes created using this storage class
kind: StorageClass
is allowed or not.
metadata:
name: ontapnas The storage class can be used to control the storage pool, or
provisioner: netapp.io/trident NetApp aggregate used to provision persistent volumes, e.g.
parameters:
a storage class named gold may be used to configure high
backendType: ontap-nas
allowVolumeExpansion: true performance back-end storage, where as one called bronze
may be configured for capacity storage.

32 The NetApp Guide to Kubernetes

Conclusion
Kubernetes is today’s most widely-used platform for container and microservices orchestration
and provides the scalability and flexibility required for deploying enterprise applications and
services. Managing storage in a Kubernetes cluster with dynamic storage provisioning massively
reduces the manual administration required for allocating cloud storage to pods and containers.

Using NetApp Trident, Kubernetes storage requests are dynamically fulfilled by Cloud Volumes
ONTAP, which similarly does for storage what Kubernetes does for containers.

Now that you’ve read this Kubernetes guide, find out for yourself how Cloud Volumes ONTAP
transforms container storage management with a free 30-day trial on AWS.

33 The NetApp Guide to Kubernetes

Refer to the Interoperability Matrix Tool (IMT) on the NetApp Support site to validate that the exact product and feature versions described in this
document are supported for your specific environment. The NetApp IMT defines the product components and versions that can be used to construct
configurations that are supported by NetApp. Specific results depend on each customer’s installation in accordance with published specifications.

Copyright Information
Copyright © 1994–2019 NetApp, Inc. All rights reserved. Printed in the U.S. No part of this document covered by copyright may be reproduced in any
form or by any means—graphic, electronic, or mechanical, including photocopying, recording, taping, or storage in an electronic retrieval system—
without prior written permission of the copyright owner.

Software derived from copyrighted NetApp material is subject to the following license and disclaimer:

THIS SOFTWARE IS PROVIDED BY NETAPP “AS IS” AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, WHICH ARE HEREBY DISCLAIMED. IN NO
EVENT SHALL NETAPP BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

NetApp reserves the right to change any products described herein at any time, and without notice. NetApp assumes no responsibility or liability
arising from the use of products described herein, except as expressly agreed to in writing by NetApp. The use or purchase of this product does not
convey a license under any patent rights, trademark rights, or any other intellectual property rights of NetApp.

The product described in this manual may be protected by one or more U.S. patents, foreign patents, or pending applications.

RESTRICTED RIGHTS LEGEND: Use, duplication, or disclosure by the government is subject to restrictions as set forth in subparagraph (c)(1)(ii) of
the Rights in Technical Data and Computer Software clause at DFARS 252.277-7103 (October 1988) and FAR 52-227-19 (June 1987).

Trademark Information
NETAPP, the NETAPP logo, and the marks listed at http://www.netapp.com/TM are trademarks of NetApp, Inc. Other company and product names
may be trademarks of their respective owners.

NA-287-0420

34 The NetApp Guide to Kubernetes