https://www.linkedin.

com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

Deploy a Kubernetes cluster using

Terraform and Ansible on AWS: Use EC2 or
EKS

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

While I was in the initial stages of learning Kubernetes, I always

struggled to get a quick Kubernetes cluster to work on my practice
labs. Though there are many online alternatives available which
can be used in such scenarios but somehow I always felt if I could
easily spin up a lab environment of my own, it will be good for my
learning. Thats when I came up with this process to spin up a
basic Kubernetes cluster easily and quickly. The process described
here can be a good learning to use Terraform and Ansible in such
scenarios too. With modifications to the steps, this can also be
used to launch actual clusters in projects.

For this post, I will go through the process of spinning up a basic

Kubernetes cluster using Terraform on AWS. I will cover two ways
to launch the cluster:

• Using Terraform to launch EC2 instances on AWS and use

Ansible to bootstrap and start a Kubernetes cluster
• Using Terraform launch an AWS EKS cluster
The GitHub repo for this post can be found Here
Pre Requisites
Before I start the walkthrough, there are some some pre-
requisites which need to be met if you want to follow along or
want to install your own cluster:

• Some Kubernetes, Terraform and Ansible knowledge. This

will help understand the workings of the process
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
• Jenkins server to run Jenkins jobs
• An AWS account. The EKS cluster may incur some charges so
make sure to monitor that
• Terraform installed
• Jenkins Knowledge
Apart from this I will explain all of the steps so it can be followed
easily. I will cover each of the steps in detail so if you have some
basic conceptual understanding of the above mentioned topics,
you should be able to follow along.

Functional Flow
Let me first explain whats the process I will be following to deploy
the cluster. I will go through two methods of deploying the cluster:

• Using EC2: In this method the cluster will be deployed using

just EC2 instances on AWS. Both control plane and the nodes
will be EC2 instances which you can directly access and
control. The controlplane and nodes EC2 instances will be
configured accordingly.
• Using AWS EKS: In this method the cluster will be deployed
on the managed platform called EKS, which is provided by
EKS. Since EKS is a managed cluster by AWS, we will be doing
a declarative config here and deploy the cluster. To have a
serverless approach, for the nodes I will be using Fargate
instances. To know more about what is Fargate, click Here.
Lets go through each of the methods and see how this will work.

Using EC2 Instances

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
First method we will go through is deploying the cluster using EC2
instances. In this we will deploy both the controlplane and some
nodes on various EC2 instances. Below flow shows the overall
process.

• Deploy the Infrastructure: This is the first step where the

infrastructure to support the cluster will be deployed. The
infrastructure will include different parts of the whole
architecture which will be supporting the cluster running on
the EC2 instances. Some of the items which will be deployed
as part of the infrastructure are:
• Networking components like VPC, Subnets etc.

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
• S3 buckets to stora various temporary files needed
during the deployment flow. Also to store the
infrastructure state during the deployment
• Deploy the Controlplane Instance: In this stage the EC2
instance is deployed on the above infrastructure. In this
process I am deploying one EC2 instance as the controlplane
of the cluster. Normally Production grade Kubernetes cluster
can have multiple controlplane nodes and add a scalability to
it. But for this example, I am deploying one EC2 instance as
the controlplane.
• Bootstrap the controlplane: The EC2 instance deployed
above is needed to be prepared so Kubernetes cluster can
start on it. In this stage multiple packages will be installed on
the EC2 instance and the Kubernetes cluster will be started
on the instance. At this point the cluster only comprises of
the controlplane node.
• Launch the EC2 instances for Nodes: To make the
Kubernetes cluster scalable, we will need more nodes added
to it. In this step I am deploying some more EC2 instances
which will become the child nodes in the Kubernetes cluster
with the controlplane EC2 instance as the controller.
• Prepare the Nodes and join cluster: For the nodes to be
part of the cluster, they need to be prepared. In this step the
needed packages are installed on the Nodes and the
Kubernetes cluster join command is executed on each of the
node EC2 instance, so they join the cluster.
That completes the high level flow of deploying the cluster on EC2
instances.
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
Using AWS EKS
In this method, the cluster will be deployed on the AWS managed
platform called EKS (Elastic Kubernetes Service). This platform
takes away many of the steps to prepare the cluster and takes a
more declarative approach. Below flow shows what I will be
following in this process:

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
• Deploy the Infrastructure: In this step, the supporting
infrastructure is deployed. Even though the cluster is AWS
managed, the supporting infrastructure has to be deployed
and can be controlled by us. As part of this step, these are the
items which are deployed:
• Networking components like VPC, subnets etc.

• S3 buckets to store any temporaray files needed during

the deployment
• Any IAM roles needed by the components
• Deploy the EKS Cluster: Once we have the infrastructure
ready, we can go ahead and deploy the EKS cluster. Since the
cluster is managed by AWS, we dont need to do any
installations manually. We can pass the configuration needed
for the cluster (like VPC, subnets etc.) and AWS handles
spinning up the cluster
• Spin up Nodes for the cluster: With the EKS cluster, AWS
spins up the controlplane for the cluster. For this example I
am using Fargate instances as Nodes. So those nodes spin up
automatically as workloads gets deployed on the cluster.
Based on the workloads more Fargate nodes spin up to
handle the scalability. Apart from Fargate instances, even
normal EC2 instances can be used as nodes and attach to the
cluster. Even if using EC2, no need of manual installations.
AWS EKS will handle spinning up the EC2 nodes and manage
as part of the cluster.
Now that we have some basic understanding of what I will be
deploying, lets dive into some technical details of the parts.

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
Tech Details
Let me explain how each of the cluster will be composed using
different AWS components. For both of the methods, I will be
using Jenkins to automate the whole deployment. Here I am
describing each of the cluster.

Cluster on EC2
This cluster comprises of EC2 instances which are deployed
separately. Below image will show the overall cluster which will
be built in this process:

• Controlplane EC2 Instance: One EC2 instance is deployed

which will be serving as the controlplane node for this
cluster. Since its the controlplane node, the size of the EC2
should be larger like t3. For the EC2 instance to act as the
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
controlplane, few packages are installed on the instance.
Ansible is used to bootstrap the instance and install all the
needed packages. At high level these are the steps which will
be followed on the controlplane instance after it is spun up:
• Install Docker

• Install Kubernetes
• Start Kubernetes using Kubeadm and start the service
on this node
• Generate the Kubeadm join command for other nodes to
join the cluster
• Auto Scaling Group for the nodes: Now we have the
controlplane for the cluster but we will need other nodes
joined to the cluster to have a scalable Kubernetes cluster. To
launch the nodes automatically based on load on the cluster,
an auto scaling group is being deployed here. The ASG scales
in and out with the number of nodes which become part of
the cluster. Based on the load on the cluster, the node count
varies and is controlled by the ASG. Once the nodes launch,
Ansible is used to bootstrap the nodes and make them join
the cluster. These are the steps which are performed using
Ansible on the nodes:
• Install Docker

• Install Kubernetes
• Run the Kubeadm join command with the join token, so
the node becomes part of the cluster
• Networking: All of the instances which are spun up by this
process are deployed inside a VPC which also gets created

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
part of the process. These are the components which gets
deployed as part of the networking for the whole cluster:
• VPC for the whole cluster

• Subnets for diferent instances across multiple AZs

• Security groups to control the traffic to these instances
• Internet Gateway for outside access to these instances
• Use of Terraform: All of the infrastructure which is
mentioned here, is being deployed using Terraform. Different
Terraform modules have been created to handle each of the
components of the cluster. I will be going through the
Terraform scripts in detail below.
• Use of Ansible: Ansible is used to handle of the package
installations and steps needed on all of the instances. Since
the installations and the cluster commands are automated,
they are all ran by different Ansible playbooks. These
playbooks are executed on the instances via SSH from the
Jenkins pipeline. I will go through these playbooks in detail in
below sections.
That should explain about the whole cluster architecture and it is
deployed across multiple EC2 instances.

Cluster on EKS
In the next method, I am describing the cluster architecture which
is deployed using AWS EKS. In this there are no direct
involvement with launching EC2 instances and bootstrapping
them. Below image describes the cluster architecture:

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

• AWS EKS Cluster: This is the central component of the

cluster. An EKS cluster is deployed using a declarative
configuration. Using Terraform the EKS cluster is deployed
and AWS manages spinning up the cluster and operational. I
will cover in detail how this is deployed.
• Networking: The networking part is the same as the above
method with EC2 instances. Even for this EKS cluster we will
need the networking components deployed separately, so
the cluster can run in it. Same networking components are
also deployed in this process
• Fargate Instances: For the nodes in the cluster, I am using
Fargate instances. These are AWS managed instances which
work as nodes in the cluster. To work with the autoscaling,
these instances spin up automatically as workloads get
deployed to the cluster. This is is also a serverless node
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
added to the cluster. Will cover in detail later how the
workloads get deployed to the Fargate instances.
• Use of Terraform: All of the infrastructure described above
is deployed using Terraform. The declarative configuration
for the cluster is specified in the Terraform script which
deploys the whole cluster.
That explanation should give a good idea of both of the processes
and what we will be setting up in this post. Lets walk through
setting up each of the clusters.

Setup Walkthrough
In this setup I will walk you through the setting up of the whole
cluster via both of the processes. You can use any of these to spin
up your own cluster. At high level this is the general flow which I
will be setting up on Jenkins to launch each of the cluster.

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

Above is a very high level depiction of the process. Let me go

through the different parts of the process in detail.

Folder Structure
Before diving in the setup process, let me explain the folder
structure of the scripts on my repo. If you are using my repo for
spinning up your cluster, this should help understand the setup. I
have divided both of the approaches in two separate folders with
their own folder structure in each.
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
• Using EC2 Instances: Below is the folder structure for this

ansible_infra:This folder holds everything related to ansible used

to bootstrap the instances in the cluster. It holds all the playbooks
and the role to install and start Kubernetes on the controlplane
and other nodes in the cluster. It also holds the Terraform script
to launch the infrastructure needed to hold the the ansible
playbooks and inventory files for the nodes to pull from. more on
that later.

instances:This folder holds the Terraform scripts to launch the

EC2 instances for the controlplane node.

networking:This folder holds the Terraform scripts to launch all

the networking components of the cluster like VPC, subnets etc.

node_asg:This folder contains all the Terraform scripts to launch

the Auto scaling group that will launch all of the other nodes in the
cluster.

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
Jenkinsfile:Finally the Jenkinsfile for the pipeline to launch the
whole cluster step by step

• Using EKS Cluster: Below is the folder structure for this

process:

cluster:This folder contains all the Terraform scripts to launch the

EKS cluster. This is defined as a Terraform module which gets
deployed via Jenkins

networking:This folder contains all of the Terraform scripts to

launch the networking components of the cluster like VPC,
subnets et. The EKS cluster gets deployed on this network

sample_app:This folder contains a Yaml file to deploy a sample

Nginx pod. This pod i deployed to demonstrate how Fargate
instances gets deployed once workloads are created on the EKS
cluster.

Jenkinsfile:Finally the Jenkinsfile to define the pipeline which will

deploy this whole cluster and its components.

That is the folder structure I have created in the repo I shared for
this post. But in the steps below, what I have done is separate out
those folders in separate branches in the same repo. Each of those
branches trigger different Jenkins stages. I will go through the
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
stages below. But just to understand the branch structure, this is
what I will be following in this post. You can take my repo as is or
convert to a branch structure as this to easily trigger separate
pipelines.

Jenkins Setup
Now that we have seen the structure of how the codebase looks,
now its the time to prepare Jenkins for running the deployment
steps. Let go through the steps one by one. But before starting to
setup Jenkins, since we will be deploying to AWS, we will need an
IAM user which the pipeline will use to connect to AWS. So first go
ahead and create an IAM user with permissions to work with EC2
and EKS. Not e down the AWS keys for the user as we will be
needing those next.

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

Once we have the IAM user lets move to setting up jenkins.

• Install Jenkins:
Of course the first step is to install Jenkins on a server or a
system as suitable. I wont go through the steps to install
Jenkins but you can follow the steps Here.
• Setup Credentials:
For Jenkins to connect to AWS, it needs the AWS credentials. In
this step we will setup the credentials in Jenkins. Login to
Jenkins and Navigate to the Manage credentials page:

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

On the credentials page, add a new Credential of kind Secret

Text. Put the AWS key value in the Secret content field and
provide a name for the credential on the ID field. This is the
name which will be referred in the Jenkinsfile to access the
credential.

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

Add two such credentials for each of the AWS key from the IAM
user we created earlier. There should be two credentials now.
If you added on a different name, update accordingly on the
Jenkinsfile too. This is what I have used for my Jenkins pipeline:

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

These two credentias will be used by the pipeline to

communicate with AWS.
• Setup Pipeline:
Now we will setup the actual pipeline which will be running
and performing the deployment. As I stated above, in this
example I have kept the EKS related and EC2 cluster related
files in two separate branches. So I will be creating a multi
branch pipeline to scan both branches and run accordingly.
Login to Jenkins and create a new Multibranch pipeline from
the create option

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

On the options page, select the GIT repo source which will be
the source for the scripts and the two branches. To be able to
connect the repo, the credentials will also have to be specified
on the same option. If its a public repo no credentials needed. I
would suggest clone my repo and use your own repo for the
pipeline.

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

I am using Jenkinsfile as the name of the file but if you are

using some other name specify name in the options. This will
identify which file to read the pipeline content from, in the
codebase when the code is checked out by Jenkins. Other
settings can be kept default or tweaked as needed. I have kept
default values for other settings

Save the pipeline. Once you save, Jenkins will start scanning for
branches in the repo. At this point it will fail since you wont
have any content in the pipeline. If you have created the
branches already, you should have the multiple branches

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
showing up as separate runnable pipeline on Jenkins

That should complete the initial Jenkins setup and you are ready
to run the pipelines for deployment. Next lets move on to
understanding the two pipelines and run them.

Deploy EC2 Cluster

Lets first go through deploying the Kubernetes cluster using EC2
instances. Below pipeline shows the whole deployment steps
which are carried out by Jenkins:

Lets understand each of the stages.

• Environment Variables: First step is to set the needed

environment variables which are used by different steps in
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
the pipeline. Here are the environment variables which are
being set:

• Create S3 Buckets for State: In this stage, the pipeline

creates the S3 buckets which are needed for storing the
Terraform state. The bucket name is mentioned on the env
variables and same name gets specified in the Terraform
scripts so the state can be managed in this bucket.

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

• Deploy Infra for Ansible: In this stage, S3 buckets are

created which will be used by Ansible to store the updated
Inventory files and read the Inventory files to run commands.
Terraform is used to create the buckets. The bucket name is
passed via the variable file.

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
• Deploy networking modules: In this stage the networking
components are deployed. All of the network components
are defined in a Terraform script. The Terraform module gets
deployed in the pipeline stage. These are the items which get
deployed in this stage:
• VPC

• Subnets
• Security Groups

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

• Deploy EC2 Instances: In this stage the EC2 instance for

controlplane get created. The EC2 instance is defined in a
Terraform module which gets deployed in this stage. But its
not just deploying the instance, in this stage, the instance
details also gets updated to an inventory file. This is what
happens in this stage.
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

An Ansible playbook is created to perform the Inventory file

update and upload to S3. The playbook can be found in the repo I
shared.

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

• Bootstrap Controlplane Instance: In this stage the

controlplane EC2 which was launched earlier, gets
bootstrapped where all the needed packages and Kubernetes
gets installed on the instance. An ansible role has been
created to perform the bootstrap steps. These are the steps
which happen in this stage:
• The inventory file which was uploaded earlier, is pulled
from the S3 bucket in the local folder within pipeline
• The ansible role is executed which installs all the
necessary packages and starts the Kubernetes service
on the controlplane node
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

After the Kubernetes service starts, there is also a test step which
runs in this stage to ensure the Kubernetes starts properly on the
controlplane EC2. The kubectl test step is also performed using
Ansible. During the test step, same Inventory file is also
downloaded from the S3 bucket, and used by Ansible to connect to
the EC2.

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

• Launch Auto Scaling Group for Nodes: Now that we have

the controlplane up and running, in this stage we will launch
the instances which will join as nodes to the cluster. There
are few things happening in this stage apart from launching
the ASG. This should explain the steps:

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

An Ansible playbook has been created to get the launched instance

details and update in an inventory file. That inventory file gets
uploaded to a S3 bucket for further steps.

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

• Bootstrap Node Instances: In this stage, the node instances

which were launched on last step, get bootstrapped. All the
needed packages are installed on these nodes and the
Kubeadm join commands are executed on these nodes so
these join the cluster. This is the flow which happens in this
step:

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

To bootstrap the nodes, another Ansible playbook has been

created which runs the commands. Once the nodes successfully
join the cluster, finally a test kubectl command is executed using
an Ansible playbook. This kubectl will test that the nodes have
successfully joined the cluster. You can check all the nodes and the
controlplane running on the console

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

That completes the deployment of the cluster using EC2 instances.

Once these steps complete, you should have a working cluster
using the EC2 instances. Once all the changes are done push the
changes to the Git repo which is the source for the pipeline.
git add .
git commit -m "updated pipeline files"
git push -u origin main
To execute the steps, run the pipeline which was created. Once the
pipeline finishes, the cluster will be up and running.

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

Note: If you are running through spinning up a cluster just for

learning, make sure to destroy the resources which were created
or you wil incur charges on AWS. Just ru the destroy phase of the
pipeline in my repo by setting the environment variable. It will
destroy all of the resources which were created by the pipeline.

Deploy EKS Cluster

Now lets move focus to the other type of process. In this process
the cluster will be deployed on AWS EKS. This deployment is also
orchestrated using Jenkins. Below is the overall pipeline:

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

Lets go through each of the stages:

• Environment Variables: First step is to set the needed

environment variables which are used by different steps in
the pipeline. Here are the environment variables which are
being set:

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

• Create S3 bucket for Terraform state: This stage is

identical as the one from the EC2 process. In this stage the S3
bucket is created which will house the Terraform state file.
• Deploy Networking modules: This stage is also identical to
the EC2 process. All of the networking components are
deployed in this stage. Terraform is used for this stage and it
follows the same steps from the EC2 process.
• Deploy EKS Cluster: In this stage the EKS cluster is
launched. Since its a cluster managed by AWS, we just need
to declaratively launch the cluster and AWS will handle the
rest. A Terraform module is created to deploy the cluster. In
the Terraform script all of the parameters for the cluster is
specified. As part of the cluster deployed, the node groups
are also deployed where the workloads will run. I am using
Fargate instance node groups which will spin up as and when
workloads starts running on the cluster. In the Terraform
module, a Fargate profile is defined which gets deployed
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
along with the cluster, This profile makes sure if any
workloads with these specific labels are launched on the
cluster, they will be launched on these Fargate instances
which will launch automatically based on load.

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
Once the cluster deploys successfully, you can view the cluster on
console

One thing to note here is that I am also launching a non-Fargate

node group so that the coredns pods can be scheduled. To run
coredns pods on Fargate instances is possible but I will not cover
that in this post.

• Deploy Sample App: Since we now have the cluster up and

running, to see how the Fargate instances spin up, we will
deploy a sample app in this stage. It will be a simple pod with
the nginx image which will be deployed to the cluster. The
pod will have the specific labels that will trigger creation of
the Fargate instance nodes.

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

Once the pods start running, the Fargate instances can be seen on
the AWS console for EKS:

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar

To test the nodes, run this command from your local terminal.
Make sure to configure AWS CLI so it authenticates to correct AWS
instance
aws eks update-kubeconfig --name cluster_name
kubectl get nodes

The sample pod should also be running now on the Fargate

instance node:

As last step of this stage, a test kubectl command is executed to

ensure the cluster is functioning properly and the nodes are up
and running too.

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
• Notify on Slack: Once the cluster is completely deployed, the
pipeline sends out a message to a specific slack channel
notifying the successful deployment of the cluster. The
message also provides the cluster name.
That completes the deployment of the EKS cluster. Once all these
script changes are done, push the changes to the Git repo so the
pipeline can execute.
git add .
git commit -m "updated pipeline files"
git push -u origin main
Execute the pipeline or the pipeline automatically runs (whatever
is configured on Jenkins), once the execution finishes the EKS
cluster should be up and running.

Note: If you are running through spinning up a cluster just for

learning, make sure to destroy the resources which were created
or you wil incur charges on AWS. Just ru the destroy phase of the

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar
pipeline in my repo by setting the environment variable. It will
destroy all of the resources which were created by the pipeline.

Improvements
The steps explained in the post are very basic way to spin up the
cluster and actual Production steps are way more. Some of the
improvements I am working on the process are:

• Add TLS to the Kubernetes endpoints

• Enable external access to the cluster (cluster using EC2
instances)
These are some ideas right now but there are so many things to be
improved so a perfect production grade cluster can be spun up. I
will plan to write another post in future for a production grade
cluster.

Follow on:
https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

https://www.linkedin.com/in/lokeshkumar-%E2%98%81%EF%B8%8F-4a2860244

Lokeshkumar