07/02/2020 Autoscaling Kubernetes apps with Prometheus and KEDA

Autoscaling Kubernetes apps with Prometheus and

KEDA
Abhishek Gupta
Oct 23, 2019 · 8 min read

Scalability is a key requirement for cloud native applications. With Kubernetes, scaling
your application is as simple as increasing the number of replicas for the corresponding
Deployment or ReplicaSet - but, this is a manual process. Kubernetes makes it possible
to automatically scale your applications (i.e. Pod s in a Deployment or ReplicaSet ) in a

declarative manner using the Horizontal Pod Autoscaler specification. By default, there
is support for using CPU utilization ( Resource metrics) as criteria for auto-scaling, but
it is also possible to integrate custom as well as externally provided metrics.

This blog will demonstrate how you can external metrics to auto-scale a Kubernetes
application. For demonstration purposes, we will use HTTP access request metrics that
are exposed using Prometheus. Instead of using the Horizontal Pod Autoscaler directly,

we will leverage Kubernetes Event Driven Autoscaling aka KEDA - an open source
Kubernetes operator which integrates natively with the Horizontal Pod Autoscaler to
provide fine grained autoscaling (including to/from zero) for event-driven workloads.

The code is available on GitHub

I would love to have your feedback and suggestions! Feel free to tweet or drop a
comment 😃

Overview
Here is a summary of how things work end to end — each of these will be discussed in
detail in this section

https://itnext.io/tutorial-auto-scale-your-kubernetes-apps-with-prometheus-and-keda-c6ea460e4642 1/13
07/02/2020 Autoscaling Kubernetes apps with Prometheus and KEDA

how things work at a high level

The application exposes HTTP access count metrics in Prometheus format

Prometheus is configured to scrape those metrics

Prometheus scaler in KEDA is configured and deployed to auto-scale the app based
on the HTTP access count metrics

KEDA and Prometheus

Prometheus is an open-source systems monitoring and alerting toolkit which is a part
of the Cloud Native Computing Foundation. Prometheus scrapes metrics from various
sources and stores them as time-series data and tools like Grafana or other API
consumers can be used to visualize the collected data.

KEDA supports the concept of Scaler s which act as a bridge between KEDA and an

external system. A Scaler implementation is specific to a target system and fetches

relevant data from it, which is then used by KEDA to help drive auto-scaling. There is
support for multiple scalers(including Kafka, Redis, etc.) including Prometheus. This
means that you can leverage KEDA to auto-scale your Kubernetes Deployment s using

Prometheus metrics as the criteria.

Sample application
The example Golang app exposes an HTTP endpoint and does two important things:

Uses Prometheus Go client library to instrument the app and expose the
http_requests metric backed by a Counter . Prometheus metrics endpoint is

available at /metrics .

var httpRequestsCounter =
promauto.NewCounter(prometheus.CounterOpts{

https://itnext.io/tutorial-auto-scale-your-kubernetes-apps-with-prometheus-and-keda-c6ea460e4642 2/13
07/02/2020 Autoscaling Kubernetes apps with Prometheus and KEDA

Name: "http_requests",
Help: "number of http requests",
})

In response to a GET request, it also increments a key ( access_count ) in Redis - this

is simple way of getting some "work done" as a part of the HTTP handler and also
helps to validate the Prometheus metrics (should be the same as value of the
access_count in Redis)

func main() {
http.Handle("/metrics", promhttp.Handler())
http.HandleFunc("/test", func(w http.ResponseWriter, r
*http.Request) {
defer httpRequestsCounter.Inc()
count, err := client.Incr(redisCounterName).Result()
if err != nil {
fmt.Println("Unable to increment redis counter",
err)
os.Exit(1)
}
resp := "Accessed on " + time.Now().String() + "\nAccess
count " + strconv.Itoa(int(count))
w.Write([]byte(resp))
})
http.ListenAndServe(":8080", nil)
}

The application is deployed to Kubernetes as a Deployment and a ClusterIP service is

also created to allow the Prometheus server to scrape the app /metrics endpoint.

Here is the manifest for the application

Prometheus server
The Prometheus deployment manifest consists of:

ConfigMap to capture Prometheus configuration

Deployment for Prometheus server itself

ClusterIP service to access the Prometheus UI

ClusterRole , ClusterRoleBinding and ServiceAccount to allow for Kubernetes

service discovery to work

https://itnext.io/tutorial-auto-scale-your-kubernetes-apps-with-prometheus-and-keda-c6ea460e4642 3/13
07/02/2020 Autoscaling Kubernetes apps with Prometheus and KEDA

Here is the manifest for Prometheus setup

KEDA Prometheus ScaledObject

As explained previously, a Scaler implementation acts as a bridge between KEDA and
the external system from which metrics need to be fetched. ScaledObject is a custom
resource that needs to be deployed in order to sync a Deployment with an event source
(Prometheus in this case). It contains information on which Deployment to scale,

metadata on the event source (e.g. connection string secret, queue name), polling
interval, cooldown period, etc. The ScaledObject will result in corresponding
autoscaling resource (HPA definition) to scale the Deployment

When a ScaledObject gets deleted, the corresponding HPA definition is cleaned up.

Here is the ScaledObject definition for our example which uses the Prometheus scaler

apiVersion: keda.k8s.io/v1alpha1
kind: ScaledObject
metadata:
name: prometheus-scaledobject
namespace: default
labels:
deploymentName: go-prom-app
spec:
scaleTargetRef:
deploymentName: go-prom-app
pollingInterval: 15
cooldownPeriod: 30
minReplicaCount: 1
maxReplicaCount: 10
triggers:
- type: prometheus
metadata:
serverAddress: http://prometheus-
service.default.svc.cluster.local:9090
metricName: access_frequency
threshold: '3'
query: sum(rate(http_requests[2m]))

Notice the following:

It targets a Deployment named go-prom-app

The trigger type is prometheus . The Prometheus serverAddress is mentioned along

with metricName , threshold and the PromQL query( sum(rate(http_requests[2m])) )

https://itnext.io/tutorial-auto-scale-your-kubernetes-apps-with-prometheus-and-keda-c6ea460e4642 4/13
07/02/2020 Autoscaling Kubernetes apps with Prometheus and KEDA

to be used

As per pollingInterval , KEDA will poll Prometheus target every fifteen seconds.
A minimum of one Pod will be maintained ( minReplicaCount ) and the maximum
number of Pod s will not exceed the maxReplicaCount ( ten in this example)

It is possible to set minReplicaCount to zero . In this case, KEDA will "activate" the

deployment from zero to one and then leave it to HPA to auto-scale it further (the same
is done the other way around i.e. scale in from one to zero). We haven't chosen zero
since this is an HTTP service and not an on-demand system such as a message
queue/topic consumer

Magic behind auto-scale

The threshold count is used to act as the trigger to scale the Deployment. In this
example, the PromQL query sum(rate(http_requests[2m])) returns the aggregated
value of the per-second rate of HTTP requests as measured over the last two minutes.
Since the threshold count is three , this means that there will be one Pod for if value

for sum(rate(http_requests[2m])) remains less than three. If it goes up, there will be an
additional Pod for every time the sum(rate(http_requests[2m])) increases by three e.g.
if the value is between 12 to 14 , the number of Pod s will be 4

Ok it’s time to try it hands-on!

Pre-requisites
All you need is a Kubernetes cluster and kubectl

https://itnext.io/tutorial-auto-scale-your-kubernetes-apps-with-prometheus-and-keda-c6ea460e4642 5/13
07/02/2020 Autoscaling Kubernetes apps with Prometheus and KEDA

Kubernetes cluster — This example uses minikube but feel free to use any other. You
can install it using this guide.

To install latest version on Mac:

curl -Lo minikube

https://storage.googleapis.com/minikube/releases/latest/minikube-
linux-amd64 \
&& chmod +x minikube
sudo mkdir -p /usr/local/bin/
sudo install minikube /usr/local/bin/

Please install kubectl to access your Kubernetes cluster.

To install latest version on Mac:

curl -LO "https://storage.googleapis.com/kubernetes-

release/release/$(curl -s https://storage.googleapis.com/kubernetes-
release/release/stable.txt)/bin/darwin/amd64/kubectl"
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin/kubectl
kubectl version

Setup

Install KEDA
You can deploy KEDA in multiple ways as per the documentation. I am simply using a
monolith YAML to get the job done

kubectl apply -f
https://raw.githubusercontent.com/kedacore/keda/master/deploy/KedaSc
aleController.yaml

KEDA and its components are installed in the keda namespace

To confirm,

kubectl get pods -n keda

https://itnext.io/tutorial-auto-scale-your-kubernetes-apps-with-prometheus-and-keda-c6ea460e4642 6/13
07/02/2020 Autoscaling Kubernetes apps with Prometheus and KEDA

Wait for KEDA operator Pod to start ( Running state) before you proceed

Setup Redis using Helm

If you don’t have Helm installed, simply use this guide. On a mac, you can get this done
quickly using

brew install kubernetes-helm

helm init --history-max 200

helm init is to initialize the local CLI and also install Tiller into your Kubernetes
cluster

kubectl get pods -n kube-system | grep tiller

Wait for the Tiller Pod to switch to Running state

One Helm is setup, getting a Redis server is as simple as running:

helm install --name redis-server --set cluster.enabled=false --set

usePassword=false stable/redis

To confirm whether Redis is ready:

kubectl get pods/redis-server-master-0

Wait for Redis server Pod to start ( Running state) before you proceed

Deploy the application

To deploy:

kubectl apply -f go-app.yaml

https://itnext.io/tutorial-auto-scale-your-kubernetes-apps-with-prometheus-and-keda-c6ea460e4642 7/13
07/02/2020 Autoscaling Kubernetes apps with Prometheus and KEDA

//output
deployment.apps/go-prom-app created
service/go-prom-app-service created

Confirm whether its running

kubectl get pods -l=app=go-prom-app

Wait for application Pod to start ( Running state) before you proceed

Deploy Prometheus server

The Prometheus manifest uses Kubernetes Service Discovery for Prometheus to
dynamically detect application Pod s based on the service label

kubernetes_sd_configs:
- role: service
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_run]
regex: go-prom-app-service
action: keep

To deploy:

kubectl apply -f prometheus.yaml

//output
clusterrole.rbac.authorization.k8s.io/prometheus created
serviceaccount/default configured
clusterrolebinding.rbac.authorization.k8s.io/prometheus created
configmap/prom-conf created
deployment.extensions/prometheus-deployment created
service/prometheus-service created

Confirm whether its running

kubectl get pods -l=app=prometheus-server

https://itnext.io/tutorial-auto-scale-your-kubernetes-apps-with-prometheus-and-keda-c6ea460e4642 8/13
07/02/2020 Autoscaling Kubernetes apps with Prometheus and KEDA

Wait for Prometheus server Pod to start ( Running state) before you proceed

Use kubectl port-forward to access Prometheus UI - you will be able to access

Prometheus UI (or API server) at http://localhost:9090

kubectl port-forward service/prometheus-service 9090

Deploy the KEDA auto-scale config

You need to create the ScaledObject

kubectl apply -f keda-prometheus-scaledobject.yaml

Check KEDA operator logs

KEDA_POD_NAME=$(kubectl get pods -n keda -

o=jsonpath='{.items[0].metadata.name}')
kubectl logs $KEDA_POD_NAME -n keda

you should see

time="2019-10-15T09:38:28Z" level=info msg="Watching ScaledObject:

default/prometheus-scaledobject"
time="2019-10-15T09:38:28Z" level=info msg="Created HPA with
namespace default and name keda-hpa-go-prom-app"

Check application Pod - should have one instance running since minReplicaCount was
1

kubectl get pods -l=app=go-prom-app

Confirm that the HPA resource was created as well

https://itnext.io/tutorial-auto-scale-your-kubernetes-apps-with-prometheus-and-keda-c6ea460e4642 9/13
07/02/2020 Autoscaling Kubernetes apps with Prometheus and KEDA

kubectl get hpa

You should see something like this:

NAME REFERENCE TARGETS MINPODS

MAXPODS REPLICAS AGE
keda-hpa-go-prom-app Deployment/go-prom-app 0/3 (avg) 1
10 1 45s

Autoscaling in action…

Sanity test: Access the app

To access the REST endpoint for our app, simply run:

kubectl port-forward service/go-prom-app-service 8080

You should now be able to access the Go app using http://localhost:8080

To access the endpoint:

curl http://localhost:8080/test

You should see a response similar to:

Accessed on 2019-10-21 11:29:10.560385986 +0000 UTC

m=+406004.817901246
Access count 1

At this point, check Redis as well. You will see that the access_count key has been
incremented to 1

kubectl exec -it redis-server-master-0 -- redis-cli get access_count

//output
https://itnext.io/tutorial-auto-scale-your-kubernetes-apps-with-prometheus-and-keda-c6ea460e4642 10/13
07/02/2020 Autoscaling Kubernetes apps with Prometheus and KEDA

"1"

Confirm that the http_requests metric count has is also the same

curl http://localhost:8080/metrics | grep http_requests

//output
# HELP http_requests number of http requests
# TYPE http_requests counter
http_requests 1

Generate load
We will use hey, a utility program to generate load

curl -o hey https://storage.googleapis.com/hey-

release/hey_darwin_amd64 && chmod a+x hey

You can also download it for Linux or Windows

Invoke it as such

./hey http://localhost:8080/test

By default, the utility sends 200 requests. You should be able to confirm it using the
Prometheus metrics as well as Redis

curl http://localhost:8080/metrics | grep http_requests

//output
# HELP http_requests number of http requests
# TYPE http_requests counter
http_requests 201

kubectl exec -it redis-server-master-0 -- redis-cli get access_count

//output
201

Confirm the actual metric (returned by the PromQL query)

https://itnext.io/tutorial-auto-scale-your-kubernetes-apps-with-prometheus-and-keda-c6ea460e4642 11/13
07/02/2020 Autoscaling Kubernetes apps with Prometheus and KEDA

curl -g 'http://localhost:9090/api/v1/query?
query=sum(rate(http_requests[2m]))'

//output
{"status":"success","data":{"resultType":"vector","result":
[{"metric":{},"value":[1571734214.228,"1.686057971014493"]}]}}

In this case, the actual result is 1.686057971014493 (in value ). This is not enough to

trigger a scale out since the threshold we have set is 3

Moar load!
In a new terminal, keep a track of the application Pod s

kubectl get pods -l=app=go-prom-app -w

Let’s simulate a heavy load with:

./hey -n 2000 http://localhost:8080/test

In sometime, you will see that the Deployment will be scaled out by the HPA and new
Pod s will be spun up.

Check the HPA to confirm the same,

kubectl get hpa

NAME REFERENCE TARGETS

MINPODS MAXPODS REPLICAS AGE
keda-hpa-go-prom-app Deployment/go-prom-app 1830m/3 (avg) 1
10 6 4m22s

If the load does not sustain, the Deployment will be scaled down to the point where only
a single Pod is running

If you check the actual metric (returned by the PromQL query) using

https://itnext.io/tutorial-auto-scale-your-kubernetes-apps-with-prometheus-and-keda-c6ea460e4642 12/13
07/02/2020 Autoscaling Kubernetes apps with Prometheus and KEDA

curl -g 'http://localhost:9090/api/v1/query?
query=sum(rate(http_requests[2m]))'

To clean up

//Delete KEDA
kubectl delete namespace keda

//Delete the app, Prometheus server and KEDA scaled object

kubectl delete -f .

//Delete Redis
helm del --purge redis-server

Conclusion
KEDA allows you to auto scale your Kubernetes Deployments (to/from zero) based on
data from external metrics such as Prometheus metrics, queue length in Redis,
consumer lag of a Kafka topic, etc. It does all the heavy lifting of integrating with the
external source as well as exposing its metrics via a Metrics server for the Horizontal
Pod Autoscaler to weave its magic!

That’s all for this blog. Also, if you found this article useful, please like and follow 😃
😃

Kubernetes Tutorial Prometheus

About Help Legal

https://itnext.io/tutorial-auto-scale-your-kubernetes-apps-with-prometheus-and-keda-c6ea460e4642 13/13