Kubernetes Notes
Kubernetes Notes
Kubernetes Notes
In
this case, the resource is the Pod named db.
Scheduler -- I ts job is to watch for unassigned pods and assign them to a node which has available
resources (CPU and memory) matching Pod requirements.
API server -- runs on the master node
All other components interact with API server and keep watch for changes.
Kubelet runs on each node. make sure that assigned pods are running on the node. It watches for any
new Pod assignments for the node. If a Pod is assigned to the node Kubelet is running on, it will pull
the Pod definition and use it to create containers through Docker or any other supported container
engine.
● kubectl allows us to execute a process in a container running in any node inside a cluster
● docker container exec is limited to containers running on a specific node.
● --container (or -c) argument can be set to specify which container should be used.
kubectl exec -it db pkill mongod
A Pod cannot be split across multiple nodes. Pods are designed to run multiple cooperative processes
kubectl get -f pod/go-demo-2.yml -o json
kubectl get -f pod/go-demo-2.yml \ -o jsonpath="{.spec.containers[*].name}" ---> spcfc
o/p = db api
kubectl exec -it -c db go-demo-2 ps aux -- display the processes inside the db container
kubectl create -f pod/go-demo-2.yml
Selector - It does not distinguish between the Pods created by a ReplicaSet or some other process.
They must match the labels defined in the spec.template. In our case, ReplicaSet
will look for Pods with type set to backend and service set to go-demo-2. If Pods with those labels do
not already exist, it'll create them using the spec.template section.
At a minimum, the labels of the spec.template.metadata.labels section must match those specified in
the spec.selector.matchLabels.
spec.template.spec.containers field is mandatory. It defines a Pod with two containers (db and api)
two containers should not belong to the same Pod. The same is true for the containers in Pods
managed by the ReplicaSet.
Operating ReplicaSets
ReplicaSets and Pods are loosely coupled objects with matching labels
We can, for example, remove the ReplicaSet we created while leaving the two Pods intact.
kubectl delete -f rs/go-demo-2.yml \
--cascade=false
We used the --cascade=false argument to prevent Kubernetes from removing all the downstream
objects. As a result, we got the confirmation that replicaset "go-demo-2" was deleted. Let's confirm
that it is indeed removed from the system.
kubectl get pods
The two Pods created by the ReplicaSet are indeed still running in the cluster even though we removed
the ReplicaSet.
We'll get to them shortly. For now, the important thing is that we are about to create the same
ReplicaSet we had before.
kubectl create -f rs/go-demo-2.yml \
--save-config
The output states that the replicaset "go-demo-2" was created. Let's see what happened with the Pods.
The apply command automatically saves the configuration so that we can edit it later on. The create
command does not do such thing by default so we had to save it with --save-config.
When we applied the new configuration with replicas set to 4 instead of 2, Kubernetes updated the
ReplicaSet which, in turn, evaluated the current state of the Pods with matching labels. It found two
with the same labels and decided to create two more so that the new desired state can match the actual
state.
Let's see what happens when a Pod is destroyed.
POD_NAME=$(kubectl get pods -o name \
| tail -1)
1. Kubernetes client (kubectl) sent a request to the API server requesting the creation of the
Service based on Pods created through the go-demo-2 ReplicaSet.
2. Endpoint controller is watching the API server for new service events. It detected that there is
a new Service object.
3. Endpoint controller created endpoint objects with the same name as the Service, and it used
Service selector to identify endpoints (in this case the IP and the port of go-demo-2 Pods).
4. kube-proxy is watching for service and endpoint objects. It detected that there is a new Service
and a new endpoint object.
5. kube-proxy added iptables rules which capture traffic to the Service port and redirect it to
endpoints. For each endpoint object, it adds iptables rule which selects a Pod.
6. The kube-dns add-on is watching for Service. It detected that there is a new service.
7. The kube-dns added db container's record to the dns server (skydns).
Since the Service is associated with the Pods created through the ReplicaSet, it inherited all their
labels. The selector matches the one from the ReplicaSet. The Service is not directly associated with
the ReplicaSet (or any other controller) but with Pods through matching labels.
The Port is set to 28017. That is the port that the Pods can use to access the Service.
NodePort was generated automatically since we did not set it explicitly. It is the port which we can use
to access the Service and, therefore, the Pods from outside the cluster. In most cases, it should be
randomly generated, that way we avoid any clashes.
Let's see whether the Service indeed works:
PORT=$(kubectl get svc go-demo-2-svc \
-o jsonpath="{.spec.ports[0].nodePort}")
IP=$(minikube ip)
open "http://$IP:$PORT"
The selector is used by the Service to know which Pods should receive requests. It works in the same
way as ReplicaSet selectors. In this case, we defined that the service should forward requests to Pods
with labels type set to backend and service set to go-demo. Those two labels are set in the Pods spec of
the ReplicaSet.
Services can be discovered through two principal modes; environment variables and DNS.
Every Pod gets environment variables for each of the active Services. They are provided in the same
format as what Docker links expect, as well with the simpler Kubernetes-specific syntax.
No matter which set of environment variables you choose to use (if any), they all serve the same
purpose. They provide a reference we can use to connect to a Service and, therefore to the related
Pods.
The key is in the IP field. That is the IP through which this service can be accessed and it matches the
values of the environment variables GO_DEMO_2_DB_* and GO_DEMO_2_DB_SERVICE_HOST.
The code inside the containers that form the go-demo-2-api Pods could use any of those environment
variables to construct a connection string towards the go-demo-2-db Pods. For example, we could
have used GO_DEMO_2_DB_SERVICE_HOST to connect to the database. And, yet, we didn't do
that. The reason is simple. It is easier to use DNS instead. Page 58
Let's go through the sequence of events related to service discovery and components involved:
1. When the api container go-demo-2 tries to connect with the go-demo-2-db Service, it looks at
the nameserver configured in /etc/resolv.conf. kubelet configured the nameserver with the
kube-dns Service IP (10.96.0.10) during the Pod scheduling process.
2. The container queries the DNS server listening to port 53. go-demo-2-db DNS gets resolved to
the service IP 10.0.0.19. This DNS record was added by kube-dns during the service creation
process.
3. The container uses the service IP which forwards requests through the iptables rules. They
were added by kube-proxy during Service and Endpoint creation process.
4. Since we only have one replica of the go-demo-2-db Pod, iptables forwards requests to just
one endpoint. If we had multiple replicas, iptables would act as a load balancer and forward
requests randomly among Endpoints of the Service.
Services are indispensable objects without which communication between Pods would be hard and
volatile. They provide static addresses through which we can access them not only from other Pods but
also from outside the cluster. This ability to have fixed entry points is crucial as it provides stability to
otherwise dynamic elements of the cluster. Pods come and go, Services stay.
We learned how to deploy our applications packaged as Pods, how to scale them through ReplicaSets,
and how to enable communication through Services. However, all that is useless if we cannot update
those applications with new releases. That is where Kubernetes Deployments come in handy.
Failures caused by circumstances outside of our control are things which, by definition, we can do
nothing about. However, failures caused by obsolete practices or negligence are failures which
should not happen. Kubernetes Deployments provide us with the tools we need to avoid such failures
by allowing us to update our applications without downtime.
Let's explore how Kubernetes Deployments work and the benefits we gain by adopting them.
Creating a Cluster
We can see that it created a new ReplicaSet and that it scaled the old ReplicaSet to 0.
Instead of operating directly on the level of Pods, the Deployment created a new ReplicaSet which, in
turn, produced Pods based on the new image. Once they became fully operational, it scaled the old
ReplicaSet to 0. Since we are running a ReplicaSet with only one replica, it might not be clear why it
used that strategy. When we create a Deployment for the API, things will become more evident.
You'll notice that contained within the name of the Pod is a hash which matches the hash in the name
of the new ReplicaSet, namely f8d4b86ff. Even though it might look like it is a random value, it is not.
If you destroy the Deployment and create it again, you'll notice that the hash in the Pod name and
ReplicaSet name remain consistent. This value is generated by hashing the PodTemplate of the
ReplicaSet
As long as the PodTemplate is the same, the hash value will be the same as well. That way a
Deployment can know whether anything related to the Pods has changed and, if it does, will create a
new ReplicaSet.
applications are supposed to have very high availability.
We cannot avoid all failures, but we can reduce them to acceptable limits
Scaled applications can not only spread the load across various instances but ensure that a failure of
one replica will not produce downtime. Healthy instances are handling the load until the scheduler
recreates failed ones.
High availability is accomplished through fault tolerance and scalability. If either is missing, any
failure might have disastrous effects.
If a Pod is unchangeable, the only way to update it with a new release is to destroy the old ones and
put the Pods based on the new image in their place. Destruction of Pods is not much different from
failures. In both cases, they cease to work. On the other hand, fault tolerance (re-scheduling) is a
replacement of failed Pods.
frequency of new releases-- Part of the reasons behind such infrequent releases was due to the
downtime they produce. If we can reach zero-downtime deployments, the frequency can change
and we can aim for continuous deployment.
that new releases result in Pods being replaced with new ones based on the new image. As long as the
process is controlled, new releases should not result in any downtime
when multiple replicas of an application are running and when they are adequately designed.
If we can reach zero-downtime deployments, the frequency can change, and we can aim for
continuous deployment.
Zero-downtime deployment is a prerequisite for higher frequency releases.
minReadySeconds --- defines the minimum number of seconds before Kubernetes starts considering
the Pods healthy. We put the value of this field to 1 second. The default value is 0, meaning that the
Pods will be considered available as soon as they are ready and, when specified, livenessProbe returns
OK. If in doubt, omit this field and leave it to the default value of 0. We defined it mostly for
demonstration purposes.
revisionHistoryLimit. It defines the number of old ReplicaSets we can rollback. Like most of the fields,
it is set to the sensible default value of 10. We changed it to 5 and, as a result, we will be able to
rollback to any of the previous five ReplicaSets.
The strategy can be either the RollingUpdate or the Recreate type.
A bug will sneak in and put our production cluster at risk. What should we do in such a case?
depends on the size of the changes and the frequency of deployments.
we are deploying small chunks. In such cases, fixing a problem might be just as fast as rolling back
Rolling back a release that introduced database changes is often not possible. Even when it is, rolling
forward is usually a better option when practicing continuous deployment with high-frequency
releases limited to a small scope of changes.
We'll imagine that we just discovered that the latest release of the vfarcic/go-demo-2 image is faulty
and that we should roll back to the previous release. The command that will do just that is as follows:
kubectl rollout undo \
-f deploy/go-demo-2-api.yml
kubectl describe \
-f deploy/go-demo-2-api.yml
It started increasing the replicas of the older ReplicaSet, and decreasing those from the latest one.
Once the process is finished, the older ReplicaSet became active with all the replicas, and the newer
one was scaled down to zero.
Since new deployments do no destroy ReplicaSets but scale them to 0, all it had to do to undo the last
change was to scale it back to the desired number of replicas and, at the same time, scale the current
one to zero.
We can undo the rollout by moving to the last revision that worked correctly. Assuming that we want
to revert to the image vfarcic/go-demo-2:2.0, reviewing the change causes listed in the history tells us
we should roll back to revision 2. That can be accomplished through the --to-revision argument. The
command is as follows:
kubectl rollout undo \
-f deploy/go-demo-2-api.yml \
--to-revision=2
kubectl rollout history \
-f deploy/go-demo-2-api.yml
We undid the rollout by moving to revision 2. We also retrieved the history.
Even though most of the time we send requests to specific objects, almost everything is happening
using selector labels. When we updated the Deployments, they looked for matching selectors to choose
which ReplicaSets to create and scale. They, in turn, created or terminated Pods also using the
matching selectors. Almost everything in Kubernetes is operated using label selectors. It's just that
sometimes that is obscured from us.
We do not have to update an object only by specifying its name or the YAML file where its definition
resides. We can also use labels to decide which object should be updated.
That opens some interesting possibilities since the selectors might match multiple objects.
Imagine that we are running several Deployments with Mongo databases and that the time has come
to update them all to a newer release.
Now that we have two deployments with the mongo:3.3 Pods, we can try to update them both at the
same time.
The trick is to find a label (or a set of labels) that uniquely identifies all the Deployments we want to
update.
Let's take a look at the list of Deployments with their labels:
kubectl get deployments --show-labels
Scaling Deployments
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: go-demo-2
annotations:
ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
The annotations section allows us to provide additional information to the Ingress controller.
The list of general annotations and the Controllers that support them can be found in the Ingress
Annotations
page(h ttps://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guid
e/nginx-configuration/annotations.md)
ssl-redirect:"false" tells the Controller that we do NOT want to redirect all HTTP requests to HTTPS.
We're forced to do so since we do not have SSL certificates for the exercises that follow.
We specified a set of rules in the spec section. They are used to configure Ingress resource. For now,
our rule is based on http with a single path and a backend. All the requests with the path starting with
/demo will be forwarded to the service go-demo-2-api on the port 8080.
create the resource.
kubectl create \
-f ingress/go-demo-2-ingress.yml
kubectl get \
-f ingress/go-demo-2-ingress.yml
create the resource.
kubectl create \
-f ingress/go-demo-2-ingress.yml
kubectl get \
-f ingress/go-demo-2-ingress.yml
We can define multiple Ingress resources that will configure a single Ingress controller.
ontinuous Integration
Versus Continuous Delivery
Versus Continuous
Deployment
Continuous integration, continuous delivery, and continuous deployment—if you do any work in the area of software delivery, it’s impossible not to encounter these terms on a regular basis. But what does each of these processes do for your product
development and release cycles? I’ll explain what they really boil down to, what practices are associated with them, and what kinds of tooling are available to implement them in a manner that clearly depicts what they offer, how they differ, and how
they work, both separately and together, to help release software to customers frequently, reliably, and with high quality.
Defining “Continuous”
Continuous here implies automation—automation in transforming source code into deliverables, in testing and validation, and even in installing and configuring software. It doesn’t imply that processes are continuously executing; rather, that when a
new change is introduced into the system it can be automatically processed quickly and thus enables more frequent builds and faster testing, packaging, and releases than without continuous practices. Further, it enables quick identification and
reporting of failures so that they can be fixed swiftly. This is commonly known as fail-fast.
Continuous here also implies that a change can proceed, without human intervention, through the stages of builds, testing, packaging, and so on. Combined, these stages form a “pipeline” of processes and applications to take source code changes and
turn them into a releasable product. This pipeline can go by various names, including continuous delivery pipeline, deployment pipeline, or release pipeline. Although hand-offs between stages are intended to be automatic as the sequence progresses,
there are two cases in which human intervention can be required:
With this background in mind, let’s look closer at what each of these three terms encompasses.
Continuous Integration
In the continuous integration (CI) phase, individual changes from a developer are merged and tested. The goal of CI is to quickly validate individual code changes that have been submitted for inclusion in the code base. An intended consequence of
this is quickly identifying any problems in the code and automatically notifying the developer of problems so that the code base is not broken any longer than necessary. The mechanism for CI detects that code changes have been made and then runs
targeted testing to prove that the code changes work in isolation (function inputs produce the desired outputs, bad inputs are flagged appropriately, and so on).
Unit Tests
These targeted tests are called unit tests, or commit tests, and developers are responsible for creating them to exercise their code. In fact, one model (known as test-driven development [TDD]) requires unit tests to be designed first—as a basis for
clearly identifying what the code should do.
In the standard CI workflow, a developer creates or updates source in their local working environment and uses the unit tests to ensure that the newly developed function or method works. Because such changes can be frequent and numerous, and
because they are at the beginning of the pipeline, the unit tests must be fast to execute, must not depend on (or make calls to) other code that isn’t directly accessible, and should not depend on external data sources or other modules. If such a
dependency is required for the code to run, those resources can be mocked by using a stub that looks like the resource and can return values, but doesn’t actually implement any functionality.
Typically, these tests take the form of asserting that a given set of inputs to a function or method produce a given set of outputs. They generally test to ensure that error conditions are properly flagged and handled. Various unit testing frameworks,
such as JUnit for Java development, are available to assist with this.
CI Tools
The developer can then push their code and unit tests into the source code repository. A variety of different source code management systems, such as Git, are available for use today. If the code is pushed and does not merge cleanly with other users’
changes, the developer must manually fix the conflicts and try again.
After the code is merged in, a CI tool such as Jenkins can detect (or be notified) that a change has been made in the source code repository. It will then automatically grab the latest set of code from the repository and attempt to build it and execute the
unit tests. Jenkins can detect changes by regularly polling the source control system to check for changes or by having a scheduled, periodic build of whatever is current. Alternatively, most source control systems can have a hook set up to send a
notification to tools like Jenkins that a change has occurred.
Prechecks
A variation on this simple workflow involves adding in prechecks on the code before it makes it all the way into the source code repository. For example, a tool called Gerrit can intercept pushes to a remote Git repository and wait for a person to sign
off on a review of the code before allowing it to go into the repository. It can also initiate early Jenkins builds of the code at that point as another pass/fail check.
GitHub, a public hosting site for Git source repositories, offers a further variation. Users wanting to contribute changes to a source repository on GitHub can submit a pull request to ask the owner of a repository to review and then merge in code
changes from the user’s copy of the project.
These variations provide additional checking and validation by multiple people earlier in the process—at the expense of reduced frequency of integration.
Continuous Delivery
In the continuous delivery cycle, the isolated changes merged and validated during CI can be combined with the remaining product code. Then, the set is put through progressively more encompassing types of testing. The goal of continuous delivery
is not necessarily to deploy the end result, but to prove that the end result is deployable. The mechanism of continuous delivery is the continuous delivery pipeline (CDP).
Artifacts
The CDP is made up of stages connected in a sequence. Each stage in turn can be made up of smaller “jobs” with specialized tasks. For example, what’s typically referred to as the commit stage usually consists of jobs to compile the code, run the unit
tests, do integrated testing, gather metrics, and publish artifacts. This then could have an automated handoff to an acceptance stage in which artifacts are retrieved and functional testing and verification are done.
An artifact refers to a file that is either a deliverable (something directly used by the final product) or included in a deliverable. For example, you might have an executable file created by compiling source that links in several other libraries. Or you
might have a compressed file such as a .war or .zip file that contains another set of files within it.
Versioning
Regardless of the type of artifact, an important dimension across the pipeline is the artifact versioning. Although deployment technology has made versioning less of a concern to users (think app updates on your phone), it is critical to work with the
desired versions of artifacts through the pipeline. This is usually managed via an artifact repository tool such as Artifactory. These tools work like a source management system for artifacts, allowing multiple versions to be stored and retrieved to and
from different repositories. In some cases, these repositories can serve as a dev/test/prod (development-quality/testing-quality/production-quality) organizational structure to separate multiple versions of releases in progress.
Testing
If incorrect versions of artifacts are pulled in or functionality is broken, the automated testing in the pipeline will present some symptom of this. The goal of the various testing phases in the pipeline is to instill more and more confidence as we
progress toward the final product. There are several types of testing, including:
Integration testing
To validate the end result of executing functions in the product are as expected.
Acceptance testing
To measure some characteristic of the system against acceptable criteria. Examples include performance, scalability, and capacity.
Infrastructure-as-Code
One other recommended practice that DevOps culture has introduced to continuous delivery is the infrastructure-as-code ideal. In this approach, configuration and setup of the tools used in the pipeline are automated and described in files that can be
stored in source control. A change to any of these files drives a reconfiguration or update of the tools used in the pipeline, and triggers a new run of the pipeline, just as a source code change for the product code would.
Continuous Deployment
Continuous deployment is a process that can be added on at the end of the continuous delivery pipeline. The idea is that since the continuous delivery pipeline has proven the latest changes to be deployable, we should automate the deployment process
and go ahead and deploy the result. Deployment can mean a lot of things, but for simplicity can just think of it as making the release available for the customer to use. That might mean that it’s running in the cloud, made downloadable, or updated in
some other known location.
Manual Checks
If a deployment is not automated, there are limitations on what can be done here. Even if it is automated, an organization might not choose to always deploy every release, so a manual check might be added in. More commonly, though, such a check
would take the form of a user acceptance test (UAT) earlier in the pipeline.
Staged Deployments
Even if you want to automatically deploy the latest version from the pipeline, you can still do it in a staged way. For example, you can deploy it to a location that is accessible only by a subset of users who might want to try out the new release, or you
can redirect some users to the new release transparently. Then, if any problems surface during an evaluation period, not all users are affected, and the subset can be directed back to the previous release. If all is well, the remaining users can then be
redirected to the new release. This sort of deployment is called a canary deployment.
Blue/Green Deployments
A broader form of the canary deployment is called blue/green deployment. In this case, two deployment environments are always available: one labeled blue and one labeled green. At any given time, one of these is production and one is in
development or testing (nonproduction). When a new release is deployed, it is deployed to the nonproduction environment for evaluation. When it is deemed production ready, the nonproduction environment is toggled to be production (that is,
customer interaction is redirected to it). And the previous production environment is toggled to be nonproduction, ready for the next candidate release.
Conclusion
As you can see, continuous integration, continuous delivery, and continuous deployment form a powerful set of disciplines, best practices, and technologies for transforming changes from source to production quickly and reliably. The end result is
high quality and easily reproducible. This process not only benefits the customer, but also greatly simplifies the worlds of the development, testing, and deployment teams. It lets them work together in a common environment from start to finish,
helping to realize one of the core goals of the DevOps movement.
● Copy
● Add Highlight
● Add Note