LivenessProbe, ReadinessProbe, StartupProbe in Kubernetes

Probes are used by kubelet in Kubernetes to indicate the container health and the state in a pod.

A probe can be a http probe or tcp probe to verify the health. (Flow Diagram of different probes)

Attributes used in defining probes

Startup Probes: Used to check if the application inside the Container has started

Readiness Probes: Used to check if the application is ready to use and serve the traffic.

Liveness Probes: Used to check if the container is available and alive.

Startup Probes:

It can be used for slow starting containers with legacy applications.
It indicates if the application inside the Container has started.
If a startup probe is provided, liveness and readiness probes wait to start after it.
After the startup probe has succeeded, the liveness probe takes over to provide a fast response.
In below example, if /manage/health request fails 3 times, it will restart the container.

startupProbe:
  httpGet:
    path: /manage/health
    port: 8080
  initialDelaySeconds: 60
  timeoutSeconds: 90
  periodSeconds: 10
  failureThresold: 3

Readiness Probes:

It helps kubelet to understand if container in pod is ready to be used
Once the container in the pod ready, traffic is sent from service to this backend pod
The logic of checking application dependencies like database/kafka/redis and/or other services the application are implemented in readiness probe.
During rolling updates, it is used by kubelet to keep the old container up and running while checking new pods to be ready for actual traffic.
It runs on the container during its whole lifecycle.
Incorrect implementation of it may result in an ever-growing number of processes in the container.
In below example, if /manage/health request fails 3 time, it won’t serve any traffic

readinessProbe:
  httpGet:
    path: /manage/health
    port: 8080
  initialDelaySeconds: 60
  timeoutSeconds: 90
  periodSeconds: 10
  failureThresold: 3

Liveness Probes:

It is used to indicate if the container has started and is alive or not
It could catch a situation, where an application is running, but unable to make progress.
Restarting a container in such a state can help to make the application available.
It doesn’t wait for readiness probes to succeed. If you want to wait before executing a liveness probe you should use initialDelaySeconds or a startupProbe.
In below example, if /manage/health request fails 3 times, it will restart the container.

livenessProbe:
  httpGet:
    path: /manage/health
    port: 8080
  initialDelaySeconds: 60
  timeoutSeconds: 90
  periodSeconds: 10
  failureThresold: 3

Reference

Git Issue : Recover deleted code files from repository

Git is a decentralised version control system that works as content tracker for codes. It store project source codes of each developer who works in a project. It provides options to create/delete/modify files and folders, create/switch/remove branches and more in local repo and remote repo.

The local repo is found on each developer’s local system.

The remote repo is hosted on a central server that has copies of the files to be available on each developer’s computer.

We hate to lose code by accidental deletion due to conflicts or error or unwanted. It is painful if we put efforts and suddenly deleted unknowingly. But git has the capabilities to recover deleted file in simple and easy steps.

I had this difficult situations multiple times and I will explain how I recovered files in different situations with few tips.

1. Deleted a file with rm command without commit

Losing your files on git after committing your work on Git is very rare. In fact, it takes deleting your entire local repository and not having any backup on the remote server to totally lose a file on Git.

If you deleted a file ( file1.txt ) without committing, and found the issue, run below command immediately.

2. Deleted multiple files with rm command without commit

If you deleted multiple files in current path without committing, and found the issue, run it

3. Deleted a file on git and committed but not pushed to remote

When you deleted a file, committed the deletion, and found out you needed the file. But you have not pushed it remote.

To recover this file, you need to do perform a reset. This reset will return you to a state before your commit. After below command you can see the deleted file is at previous path.

4. Deleted a file on git, committed and pushed to remote

In this case also we can recover the file with a new commit with undo function of git. You should not use reset and push file, which will rewrite the local repos history and file can not be retrieved.

With below –no-commit command, we do a fresh commit which will alter the old commit.

You need to copy the old commit id [4ad8839 in above example] from commit history[git log or github Web UI] which was used to delete the file.

5. Deleted file on git, committed and make some other commits

While working, we may committed a deletion of file1.txt, and then went ahead to work and made more commits. You need to find the commit from commit history[git log — file1.txt or github Web UI] which was used to delete the file.

These are the few tips I used in my work to recover accidental deleted files. Please comment if any one knows more tricks.

ArgoCD Local Setup : deployment with Ingress

Pre-req : Kubernetes cluster should be running and kubectl command should be configured in MAC.In some other articles I will share how to setup kubernetes cluster.

Note : This is a insecure setup for local testing only. In another article I will share the setup with ArgoCD app of apps model how it is used in enterprises.

Install Argocd

kubectl create namespace argocd

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Install argocd cli

brew install argocd

Install nginx controller

helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace

check nginx pods

kubectl get pods --namespace=ingress-nginx

wait for nginx pod creation complete

kubectl wait --namespace ingress-nginx --for=condition=ready pod --selector=app.kubernetes.io/component=controller --timeout=120s

Local testing with a dummy deployments

kubectl create deployment demo --image=httpd --port=80

kubectl expose deployment demo

Expose via ingress

kubectl create ingress demo-localhost --class=nginx --rule=demo.localdev.me/*=demo:80

Access the above domain url : demo.localdev.me

Get the argocd admin secret password

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

Port forward to 8080

kubectl port-forward svc/argocd-server -n argocd 8080:443

Login with the admin secret password

argocd login 127.0.0.1:8080

Update the password

argocd account update-password

Apply the ingress manifest for argocd

kubectl apply -f https://github.com/divyaimca/my-k8s-test-projects/blob/main/argocd/argocd_local_ingress.yaml

If you face issue in above step download it and apply the ingress locally.

Access argocd by using the hostname defined in the above manifest(argocd.k8s.local)

Reference : https://github.com/divyaimca/my-k8s-test-projects/tree/main/argocd

ArgoCD Experience : what and why

ArgoCD is a declarative Continuous Deployment tool used to deploy kubernetes workloads using GitOps principles. It is basically a opensource kubernetes native tool and use pull model to pull application configurations from git repos and deploy them in kubernetes cluster.

It helps in regular deployments of micorservice workloads faster in n number of environments.

My experience on ArgoCD:

I have automated end to end continuous deployment of 100+ microservices in 90+ Kubernetes environments(25 Production, 25 stage, 15 sales, 15 QA, 10 perf) in my organisation using ArgoCD, GitHub, Jenkins. Now CD is in autopilot mode with very minimal intervention.

ArgoCD Capabilities

Deployment	support both manual and automatic
Interface	Web UI and Command Line
Synchronisation	automatic app state is synced from app config
UI Capability	show each resource of kubernetes in UI
CD Issues	Ability to visualise deployment issues, detect and remediate configuration drift
Authentication	Single sign-on (SSO), OAuth2, OIDC, LDAP, and SAML 2.0
Authorisation	Granular access control with RBAC

What is GitOps in ArgoCD:

GitOps is a software engineering practice that uses a Git repository as its single source of truth. Engineers commit declarative configurations into Git, and these configurations are pulled to create environments needed for the continuous delivery process.

High level Argo CD process:

We makes changes to an application, pushing a new version of Kubernetes resource definitions to a Git repo.
Continuous integration is triggered, resulting in a new container image saved to a registry.
We issue a pull request that creates/changes Kubernetes manifests, which are created either manually or automatically.
The pull request is reviewed and changes are merged to the main branch. This triggers a webhook which tells Argo CD a change was made.
The git repos are pre registered in ArgoCD during setup.
Argo CD clones the repo and compares the application state with the current state of the Kubernetes cluster. It applies the required changes to cluster configuration.
Kubernetes uses its controllers to reconcile the changes required to cluster resources, until it achieves the desired configuration.
Argo CD monitors progress and when the Kubernetes cluster is ready, reports that the application is in sync.
ArgoCD also works in the other direction, monitoring changes in the Kubernetes cluster and discarding them if they don’t match the current configuration in Git.
In case any issues ArgoCD UI gives option to manually
- refresh(read new config from git repo)
- sync(load the new config from git repo)
- force(sync forcefully all out of sync resources)
- prune(remove the selected workloads)
- replace(recreate the selected workloads from scratch)

A visual diagram of how ArgoCD GitOps works:

Docker Swarm Quick Guide Part 2

docker swarm commands

docker swarm init	initialises a new docker swarm
docker swarm ca	display and rotate the root CA
docker swarm join-token	join a worker node to an existing swarm
docker swarm leave	leave the swarm
docker swarm init –autolock	enables autolock on the swarm
docker swarm update	update the swarm with new parameter values

docker stack commands

docker stack deploy	used to deploy a new stack [need the docker compose file]
docker stack ls	lists all the stacks and services that make up the stacks
docker stack services	list the services in a given stack
docker stack ps	list all the tasks for a given stack
docker stack rm	removes the stack
docker stack config	output the final config file after merges and interpolations

docker service commands

docker service create –replicas 3 -p 80:80 –name web https	creates a new docker service, based on httpd image with 3 replicas
docker service logs	outputs the logs of a service
docker service ls	lists all the services in the swarm
docker service ps	list the tasks of the service
docker service rm	deletes a service
docker service scale	used to add or remove replicas from a service
docker service update	used to update a service with a new image
docker service inspect	display detailed information on service

docker node commands

docker node ls	lists the docker nodes in a swarm
docker node ps	lists the tasks on a docker node
docker node rm	removes a node from the swarm
docker node demote	demotes a manager node
docker node promote	promotes a node to swarm manager

docker swarm network commands

docker network ls	lists networks
docker network create -d overlay network_name	create overlay network
docker network rm network_name	remove network

Docker Swarm Quick Guide Part 1:

A swarm consists of multiple Docker hosts which run in swarm mode and act as managers (to manage membership and delegation) and workers (which run swarm services).

Why Docker Swarm ?

Secure by default	nodes in the swarm enforces TLS mutual authentication and encryption to secure communications between itself and all other nodes.
cluster management	fully integrated with the docker engine. No additional software required for docker swarm mode
decentralized design	You can deploy both kinds of nodes, managers and workers, using the Docker Engine.
declarative service model	Docker Engine uses a declarative approach to let you define the desired state of the various services in your application stack.
scaling	For each service, you can declare the number of tasks you want to run.
desired state	The swarm manager node constantly monitors the cluster state and reconciles any differences.
multi-host networking	You can specify an overlay network for your services.
service discovery	Swarm manager nodes assign each service in the swarm a unique DNS name
load balancing	You can expose the ports for services to an external load balancer.
rolling updates	If anything goes wrong, you can roll back to a previous version of the service.

Some Key Terms in Docker Swarm

Node	A physical or virtual machine on which docker is running
Manager	Performs swarm management and orchestration duties. Also acts as a worker node by default
Worker	Runs Docker Swarm tasks
Swarm Cluster	A group of Docker nodes working together
Stack	A collection of services that typically make up an application
Service	When you create a service, you define its optimal state (number of replicas, network and storage resources available to it, ports the service exposes to the outside world, and more).
Task	Services start tasks. A task is a running container which is part of a swarm service and managed by a swarm manager, as opposed to a standalone container.

Custom default backend error pages of kubernetes ingress

The kubernetes nginx ingress controller has a default backend which show error pages of 404, 502 with nginx string in the error page. But some times we need to show a valid custom error page instead of showing the pages served by the default backend.

The process is simple. We need to create a configmap with custom error pages, create deployment with image k8s.gcr.io/ingress-nginx/nginx-errors with mounting the config map in /www. Also we need to create service which will be used as the default backend service of the ingress controller.

Configmap Manifest : https://github.com/divyaimca/my-k8s-test-projects/blob/main/ingress-nginx/custom-default-backend.yaml#L72-L539

Note : update the custom error pages under data with the required error HTML content

Deployment Manifest : https://github.com/divyaimca/my-k8s-test-projects/blob/main/ingress-nginx/custom-default-backend.yaml#L19-L70

Service Manifest : https://github.com/divyaimca/my-k8s-test-projects/blob/main/ingress-nginx/custom-default-backend.yaml#L2-L17

Modification in ingress controller arguments : https://github.com/divyaimca/my-k8s-test-projects/blob/main/ingress-nginx/ingress-deploy.yaml#L337

Note: Here update the service name matching the custom error service name

Next thing we need to update the ingress definition file for which we want to use the custom error pages.

We need to add 2 annotations for this :

Pointing to the custom error service name
mention the custom error to be served.

Ingress manifest update Example : https://github.com/divyaimca/my-k8s-test-projects/blob/main/rabbitmq_kustom/rabbitmq-ingress.yaml#L9-L10

Now if you want to access the webpage served by the ingress with some error, the ingress will serve the customised backend error pages instead of the default backend error pages.

Custom default backend error pages of kubernetes ingress controller

The kubernetes nginx ingress controller has a default backend which show error pages of 404, 502 with nginx string in the error page. But some times w need to show valid custom error pages instead of showing the pages served by the default backend.

Configmap Manifest : https://github.com/divyaimca/my-k8s-test-projects/blob/main/ingress-nginx/custom-default-backend.yaml#L72-L539

Note : update the custom error pages under data with the required error HTML content

Deployment Manifest : https://github.com/divyaimca/my-k8s-test-projects/blob/main/ingress-nginx/custom-default-backend.yaml#L19-L70

Service Manifest : https://github.com/divyaimca/my-k8s-test-projects/blob/main/ingress-nginx/custom-default-backend.yaml#L2-L17

Modification in ingress controller arguments : https://github.com/divyaimca/my-k8s-test-projects/blob/main/ingress-nginx/ingress-deploy.yaml#L337

Note: Here update the service name matching the custom error service name

Next thing we need to update the ingress definition file for which we want to use the custom error pages.

We need to add 2 annotations for this :

Pointing to the custom error service name
mention the custom error to be served.

Ingress manifest update Example : https://github.com/divyaimca/my-k8s-test-projects/blob/main/rabbitmq_kustom/rabbitmq-ingress.yaml#L9-L10

Now if you want to access the webpage served by the ingress with some error, the ingress will serve the customised backend error pages instead of the default backend error pages.

Jenkins slave as a service in windows to start automatically

In Jenkins many time we have to add windows machine as slave, where we need the agent to be up and running as windows service.

There are many ways to do it, but I struggled to find the correct configuration steps. I used windows resource toolkit to make it work and adding the steps here.

Configuration Steps:

(A) Adding the windows slave in Jenkins server :

1. Add the agent in Jenkins Master with Launch Method : Launch Via Java Web Start

Screen Shot 2019-09-25 at 4.07.06 PM

(B) Creating the Service in windows Server for starting the slave agent: (In Windows Server 2016)

1. Download and install the Java 8.
2. Down and Install Windows Resource Kit Tools (https://www.microsoft.com/en-us/download/details.aspx?id=17657)
3. Create a blank service called “Jenkins Slave” by running the following from a command prompt

“C:\Program Files (x86)\Windows Resource Kits\Tools\instsrv.exe” “Jenkins Slave” “C:\Program Files (x86)\Windows Resource Kits\Tools\srvany.exe”

4. Open Registry Editor and go to below location :

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Jenkins Slave

Now Follow the below steps carefully.

Create a string value “Description”
Populate it with “Jenkins Continuous Integration Slave”

Screen Shot 2019-09-25 at 2.13.33 PM

Create a new key “Parameters”
Under “Parameters” create a new string value “Application”
Populate it with the full path to java.exe, something like “C:\sapjvm_8\bin\java.exe”
Under “Parameters” Create a new string value “AppParameters”
Populate it with

“-jar E:\jenkins_agent\agent.jar -jnlpUrl http://m1-hostname.lab.saas.company.corp:8080/computer/hostame-of-machone/slave-agent.jnlp -secret <secret-name> -workDir E:\jenkinsWorkSpace”

The slave.jar should point to the correct location
The Jenkins master machine name should be correct
The new Jenkins slave machine should be correct
Make sure you use the secret for this machine that you copied from the master when adding the new node

Screen Shot 2019-09-25 at 2.24.06 PM

Open the Services application from Control Panel – Administrative Tools, find the “Jenkins Slave” service, right click on the service and go to “Properties”.

Go to the “Recovery” tab and change “First failure” and “Second failure” to “Restart the Service” – occasionally we found it wouldn’t start up first time out
Go to the “Log On” tab and set an account and password- we found that using an account with local admin rights on the slave machine worked best but this is probably unnecessary
Go to the “General” tab and change the “Startup type” to “Automatic” – make sure the service
starts up when you restart the slave
Click the “OK” button
Now start the service

6. The service will run by default during startup of the windows machine.

7. Now Verify the agent is up and running in Jenkins Web Page.

Screen Shot 2019-09-25 at 4.03.00 PM

FOSS -THE FUTURE IS HERE

The open source is dominating the IT market as the alternative of every type of proprietary s/w .So why not use your freedom !!

Git Issue : Recover deleted code files from repository

1. Deleted a file with rm command without commit

2. Deleted multiple files with rm command without commit

3. Deleted a file on git and committed but not pushed to remote

4. Deleted a file on git, committed and pushed to remote

5. Deleted file on git, committed and make some other commits

ArgoCD Local Setup : deployment with Ingress

ArgoCD Experience : what and why

ArgoCD Capabilities

What is GitOps in ArgoCD:

Custom default backend error pages of kubernetes ingress

Custom default backend error pages of kubernetes ingress controller

Jenkins slave as a service in windows to start automatically

(A) Adding the windows slave in Jenkins server :

(B) Creating the Service in windows Server for starting the slave agent: (In Windows Server 2016)

Startup Probes:

Readiness Probes:

Liveness Probes:

Share this:

1. Deleted a file with rm command without commit

2. Deleted multiple files with rm command without commit

3. Deleted a file on git and committed but not pushed to remote

4. Deleted a file on git, committed and pushed to remote

5. Deleted file on git, committed and make some other commits

Share this:

Share this:

ArgoCD Capabilities

What is GitOps in ArgoCD:

Share this:

docker swarm commands

docker stack commands

docker service commands

docker node commands

docker swarm network commands

Share this:

Share this:

Share this:

Share this:

Share this:

(A) Adding the windows slave in Jenkins server :

(B) Creating the Service in windows Server for starting the slave agent: (In Windows Server 2016)

Share this: