1424, 410M Comprehensive AWS EKS Cluster Monitoring with Promethous, Grafana.and EFK stack{il- 1Oweeksofeleudops & Follow Comprehensive AWS EKS Cluster Monitoring with Prometheus, Grafana,and EFK stack|j]- 10weeksofcloudops > Mallikarjun Konemane Jul 22,2023 - [T] 22 min read Q bitpssdevo hashnode. devlcomprehensive-aws-eks.clustor-monloing-with-prometheus-gralanaand-afkstack-1Oweeksofcloudops 18‘14124, 4:10PM Comprehensive AWS EKS Cluster Monitoring with Promethous, Grafana.and EFK stack{il- 1Oweeksofeleudops @ Gen Introduction 3% Hi folks, this is Mallikarjun! €% | recently graduated from Engineering and am currently exploring and learning DevOps and cloud technologies. 2 This is my first blog post, and I'm very excited to learn and grow as part of our vibrant cloudops community. #3 | would like to express my heartfelt gratitude to my guru, Piyush Sachdeva, for providing invaluable guidance and support throughout my journey. Piyush's expertise and mentorship have been instrumental in shaping my understanding of DevOps and cloud technologies. J During my quest to expand my knowledge, | stumbled upon the captivating #10weeksofcloudops challenge curated by the amazing Piyush Sachdeva on YouTube. 3% @ Intrigued by the opportunity to dive deeper into cloud operations, | eagerly joined the dedicated Discord channe! to actively participate in the challenge. This challenge has been a catalyst for my growth, pushing me to explore new concepts and connect with fellow cloud enthusiasts. To gain a comprehensive understanding of Monitoring tools, | delved into numerous YouTube videos by Abhishek Veermalla and Techworld with Nana. Their insightful tutorials and explanations have been an incredible resource in helping me accomplish my goal of learning about monitoring solutions. (aij Their dedication to sharing knowledge and expertise in the DevOps domain is truly inspiring. () | am immensely grateful to all the commhity folks who have extended their helping hands and contributed to my learning experience. @ bitpssdevo hashnode. devlcomprehensive-aws-eks.clustor-monloing-with-prometheus-gralanaand-afkstack- 1Oweeksofcloudops 2148‘14124, 4:10PM Comprehensive AWS EKS Cluster Monitoring with Promethous, Grafana.and EFK stack{il- 1Oweeksofeleudops & Synopsis In this blog post, we will guide you through the step-by-step process of deploying a sample chatbot on AWS Elastic Kubernetes Service (EKS) and implementing a comprehensive monitoring solution using Prometheus, Grafana, and the ELK stack. @Q We'll explore how users can access the web application and how a DevOps engineer can monitor real-time application logs. For this purpose, we'll use a sample chatbot application built using HTML, CSS, and JavaScript, integrated with Google Dialogflow for natural language understanding. The chatbot will be deployed on AWS EKS, a managed Kubernetes service that provides scalability and reliability. By the end of this guide, you'll have a fully functional chatbot application running in a containerized environment and a robust monitoring solution in place to ensure its optimal performance. #57 Overview [7 To achieve comprehensive monitoring for an AWS EKS (Elastic Kubernetes Service) cluster, we will leverage open-source tools such as Prometheus, Grafana, and the EFK stack (Elasticsearch, Fluentd, Kibana). Each tool plays a crucial role in the monitoring setup: [i] Prometheus: It collects and stores metrics from various sources within the cluster, allowing us to gain insights into resource utilization, application performance, and more. Z Grafana: It provides a powerful visualization platform to create interactive dashboards using the data cflJected by Prometheus. bitpssdevo hashnode. devlcomprehensive-aws-eks.clustor-monloing-with-prometheus-gralanaand-afkstack- 1Oweeksofcloudops 3148‘14124, 4:10PM Comprehensive AWS EKS Cluster Monitoring with Promethous, Grafana.and EFK stack{il- 1Oweeksofeleudops Grafana enables us to monitor the health and performance of our chatbot application in real time. Q EFK Stack: This combination of Elasticsearch, Fluentd, and Kibana facilitates centralized logging. Elasticsearch stores and indexes logs, Fluentd collects and forwards logs to Elasticsearch, and Kibana offers a user-friendly interface for log exploration and analysis. By incorporating these tools into our monitoring setup, we can effectively monitor and troubleshoot our AWS EKS cluster, ensuring smooth operation and a delightful user experience for our chatbot application. @& ml Table of Contents |} + Prerequisites [] * Setting up the AWS EKS Management Host * Creating an IAM Role and Attaching it to the EKS Management Host * Creating the EKS Cluster using eksctl * Deploying a sample application for Logs Monitoring using Prometheus and Grafana * Monitoring CPU, Memory, Disk, and Error Codes Jali [iI * Setting up Metrics Alerts AQ * Deploying the EFK Stack for Log monifeting using manifest files @L7 bitpssdevo hashnode. devlcomprehensive-aws-eks.clustor-monloing-with-prometheus-gralanaand-afkstack-1Oweeksofcloudops 4188‘14124, 4:10PM Comprehensive AWS EKS Cluster Monitoring with Promethous, Grafana.and EFK stack{il- 1Oweeksofeleudops + Streaming Logs to EFK @ & Architecture Diagram Step 1 Prerequisites Before getting started, make sure you have an AWS account, basic knowledge of AWS, Kubernetes, EKS and the necessary permissions to create resources on AWS. Step 2: 1. Creating an EKS Management Host on AWS Before you begin, ensure that you have an AWS account, a basic understanding of AWS, Kubernetes, EKS, and the required permissions to create resources within AWS. « Launch new Ubuntu VM using AWS EC2 (t2.micro). + Connect to machine using SSH or preferred method. bitpssidevo hashnode. devlcomprehensive-aws.eks.clustor-monloring-with-prometheus-gralanaand-afkstack-1Oweeksofcloudops 5148‘14124, 4:10 PM Comprehensive AWS EKS Cluster Monitoring with Prometheus, Grafana.and EFK stack{il- 1Oweeksofeleudops 2. Connect to machine and install kubect! using below commands. local, 3. Install AWS CLI latest version using below commands. cd “https: //awscli.amazonaws .com/awscli-exe-Linux-x86_64.zip" 4. Install eksctl using below commands. “https: / gp thub. com/weaveworks/eksct1/re local, bitpssdevo hashnode. devlcomprehensive-aws-eks.clustor-monloing-with-prometheus-gralanaand-afkstack-1Oweeksofcloudops ess‘14124, 4:10PM Comprehensive AWS EKS Cluster Monitoring with Promethous, Grafana.and EFK stack{il- 1Oweeksofeleudops Step - 3: * Create New Role using IAM service ( Select Usecase - ec2 ) « Add below permissions for the role IAM - fullaccess VPC - fullaccess EC2 - fullaccess CloudFomration - fullaccess Administrator - acces Enter Role Name (eksroleec2) « Attach created role to EKS Management Host (Select EC2 > Click ‘on Security + Modify IAM Role > attach IAM role we have created). Step - 4: Create EKS Cluster using eksctl. ‘Syntax: eksctl create cluster --name cluster-name --region region-name --node-type instance-type --nodes-min 2 ~-nodes-max 2 \ --zones Q bitpssdevo hashnode. devlcomprehensive-aws-eks.clustor-monloing-with-prometheus-gralanaand-afkstack-1Oweeksofcloudops 7188‘14124, 4:10PM Comprehensive AWS EKS Cluster Monitoring with Promethous, Grafana.and EFK stack{il- 1Oweeksofeleudops Choose your preferred name and region.Here | used name as bot and N.Virgina region. Note:The cluster setup will take around 5-10 mins.After done with the setup.We can check the nodes status using below command. ues ees cesta Gea ~ SRC Cecracrys eo : Smee rac races trys avr s | We cannot see the control plane here, as it is managed by the Elastic Kubernetes Service (EKS). Once the setup is complete, you will see the console displayed like this: bitpssdevo hashnode. devlcomprehensive-aws-eks.clustor-monloing-with-prometheus-gralanaand-afkstack- 1Oweeksofcloudops e148‘14124, 4:10PM Comprehensive AWS EKS Cluster Monitoring with Promethous, Grafana.and EFK stack{il- 1Oweeksofeleudops Now in console we can see that EKS client VM in order to communicate with Control plane and also we can see the two worker nodes up and running. bot-clustera Deploying the Sample Bot Application | have deployed a sample bot application using yaml file that leverages HTML, CSS, JavaScript, and Google Dialogflow. To ensure efficient resource monitoring and performance analysis, Prometheus and Grafana have been implemented to monitor the cluster resources and gather valuable insights. Q bitpssdevo hashnode. devlcomprehensive-aws-eks.clustor-monloing-with-prometheus-gralanaand-afkstack-1Oweeksofcloudops 9148‘14726, 4:10 PM Comprehensive AWS EKS Cluster Monitoring with Promethous, Grafana.and EFK stack{il- 1Oweeksofeleudops The provided above YAML configuration represents a Kubernetes Deployment and Service for the sample bot application. It deploys two replicas of the bot application and exposes it externally through a LoadBalancer Service on port 8080. Access the application through loadbalancer url http://af07c2eae913e4f28a275d33ec10f0a8-1581982861.us-east- 1.elb.amazonaws.com:8080, a Step 5: bitpssdevo hashnode. devlcomprehensive-aws-eks.clustor-monloing-with-prometheus-gralanaand-afkstack-1Oweeksofcloudops 10188‘14124, 4:10PM Comprehensive AWS EKS Cluster Monitoring with Promethous, Grafana.and EFK stack{il- 1Oweeksofeleudops Setting up Prometheus and Garafana using Helm charts. To set up Prometheus and Grafana, we will utilize Heim, a Kubernetes package manager that simplifies the installation of various Kubernetes plugins. Helm provides predefined configurations called Helm charts, which streamline the deployment process and minimize the potential for human error. Instead of manually creating multiple services and files in the correct order, Helm allows us to use a straightforward command for installation. Helm Installation HEH HH HH > check do we have metrics server on the cluster. Q bitpssdevo hashnode. devlcomprehensive-aws-eks.clustor-monloing-with-prometheus-gralanaand-afkstack- 1Oweeksofcloudops 8‘14124, 4:10PM Comprehensive AWS EKS Cluster Monitoring with Promethous, Grafana.and EFK stack{il- 1Oweeksofeleudops # Before you can install the chart you will need to add the metrics- server repo to helm. # Install the chart After adding metric server: $ kubectl top po meres) eee eee eet in ry an aT ten et aa im Soest Py im Cees fitter ey MEMORY(bytes) PEMORY® pry a Deploy Prometheus and Grafana in a Kubernetes Cluster using HELM Charts. Q bitpssidevo hashnode. devlcomprehensive-aws.eks.clustor-monloring-with-prometheus-gralanaand-afkstack-1Oweeksofcloudops 12884,410 °M Comprehensive AMS EKS Clister Monitoring wih Promothes, Grafana.and EFK sacki- Owecksofcloudops To get started, follow the steps below to install Prometheus and Grafana using HELM: * Add the latest HELM repository in Kubernetes: + Add the Prometheus repository to HELM: + Update the helm repo: $helm * Install Prometheus using the prometheus-community Helm chart: * Verify that the Prometheus pods argyunning: titpssdevo hashnode. devlcomprehensive-aws-eks.clustor-monloing-with-prometheus-gralanaand-afkstack- 1Oweeksofcloudops 13148‘14726, 4:10 PM Comprehensive AWS EKS Cluster Monitoring with Promethous, Grafana.and EFK stack{il- 1Oweeksofeleudops Perna Ars aan Cree} cores) * Edit the Prometheus service and change the service type to LoadBalancer: * Repeat the previous step to edit the Grafana service and change the service type to LoadBalancer: Verify that the services have changed to LoadBalancer: Q bitpssdevo hashnode. devlcomprehensive-aws.oks.clustor-monloing-with-prometheus-gralanaand-afkstack- 1Oweeksofcloudops alas‘14124, 4:10PM Comprehensive AWS EKS Cluster Monitoring with Promethous, Grafana.and EFK stack{il- 1Oweeksofeleudops Now, you can access Prometheus and Grafana using the LoadBalancer URL: Prometheus URL: http://LBR-URL:9090/ Grafana URL: http://LBR-URL/ Prometheus UI: Oto a itpssdevo hashnode. devlcomprehensive-aws-eks.clustor-monloing-with-prometheus-gralanaand-afkstack-1Oweeksofcloudops 15148‘14124, 4:10PM Comprehensive AWS EKS Cluster Monitoring with Promethous, Grafana.and EFK stack{il- 1Oweeksofeleudops Grafana UI: PaNECIELEy Login into grafana dashboard using default username and password: Username: admin Password: prom-operator Once logged in, you can start monitoring your Kubernetes cluster using Grafana’s user-friendly interface. Grafana will provide you with insightful graphs and metrics for analysis and optimization. Step 6: Monitoring CPU, Memory, Disk, and Error Codes. Before we dive into monitoring CPU, memory, disk, and error codes in a Kubernetes cluster using tools like Prometheus and Grafana, let's first understand the concept of Kubernetes monitoring and the architecture Q of Prometheus,and grafana features. bitpssdevo hashnode. devlcomprehensive-aws-eks.clustor-monloing-with-prometheus-gralanaand-afkstack-1Oweeksofcloudops 16188suze 410 °M Comorensve ANS EXS Chister Mortng wih Prometheus, Grafngand EK wack TOwesksfcoudope Kubernetes monitoring is a critical process that involves examining and reporting the health status of various components within a Kubernetes cluster. It helps track the utilization of cluster resources such as memory, CPU, and storage, providing valuable insights into the overall performance and availability of the cluster. Kubernetes Monitoring Master Nodes Nodes ol cAdvisor Kubernetes brings challenges in managing high availability, performance, and container deployment. Monitoring these components becomes crucial. Fortunately, Kubernetes provides metrics through CAdvisor, which collects CPU, memory, network, and disk usage. These metrics are then pushed to the Metric Server using kubelet APIs. While Kubernetes simplifies containerized application management, it introduces additional layers and abstractions, leading to a complex monitoring landscape. However, we have a range of excellent open- source tools available today to assist us in this endeavor. Q bitpssidevo hashnode. devlcomprehensive-aws.eks.clustor-monloring-with-prometheus-gralanaand-afkstack-1Oweeksofcloudops iss‘14124, 4:10PM Comprehensive AWS EKS Cluster Monitoring with Promethous, Grafana.and EFK stack{il- 1Oweeksofeleudops Prometheus Architecture Service Decay, Cesterie —agersuty notes Notations ‘Artmanager © Push Ales Prometheus Pon Server Data Stores Data Visualization Eisretenes) PuilMetries Application se cs * Prometheus Server is the main component of Prometheus Local Storage responsible for monitoring. « Itincludes a Time Series Database for storing metric data such as CPU and memory usage. * A Data Retrieval Worker pulls data from applications, services, servers, etc., and pushes it into the database. * The Server API accepts queries for the stored data, allowing for data display in dashboards or web Uls. Why use Prometheus for Kubernetes monitoring@ @?? Qa Prometheus has become the preferred choice for Kubernetes monitoring due to the following reasons: bitpssidevo hashnode. devlcomprehensive-aws.eks.clustor-monloring-with-prometheus-gralanaand-afkstack-1Oweeksofcloudops 18188‘14124, 4:10PM Comprehensive AWS EKS Cluster Monitoring with Promethous, Grafana.and EFK stack{il- 1Oweeksofeleudops © [iJ Mutt-dimensional data model: Prometheus employs a flexible key-value pair data model, similar to Kubernetes labels, allowing precise and flexible time series data querying. * @ Accessible format and protocols: Prometheus metrics are published using a straightforward HTTP transport, making them easily accessible and human-readable with self-explanatory formats. « ® Service discovery: Prometheus server automates the process of scraping metrics from targets, eliminating the need for applications and services to produce data actively. It can automatically discover and scrape targets, which is ideal for dynamic Kubernetes workloads. © & Modular and high availability: Prometheus is built with modular components that handle tasks like metric gathering, alerting, and visualization. These components are designed to support sharding and redundancy, ensuring high availability. With these features, Prometheus simplifies Kubernetes monitoring, providing a scalable and flexible solution to capture and analyze metrics in containerized environments How does Prometheus work with Kubernetes? Q bitpssdevo hashnode. devlcomprehensive-aws.oks.clustor-monloing-with-prometheus-gralanaand-afkstack- 1Oweeksofcloudops 19148‘14124, 4:10 PM Comprehensive AWS EKS Cluster Monitoring with Promethous, Grafana.and EFK stack{iJ- 1Oweeksofeleudops [i] Prometheus on Kubernetes monitors various targets, including Linux/Windows servers, applications, Apache servers, and databases. These are referred to as "Targets". Q It collects metrics such as CPU usage, memory usage, request counts, and exceptions. These metrics are referred to as "Metrics". @ Prometheus categorizes metrics into three types: * Counter: Tracks the number of occurrences of a specific event, such as exceptions or requests. « Gauge: Represents a metric that can fluctuate, providing current values for CPU usage, disk usage, etc. * Histogram: Monitors the duration or size of requests or any other required metric. £8 Prometheus pulls metric data from targets via HTTP endpoints (/metrics). If a target doesn't expose this endpoint, exporters come into -ntps:ildevo hashnode devicomprehensive-aws-ks-cluster- monitoring. with promethous-grafanaand-efk-stack-tOwoeksofeloudops 20188