Visualpath ea Decoding DevOps A practical guide for DevOps implementation By Imran TeliDecoding DevOps “A practical guide for DevOps Implementation” 5 Your personal toolkit for DevOps tasks. , by Imran Teli Co-Authors > Gayathri Kilaru (DevOps Engineer) > Kiran P (DevOps Engineer) > Waheed Khan (Full Stack Developer) Feedback Email: [email protected] Author’s Email: [email protected] VisualPath sualpath. Flat no: 205, 2nd Floor, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad-16 Phone No : +91 - 970.445 5959, 961 824 5689 E-Mail ID : [email protected] Visualpath Training & Consulting, Flat no: 205, Nilgiri Block, Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.|. DEVOPS INTRODUCTION. 1. WHAT 5 DEVOPS?. srs ol 2. Joa RoLes & THER MOTTO, 2 '3, SOFTWARE DEVELOPMENT PROCESSES. oo 4, THe PROBLEM: 6 25. Ener mie DevOrs, au * 6. DevOrs LFeCvcte.. 18 7. DevOrs avo SoFTwae DevetorMenT LE Cre f a 8. Toots FoR DevOPs LiFecvcLe 9. DevOPs LFEC¥CLE WITH IMAGES OF DEVOPS TOOLS. I, BASH SCRIPTING. Linrrooucrion. 26 2.VARIABLES: eee eeeeey ae 2B 3. User INPUT f a f.-- R 34 4. Ie STATEMENTS. 36 5. Loorst 40 6. REALTIME USE CASES... rd 7. How To Ser UP SSH Keys. 49 £8. Few SAMPLE SCRIPTS... 52 MLVIRTUALIZATION 1, LiFe WITHOUT VITUAUZATION 87 2. ENTER THE VMWaR 58 23.THE VIRTUAL MACHINE... _ 58 4. Key PROPERTIES OF VIRTUAL MACHINES... 58 St tire mnsons Seem 59 16. YOUR CHOICE OF HYPERVISOR nnn 60 7.SOME TERMINOLOGIES es 8. INSTALLING VIRTUALBOX ON WINDOWS. ‘ ea ‘9. INSTALLING OS ON THE NEWLY CREATED VM. 69 IV, INTRODUCTION TO VAGRANT... 1. Way VaGRANT?, BB 2. NSTALUNG VAGRANT ON WiNDOWS. 4 3.G9r Bash CU 75 4.Macrant Coup, ) 75 5. VAGRANTFILE 76 6. CREATING CenTOs & UBUNTUVM's USING VAGRANT TOOL n ‘7. CREATING A VAGRANTFILE 78 £8. FIRST LOOK AT A VAGRANTEILE 9 9. VAGRANT UP, 79 10, VAGRANT SSH. BL LLVersicarion, 22 12, TEARDOWN sn 83 113, VAGRANT BOX COMMANOS.. 8S 14, USEFUL VAGRANFLE SETTINGS. 86 15, BRIDGE NETWORKING... 87 16, PROVISIONING en 90 7, EXTERNAL SCRIPT soe 2 Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 59: Mail ID : [email protected], Website : www.visualpath.in. 961 824 5689 E-18, SYNCED FoLorRs.. 82 19. Dertunc MuLTPLe MACHINE. 8B 20, CONTROLUNG MULTIPLE MACHINES... 94 21, FORWARDED PORTS . 98 22, FEW SAMPLE VAGRANTFILE'.. 94 \V. AMAZON WEB SERVICES 11. WHATIS CLOUD COMPUTING? 100 * 2. Coup Service Mopets: 101 3, ADVANTAGE AND DISADVANTAGES OF CLOUD COMPUTING. 102 4, WHAT 1s AWS? _ 108 5, REGIONS ANO AVALABLITY ZONE.. 103 16. AWS Seavics. sv 106 7.1AM 106 8. AMAZON EC2 114 9. AMAZON EBS 7 / . 127 10. AWS VPC. 140 111, CREATING HIGHLY AVAILABLE VPC. 144 412, ELASTIC LOAD BALANCER. 154 13, AWS AUTO SCALING. 167 114, AWS CLouD Wares ns 190 15, AWS ELASTIC BEANSTALK + JENKINS 203 116. AMAZON S3 (SIMPLE STORAGE SERVICE). 22 17, AWS RDS... 226 18, ROUTES3 231 19, AWS CLI 239 20, $3cm0 248 21, SBEMO-SYNC sn 253 22. MOUNT NW AMAZON S3 BUCKET TOA LOCAL LNUX FE SYSTEM 255 VIPYTHON SCRIPTING. .. 1. PYTHON INTRODUCTION... 262 2. Basic SyNTAx 263 23, VARIABLE TYPES. 267 4, PTHON OPERATORS... 27 5. DECISION MAKING... 276 6. Loors. Lenn 280 7. BUILT IN MeTHO0s/FUNCTIONS. 286 ‘8: FuNcTIONS. J 288, 9, MODUL... 4 294 10, PYTHON FoR OS TASK. 296 111 FABRIC FOR AUTOMATION 298 12, BOTO FORAWS -ceo 307 VII. WEB APPLICATION ARCHITECTURE, 1. WEB APPLICATION ARCHITECTURE 1 316 2. THeCueNT. 316 3. LOAD BALANCING: 316 4. LOAD BALANCER. 317 5. LOAD BALANCING AtGoniTHMs. 317 6. DYNAMIC CONFIGURATION OF SeRVER GROUPS. 317 7. HARDWARE VS. SOFTWARE LOAD BALANCING 317 Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 59: Mail ID : [email protected], Website : www.visualpath.in. 961 824 5689 E-8. _NGINX/ NGINX PLus As Loao BaLANcER.. 318 9. APACHEHTTP SERVER 320 10, TOMCAT SERVER 320 11, Darasase(MySQU) 321 12. WEB APPLICATION ARCHITECTURE 2. 324 13, Wes Browser, 324 14, SETUP FOR NGINX AS LoD BALANCER 325 +15. 8085 325, "16. Rass, 7 327 17, Memexacieo 328 18, WEB APPLICATION ARCHITECTURE 3 330 19, _ NooelS Senven: 20. ELASTISEARCH, 21, MoncoDB. Vill, SOFTWARE DEVELOPMENT 1. SOFTWARE DeveLopmenr Process. 339 2. SoFTwaRe Devetorment Lie Crcte (SDLC). 1K. VERSION CONTROL SYSTEMS. 1. Whew rouse VCS. — od 345 2. VCS TenMinoLosies. A > y 346 3. FAMOUS VERSION ConTROt SysTEMS. 7 307 4. WaarisGir 349 5. Wav use cn? 7 7 349 6 — GirQuiexserur. 350 7. Gmrmperan. 355 8 INSTAWNG Grr... 355 9. SernING UPA REPOSITORY 357 10, SAVING CHANGES.. 361 LL, THESTAGINGAREA 362 12, SYNCING vain 364 13, RepostToRy URLS. 365 14, GrrrercH., 366 15, GmrPuL o 368 16. PULINGVIAREBAE. 368 17, GITPUSHs o 369 18, Girhus SSH LosIw 370 19). GIT CueaTSueeT ROM ATLA 373, X. MAVEN.. 1. Butlo process 379 2. ANTS MAVEN Vs GRAOLE. 380 3. UNDERSTANDING THE COMMON PROBLEM WITHOUT MAVEN. 381 4, Warr o0682 381 5. WHATIS BUILD Toot. aoe 6. USES OF APACHE Maven a 7. SETUP AND INSTALLATION FOR MAVEN. 382 8. FinstSawpie APPLICATION: 385 9. Maven roM.xm rte 385 10, Maven - Buin LireCvcte. XI, CONTINUOUS INTEGRATION.. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.1. INTEGRATIONS PAINFUL. 390 2. SOMETERMINOLOGIES BEFORE WE BEGIN, 390 3. WHATIS CONTINUOUS INTEGRATION. 391 4, WHS JENKINS. 393 5. FEATURES OF JENKINS sn 394 6 Jenkins Sevur, 395 7. CREATING FIRST JENKINS 8... 400 8, SETUPAJAVA BUILD JOB WITH MAVEN. 403 "9. Jennins ApMinIsTRATION. 407 10. ConTINUoUS INTEGRATION PROLECT. 417 11, _ STATIC CODE ANALYSIS FOR GaMe OF UFE DEV PROLECT. 423 12. ConmINUoUS DeLVERY WITH JENKIN 28 13, Jenkins BUILD TrisgeRs 433 114, SONARQUBE INTEGRATION JENKINS. 35, 415, _ARTFACTORY 449 16. Wiersiouto use cenTER OVER MAVEN CENTRAL FOR DOWNLOADING DEPENDENCE? 452 417, JFROG INTEGRATION WITH JENKINS, 1 A86 XII, ANSIBLE TUTORIAL ..neemnsnntninnennnstinmniinnnnnnnnnnnnnnssnnsnne ne 86 1. CONFIGURATION MaNacemenr. 436 2. SOME TERMINOLOGIES, 87 3. ANSIBLE INTRODUCTION, 490 4, INSTALLING ANSIELE 490 5. INVENTORY snes 891 6. _IwvenToRy Fox PRODUCTION SYSTEMS REALTIME. 492 7. AoHoc COMMAND. 493 8 ABOUTMODULES.... 496 9, ANSIBLE CONFIGURATION svn 499 10, SETUP- GATHERS FACTS ABOUT REMOTE HOSTS 499 11, Puavoooxs 501 12. PLaveoox LancuAce EXAMPLE. ae 501 13, YAML BASICS ain 502 14, First PLsyeoox EXER... 503, 15, PLAYBOOK EXECUTION... * 507 16. Vaniaates 510 17, IwcwoINs PuavBooKs 513 18, STORE OUTPUT OF A COMMAND. 513 19, Deaue MODULE 514 20. PRoMPTING FOR INPUT. 0515, 21. HANDLERS. a 516 22, CONDTONAL EXECUTION. S17 23, TEMPLATES... 1518 2. A sae PLAYEOOK WH VARAGLES, TEMPLATES, CONOTTONS AND HANDLERS 519 25, ROLES... 522 26, ANSIBLE GALAXY. 526 27, ANSBLE VAULT MANAGING SECRETS WTH ANSLE VAULT. 528 28, LOOPING IN ANSIBLE 530 29, CONFIGURE APACHE USING ANSIBLE. 531 30, MULT Tek Wes APPUCATION STACK DEPLOYMENT USING ANSE ne 542, 31, WoROPRESS SETUP WITH ANSIBLE S71 Xill. PUPPET. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 59: Mail ID : [email protected], Website : www.visualpath.in. 961 824 5689 E-1. WHATS PUPPET?... 2. INSTALLING PUPPET .. 3. CONFIGURE PUPPET MASTER 4, SSL SiGN Cenmircare SETUP 5. CREATING ENVIRONMENTS. 6. Ser. 7.Mooutes. +8, Manirests, "9. cussses: 10. VALIDATING THE SYNTAX OF MANIFEST Fs) 111, APPLYING MODULES ON PUPPET AGENT. 412. CONFIGURING THE RUN INTERVAL, 13, Deep Dive ivro PuPrer ConINs. 14, Conomionats 15, Case STATEMENTS. 16, SeLEcTORS. ee 117, RELATIONSHIPS AND ORDERING... 18, ERB TEMPLATES. 19, ITERATING OVER VALUES 20, INTRODUCING HiERA 21, RESOURCE TYPES snnonnnennn 22. THE PUPPET EXCERCISE XIV. DOCKERS... 1. APPLICATIONS ERA. . 629 2. VIRTUALIZATION REVOLUTION, os -630 ‘3, PROBLEMS WITH HYPERVISOR ARCHITECTURE... 631 4, CONTAINERS. 631 5. Dockers. 632 6. INSTALLING DocKER. — 634 7. DOCKER ENGINE'S BIG PICTURE nn ae 636 8. IMAGES. 639 9. CONTAINERS... 648, 10, BuiLoIns & SHIPPING ImaGes * 651 111, CONTAINER NETWORKING Basics. 663, 12. THe Container Network Moot. 667 113, LOCAL DEVELOPMENT WorkFLOW WITH DocxER. 674 14, Using Docker ComPase FoR DEVELOPMENT STACKS. 679 XV, KUBERNETES 1 KuBeRNeTEs IntRoDUCTION: 685, 2. WY KUBERNETES?, 686 23, WHAT IS KUBERNETES?... 687 4, Wat KuBeRNeTEs CaN 002. ess, ‘5, KUBERNETES ARCHITECTURE 689 66, KUBERNETES SETUP, sa se 692 ‘7. KUBERNETES DETAILED SETUP & EXERCISES. 692 ‘8, MINIKUBE SETUP LOCALLY 692 8. KOPS ns 695 10, CONTAINERS & IMAGES. 705 111 FIRST APP ON KUBERNETES. 708 12. Services. 710 Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 59: Mail ID : [email protected], Website : www.visualpath.in. 961 824 5689 E-13, NoDe ARCHITECTURE 14, Scauins Pos. 15, DEPLOYMENTS non 16, More sour Seavices, 7, LABELS. 18. Secners 19, WonoPress DePLovmenr. Xi. NAGIOS 4 NeTwork MONITORING... 2, SERVER MONITORING. 13, APPLICATION MONITORING... 4. Preneauisies 5. LAMP STACK Sc... 6. NAGios SeTUP 7. LOGIN TO NaGios. ' / >, 8. MONITOR A LINUX Host wiTH NRPE. 748, 9. Directives ey 748, 10, Monrronin a Winpows Host. XVI, MONIT. 1 SeTUP MoNiT AND CONFIGURE IT 10 MONITOR NGIvx SERVICE _ y 762 22. MONITORING SETUP FOR NGINX SERVICE. 7 763 XVIII, SL SELF-SIGNED CERTIFICATE FOR AWS ELASTIC LOAD BALANCER. L.Wuars SSL?, nnn o 2. List oF POPULAR SSL CenriFicare AUTHORITES{CA):, y >: 765 3, WHAT IS SELF-SIGNED CERTIFICATE? 765 4, How 1T Works? y. 766 ‘5.Wuar is HTTPS PRoTocol?.. . J 766 16. How 10 GENERATE SELF-SIGNED CERTIFICATE USING OPENSSL? (iN CENTOS) 768 ‘7.UPLOAD KEYS INTO AWS my 8. VERIFICATION. 7 773 9. ALGORITHM DiacRAM: 776 XIX. A SAMPLE CONTINUOUS DELIVERY PROJECT. 7” XX. A WORD ABOUT SECURITY .nneninnsnnnsntiinninniininninnnninnnininninnnninninnnnnssnncT 86 Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.About Author Imran Teli has been working as a DevOps Consultant and Architect for a quite a long time. Has a deep understanding of DevOps Processes and tools. Imran has worked in IT industry for 9 years with nearly 5 of those years spent in desigining and managing solutions in some of the most demanding corporate IT environments in the world, Imran has worked his way up from System Admin, all the way upto DevOps consultant for designing resilient, high performance and secured sblutions, as well as training people about building and managing such solutions. Have designed & implemented some of the most complex and mission critical Continous Delivery solutions. Very passionate about sharing the experience and knowledge through training in various technologies in DevOps domain, Current interests include Microservices and Information Security in DevOps domain. How to contact Author? ‘You can get in touch with author through Visualpath Trai domain. 1g & Consulting for Training in DevOps www.visualpath.in Flat no: 205, 2nd Floor, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad-16 Phone No : +91 - 970.445 5959, 961 824 5689 E-Mail ID : [email protected] Visualpath Training & Consulting, Flat no: 205, Nilgiri Block, Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.PREFACE Who should read this book? Decoding DevOps is written for anyone who wants to learn DevOps tools and techniques. Its main focus is in doing rather than reading That being said, Basic knowledge of Linux and networking is required to grasp concepts and techniques in this book. While its not necessary to read this book cover-to-cover, we make an attempt to cover the topics in logical order. That being said, here are few short-cuts: © Ifyou are a system admin and want to learn cri scripting. © If you create virtual machines on your laptop/desktop and want to automate your vm lifecycle, checkout Virtualization & Vagrant chapters ‘* Ifyou want to learn AWS SysOps services, check out Amazon Web Services chapter. ‘© If you are Developer and want to learn Build & Release and Continuous Integration, check out chapters Maven & Continuous Integration. * Ifyou want to learn new way of automation through Configuration Management tools, check out Ansible & Puppet chapters. ‘© If you want to learn containers and dockers for microservices, checkout Docker and Kubernetes chapters. ing, check out Bash scripting & Python What's in this Book? The book begins by explaining DevOps culture and process. It talks about the problems in delivering software to users and then explaining how DevOps culture solves this problem. First chapter also talks about DevOps lifecycle and the tools that should be used to create that lifecycle. Scripting with Bash and Python language is covered if you are interested in doing automation with scripts. Automating virtual machine lifecycle on your local system is covered which has become so much necessary in day to day practice in IT industry. Vagrant tool helps you doing it and you don’t have to waste your time in creating vm manually and installing OS on it. IT industry is slowly moving their infrastructure to Cloud and Amazon Web services talks about, how to use AWS services and create your own infrastructure on cloud. Here we are focusing on AWS sysops services that is used by Sys Admins and DevOps. AWS has lot many services and features for Developers and other DevOps practices which is out of scope for this book to cover. Web Application architecture, Maven build tool and Software Development process are covered. If you are coming from Sys admin background then you will understand how Development process happens. Also, if you are not aware of different kind of services that is used in Web application platform you will learn general Architectures and services for web apps. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block, Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.Jenkins tool in covered in detail which is the most famous continuous integration tool and also the ‘most important DevOps tool. Integration of Jenkins with various other tools is also covered. Learning Jenkins is important for Developers, Build & Release, Testers, Sys Admins & DevOps engineers ofcourse. Ansible and Puppet is covered as a part of configuration management tools tutorial. It is highly important as these are the most famous automation tool used by DevOps and Sys admins around the world. Lastly Dockers and Kubemetes is explained in detail with all the commands and implementation. Docker is highly important for Microservice architecture and is going to revolutionize the IT industry big time. Docker or containers is important to learn for both Developers and Operations team. Highly important for DevOps engineers How to contact us? We'd like to hear from you As you use this book and try exercises, we invite you to comment and feedback. We always take feedback positive or negative and learn from our mistakes or cherish positive feedbacks. We hope to continue to do this, This book was written quite quickly. There would be places where we would have made mistakes or errors, your feedback will help us make the next version better. It was not possible to cover everything in DevOps in one book but we have given our best shot. You can send us some tips, tools or techniques information which we will include in next version or in another book which covers DevSecOps, Microservices, Log management and some Real time POC in DevOps domain. Please let us know what we need to correct or add; share your insights; and help us create a resource that will server uou better. You can do so by sending email at feedback @visualpath.in Feedback Email: feedback @visualpath,in Author’s Email: [email protected] Acknowledgements I'd like to thank each one of them who helped and inspired me in writing this book. Some of my students have researched and documented scripts and tools implementation, I have included their name wherever their contributation is done in the book. My co-authors have put great deal of time and effort in creating documents on Various topics without them it was not possible to complete this book on time. I’d like to especially thank Eswar for thinking that I should write this book in the first place. Last but not the least my Wife Ziya Teli who spent some sleepless nights in formatting, designing book cover and doing spelichecks for this book because this book was converted from libreoffice to MS office. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block, Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.I. DevOps Introduction 1. What is DevOps? T know that's the first answer everybody would be looking for and I will not try to define it Because so many people have done it already before me and guess what there are many different definitions for it. Everybody in the IT industry is talking about it. Organization are running post to pillars to hire DevOps Engineers and there is a great demand for it. Some say its automation but some say it’s all about culture, both this belief contradicts with each other. Well if its automation then system admins are doing automation from ages and there they used Scripting languages and some tool to achieve it but we did not call it DevOps back then, we just said automation, So, then you would say it’s the culture but if it’s just the culture then why so many automation tools? You would be feeling now that I am confusing you but trust me I am going to prove some points later, just keep reading If you are reading this you would somehow related to software industry. Yes, software industry, its only goal is to make software's and deliver it to the user. We can say from the very beginning software industry is divided into two parts and those two parts are Development and Operations. Development focuses on creating and testing software's. Operations is focussed on delivering those software's to user in form of a website or as an installable software. Once delivered we maintain the software, we deliver new features to our users and make sure the software's stay up and running and healthy. 2. Job Roles & their Motto. In Development, we have distinct roles like Developers, Software testers(QA), DB Developers & Architects. Their aim is to develop all the latest and greatest features in the software's, Rapidly or quickly. In Operations, we have roles like System Admins, Cloud Engineers, DB admins & Security professionals. Here the aim is to Keep the systems up and running all the time. Systems on which the software is hosted, like your websites and databases hosted on some servers. ‘You would have understood by now that both the parties have different aims and goals. One focusses on Quick Change and other focusses on Stability. These are poles apart, if we make quick changes (adding new features continuously) then stability becomes an issue. A system that’s changes continuously will have issues with stability. It’s also true the other way around. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block, Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.We are living in a world where there are frequent changes to our software’s and apps. Think about it if you are using a software with some old features and some other software comes into the market with the latest & greatest feature you will ofcourse migrate to the new software. So that means if an organizations Dev and Ops team does not give you latest and greatest feature with stability they may lose the business. So far, I have established few points that I will list below. ‘© Developers aim to create latest features quickly and rapidly. * Operations aim is to keep systems stable. © Quick changes are the user demand © User also needs stable software or Apps. DevOps whole and sole objective is to Deliver latest and greatest features to the user with stability. It’s no more about just creating new features it’s also about delivering those features to the user otherwise what's the point of creating if we cannot deliver it on time. So how does DevOps solve that problem? To understand it first we need to understand the development procedure and then we will focus on the operations. 3. Software Development Processes Development process is explained in detail in later chapters here we will try to keep it to the minimum, Development team uses some Software development model to create the software. In layman terms, these software development models are set of rules that everybody in the team follows to get the things done. Waterfall Model ‘There is Waterfall model which is a traditional model and does not fit well in today’s fast-moving world In Waterfall model, the following phases are followed in order: L.System and software requirements: captured in a product requirements document 2.Analysis: resulting in models, schema, and business rules 3.Desi resulting in the software architecture 4.Coding: the development, proving, and integration of software 5.Testing: the systematic discovery and debugging of defects 6.Operations: the installation, migration, support, and maintenance of complete systems Visualpath Training & Consulting, Flat no: 205, Nilgiri Block, Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.ry rs oa Thus, the waterfall model maintains that one should move to a phase only when its preceding phase is reviewed and verified. This model is good for the Operations team as they get the whole software developed at once which they can deploy and maintain, New changes will also be less frequent and does not put so much burden on the Operations team, But this model does not Scale well with the current fast-moving world. There are so many downfalls with this model and most of the Development is happening with Agile model now. Agile Model Agile model developed software in small iterations instead of developing entire software at once. Entire products feature list is divided into multiple list of features. For example, if there are 50 features in a software we can create 5 lists of 10 features each. Now developers will work on 10 features at a time, create and deliver those 10 features in first iteration and continue with the rest of the features until you get the final product, Every iteration involves cross functional teams working simultaneously on various areas like ~ Planning ‘© Requirements Analysis * Design * Coding © Unit Testing and * Acceptance Testing. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.Iteration 1 Iteration 2 Iteration 3 Iteration N Now we are not just talking about creating but also delivering it to different environments like Dev, QA, Staging, UAT and Prod. Now this puts lot of burden on the Operations team as they must continuously deliver these changes on multiple environments. By the way these different environments are just some group of servers owned by different teams like QA is owned by software testers where they test the software, The general approach that Developers take is that once they are done creating a new feature they will send a procedural document to the operations team explaining how to deploy it. Development vs Operations 2 Agile = ee SysAdmins Support Developers test it on their machine and feels that it should work same in production also, But production systems are different in design as there would be multiple servers for webservice, database service and backend services secured by firewalls and NACL. There would be all stable OS and software with a different version that of Developers systems. Development servers are simple one machine and all the services are generally deployed in one server. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.ala Operations 4. The Problem After following the procedural document and having their own skills and knowledge operations will deploy it to production, This is where the problem comes in, the deployment may fail, failing the entire service. This happened because of lack of communication between Dev and Ops team. Dev does not understand Ops part and the reverse is also true. So, Ops feel that frequent changes like that may break the system and Dev feels that there is too much restriction on delivering latest changes. We also need to think about security here. Security testing is done before it goes to production. This entire Delivery procedure is slow and manual most of the time. » Developer always looks for changes They By to implement every new techniques introduced o e » Change is the enemy for Operations > itis not reliable and leads to Development. instabty Think about quick changes now through agile model, it’s not helping the operations team to deliver the code faster. So, no matter how agile development is Operations is still Waterfall. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.If you think about it for a while you will understand that it’s not the technological problem but the culture problem, Both the parts in Software Industry follow different culture. If this culture is not changed then we will not be able to deliver better feature quickly to the users. A Very famous joke describes the communication problem. How the customer | [How the Project How the Analyst How the Programmer | [How the Business fexolaned Leader understood t_} [desioneat wrcte [Consult described t nat operations | How t was supported states 5. Enter the DevOps DevOps solves this problem by changing the culture and making it one culture Dev+Ops. There would be one team, DevOps, with one goal, Rapid delivery with stability. But how? First step is to establish communication and collaboration between Dev and Ops. Dev must understand the Ops part and Ops must understand the Development Procedure. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.Eee j Agile Dev + (T Ops = DevOps! DevOps is the practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support. We have seen previously in waterfall and agile lifecycle, that Development and Operations team are separate, they work separately in their own silos and have very different motto. 6. DevOps LifeCycle DevOps Lifecycle includes Development and Operations teams working together. As Developers ‘work on their agile iterations, Ops must work in setting up systems and automating the procedure of deployment. Automation is the key factor here, because agile model gives code repeatedly to deploy it on systems, that’s going to be continuous release of code and that must be continuously deployed to many servers in Dev, QA, Staging & Production environments. If the code deployment process is not automated then ops team must manually do the deployment. Deployment may include below mentioned procedures. © Greate servers if they don’t exist (On cloud or virtual env). ‘* Install and setup prerequisites or dependencies on servers. * Build the software from raw source code (If not done by Developers). * Deploy software to servers. © Do config changes to OS and software. + Setup Monitoring. © Feedback & Report. Note: All the above process may be less or more depending on the kind of deployment. We will discuss this in later chapters. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.As soon as we get a new code change, it must be deployed to Production or at least staging servers. For this, all the process must be automated, we must automate first Build and Release process which includes. © Developers push the code in a centralised place. «Fetch the developers code. © Validate code. © Build & test code © Package it into distributable format(software/artifacts). © Release it Next phase is to deploy this released software to servers, which we discussed already before this. Combining this Build & Release with Deployment process gives us the DevOps Lifecycle which is fully automated. DEVOPS LIFE CYCLE x SS ass € \ Y Fetch Changes Y Run Unit Tests j ee Y eulantiats i aes N yf Y Provision environment i. Deploy Your Build eu ‘¥ Run Load & Functional Tests DevOps - Spanning across entire delivery pipeline Y Dev->QA-> Staging -> Production Contnwous negra | Continuous Dever) DevOps Engineers must automate all the above process it should be so seamless that when developers push their code to a central repository it should be fetched and run through all the above process and sends it to production systems. As I say, “From Code to Prod”. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.What is continuous Integration. Developers will push their code several times in a day to a central repository, every time there is code change it should be pulled, built, tested and notified. There will be continuous code change so continuously we must do these steps. That’s why its Continuous integration. We have a separate chapter for this where it will be discussed in detail. As of now you can understand from above diagram that from step 1 to step 5 is CI. & What is Continuous Delivery? After CI, we should be also able to deliver our code changes to all the servers in different environments like Dev, QA, Staging, It should be automatically delivered to QA servers where testers will do functional tests, load tests etc. After it passes the QA tests it should automatically deliver the code to staging area where customer or some set of users can check the changes and give approval to deploy it to Production, As per Wikipedia. Continuous delivery and DevOps are similar in their meanings and are often conflated, but they are two different concepts. DevOps has a broader scope, and centres around the cultural change, specifically the collaboration of the various teams involved in software delivery (developers, operations, quality assurance, management, etc.), as well as automating the processes in software delivery. Continuous delivery, on the other hand, is an approach to automate the delivery aspect, and focuses on bringing together different processes and executing them more quickly and more frequently. Thus, DevOps can be a product of continuous delivery, and CD flows directly into DevOps. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.What is Continuous Deployment? If the approval is manual process then code delivery is Continuous Delivery but if the approval process becomes automated then after staging, the code change is done directly to Production systems. This is called as Continuous Deployment. _ CONTINUOUS ? YEP IED ED ED ED E> (Continuous integration > Continuous Delivery > Continuous Deployment Devops 7. DevOps and Software Development Life Cycle ‘The DevOps Lifecycle Looks Like This: 1. Check in code 2. Pull code changes for build ¥ Run tests (continuous integration server to generate builds & arrange releases): Test individual models, run integration tests, and run user acceptance tests. Store artefacts and build repository (repository for storing artefacts, results & releases) Deploy and release (release automation product to deploy apps) Configure environment Update databases Update apps een ons Push to users — who receive tested app updates frequently and without interruption Visualpath Training & Consulting, Flat no: 205, Nilgii Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - #91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpathi.10. Application & Network Performance Monitoring (preventive safeguard) 11. Rinse and repeat The above process is also called a Code Delivery Pipeline. 8. Tools for DevOps Lifecycle We have discussed earlier that everything starts from communication and collaboration between Dev and Ops. Once we understand the culture and process of the project/product we can start ‘working with everyone in designing Code Delivery Pipeline. We must decide what automation tools to use to create entire pipeline, We need to decide where our infrastructure would be hosted? On the loud, virtual machines or physical machines? In today’s world we have lot of automation tools, but first we need to understand their categories what tools is used for what purpose? If we don’t understand that then we won’t be able to decide where to use them in our code delivery pipeline. Version Control Systems: Is used to store the source code, a central place to keep all the code and tracks its version. For Example: ° Git * SVN © Mercurial * TES Build tools: Build process is where we take the raw source code, test it and build it into a software, This process is automated by build tools. For Example * Maven * ANT + MSBuild © Gradle * NANT Continuous Integration Tools: Visualpath Training & Consulting, Flat no: 205, Nilgiri Block, Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.For example: * Jenkins © Circle CI © Hudson + © Bamboo © Teamcity Configuration Management tools: Also known as automation tools, can be used to automate system related tasks like software installation, service setup, file push/pull etc. Also used to automate cloud and virtual infrastructure. For example: © Ansible © Chef © Puppet * Saltstack Cloud computing: Well this is not any tool but a service accessed by users through internet. A service that provides us with compute resource to create virtual servers, virtual storage, networks etc. There are few providers in the market who gives us public cloud computing services. For example: « AWS © Azure © Google Cloud © Rackspace Monitoring tools: Is used to monitor our infrastructure and application health. It sends us notifications and reports through email or other means. For example: + Nagios © Sensu © Icinga © Zenoss © Monit Visualpath Training & Consulting, Flat no: 205, Nilgiri Block, Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.Containers & Microservices: Well to be very frank this cannot be described to you right now. Its described in detail in separate chapter. We need to have lot of Infra & Development knowledge to understand this category of tool. * Docker © RKT ‘© Kubernetes 9. DevOps Lifecycle with images of devops tools. Osi kubernetes Qvenkins ° checkstyle Nagios’ exe \ Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.DEVOPS . LIFE CYCLE y x SN v Push Code ® git i \ tt seg € ) \ ae ! Y Build Artifacts pce i Vammene NEE AN / Y Pr “S YY’ Deploy Your Build @ © tuverneres > axtmuc-GHEF ¥ Runoad&FunctionatTests GY / jeter” aiblmeenttnleneal Bovops - Spanning across entire delivery pipeline ¥ Dev -> QA -> Staging -> Production ‘Continuous Integration | Continuous Delivery Summary: ¥ DevOps cannot be defined in one sentence. DevOps is the culture and also the implementation of automation tools. It depends from which area you are defining it. Y Majority of the Software development is happening with Agile model and that chums out code changes incrementally and frequently. Y Operations does not gel well with the Agile team as both have differences in their principles. Y Agile team wants quick change, Ops wants to keep system stable by not making frequent changes. ¥ DevOps helps in creating communication, collaboration and integration between Dev and Ops and culture, practices and tools level. ¥ DevOps Engineer must understand DevOps lifecycle and implement right tool of automation at right place. Learning automation at every level in the lifecycle is highly important if you want to make a carrier in DevOps domain. You should understand Infrastructure, Development & Automation. Later in the book we will dig more in detail into tools and learn them. We will also understand CI & CD from tools point of view. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.II. Bash Scripting 1.Introduction This tutorial will give you a solid platform in how to create bash scripts and automate day to day system admin tasks. Certainly, everything cannot be covered in this chapter but you will be equipped with right amount of knowledge to make your own scripts and excel in it if you put in your own efforts. Bash scripting is used by many of the system admins and DevOps geeks to get things done quickly and efficiently. There are so many automation tools in the market like Ansible, Puppet, Chef etc. Which are way more sophisticated but sometimes to get things done quickly in Linux systems we use Bash scripts. Also, scripting will make you understand what automation means and then you can quickly grasp the features that is used in Configuration Management tools like Ansible or puppet. What are scripts? A Bash script is a plain text file which contains a series of commands. These commands are a mixture of commands we would normally type ourselves on the command line (such as Is or ¢p for example) and commands we could type on the command line but generally wouldn't (you'll discover these over the next few pages). A crucial point to remember though is: Anything you can run normally on the command line can be put into a script and it will do exactly the same thing. Similarly, anything you can put into a script can also be run normally on the command line and it will do exactly the same thing. First Script As we discussed eatlier that script is a normal text file with commands in it. We will open a file vi editor and add some commands in it. Ic is convention to give files that are Bash scripts an extension of .sh (print.sh for example) $ Ai Brint.sh ‘echo Hello World! Explanation: Line 1 - #1 is called as the SHEBANG character, it tells the script to interpret the rest of the lines with an Interpreter /bin/bash. So, if we change that to /ust/bin/python then it tells the script to use python interpreter. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block, Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.Line 2- This is a comment. Anything after # is not executed. It is for our reference only. Comments is for us so oF anybody who reads this script will have some reference. ine 3- Is the command echo which will print a message to the screen. You can type this command yourself on the command line and it will behave exactly the same. Running or Executing a script? Running a Bash script is fairly easy. Sometimes you will hear people saying execute the script, both means same thing. Before we can execute a script, it must have the execute permission set. If you forget to grant this permission before running the script you'll just get an error message “Permission denied”. Whenever you create a file in Linux system by default it will not have an execute permission, this is for security reasons. You make your script executable and then you can nun it imran@Devops:.../bash$ ./print.sh bash: ./print.sh: Permission denied imran@DevOps: bash$ 1s 1 total 4 -rw-zw-r-- 1 imran imran 53 et/2117:33 print. sh imean@Devop: Moash$ chnod 755 print. imran@Devop: /bash$ 1s -1 total 4 -rwxr-xr-x 1 imrangimran 53,0ct 21 17:33 print.sh imran@Devops:..«/bash$.../print.sh Hello World! Without giving execute permission also we can run the script but then we provide a shell and ask it to run all the command in the scripton that shell. imran @Devop: /ash§ bastwprint.sh Hello World! Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 59: Mail ID : [email protected], Website : www.visualpath.in.2.Variables Temporary stores of information in memory. How do they Work? A variable is a temporary store for a piece of information. There are two actions we may perform for variables: Setting a value for a variable. Reading or using the value for a variable. To assign a variable we use = sign, VariableName=Value To read/access the value of variable we use $VariableName imran@Devops:.../bash$ VARI=123 imran@Devops:.../bash$ echo $VARI 123 Command line arguments When we run a program on the command line you would be familiar with supplying arguments after it to control its behaviour. For instance we could run the command Is 4 /tmp. -I and /tmp are both command line arguments to the command ks, ‘We can do similar with our bash scripts. To do this we use the variables $1 to represent the first command line argument, $2 to represent the second command line argument and so on. These are automatically set by the system when we'Fun our script so alll we need to do is refer to them, Let's look at an example, copyseript.sh - #!/bin/bash .# A simple copy S8zipt ep $1 $2 # Let's verify the copy worked echo Details for $2 1s -1h $2 Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 59: Mail ID : [email protected], Website : www.visualpath.in.imran@Devops:.../testcopy$ mkdir dirl dir? imran@Devops /testeopy$ touch dirl/tesla imran@Devops -/testcopy$ ./copyscript.sh dirl/tesla dir2/ Details for dir2/ total 0 Ew-rw-r-- 1 imran imran 0 Jun 19 23:01 tesla Explanation: Line 3 -run the command ep with the first command liné argument as the source and the second command line argument as the destination, Line 5 - run the command echo to print a message. Line 6 - After the copy has completed, run the command Is for the destination just to verify it worked. ‘We have included the options Ito show us extra information and h to make the size human readable so ‘we may verify it copied correctly. Some System Variables ‘There are a few other variables that the system sets for you to use as well, $0-The name of the Bash script. $1- $9 - The first 9 arguments to the Bash script. (As mentioned above.) S#- How many arguments were passed to the Bash script. $@ - All the arguments supplied to the Bash script. $2 ~The exit status of the most recently run process. $5 - The process ID of the current Setip. SUSER - The username of the user running the script. SHOSTNAME - The hostname of the machine the script is running on. SSECONDS = The number of seconds since the script was started. SRANDOM - Returns a different random number each time is it referred to. SLINENO - Returns the current line number in the Bash script. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 59: Mail ID : [email protected], Website : www.visualpath.in.Setting your own variables /bash_scripts$ cat 1_print.sh imran@Devops inta=20 float! 0.20 skringAs"first_string" DIR_PATH="/tmp" echo echo "HENEEHERUHEREOHERIEEEEAEE® echo "Value of integer A is Sintam echo "#eeeeeeeedeRERHERETEREERE” echo "Value of Float B is $intB” echo “ERHERRESHHERESEEEHERHESHEM echo "Value of string A is $stringa™ echo "#e#HEHERHE REE HEE ERHERHE” echo "Directory path is SDTRUPATH" echo “SRHERRESPEERERREEHERHERHE echo "Content 6f TMP directory.” echo “S#HERHESFEEEERREEAERHERHE" 1s SDIR_PATH Quotes Storing a single word ina variable works fine without quotes, but if we want to store a sentence and also want to store special characters like $,%,@ etc our normal variable assignment will not work. Imran@Devops:.../bash$ myvar=Hello World World: command not found imran@Devops:. . ./bash$ Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.‘When we want variables to store more complex values however, we need to make use of quotes. This is because under normal circumstances Bash uses a space to determine separate items. ‘When we enclose our content in quotes we are indicating to Bash that the contents should be considered as a single item. You may use single quotes (‘) or double quotes ( ). Single quotes will treat every character literally. Dpuble quotes will allow you to do substitution (that is include variables within the setting of the value). imran@Devops: ~/bash$ myvar="Hello World” imran@Devops -/bash$ echo Smyvar Hello World imran@Devops:.../bash$ newvar="More $myvar™ imran@Devops:.../bash$ echo Snewvar More Hello World imran@Devops. -/bash$ newva "More! $myvar? imran@Devops:.../bash$ echo Snewvar More Smyvar imran@Devops:. . . /bash$ Command Substitution We have how to store a string/text into a variable but sometimes you want to store output of a command to a variable. Like you may need to store of Is command output to a variable. For this we use Command Substitution. There are two syntax for doing this. 1 imran@Devops:..-/éestcopy$ file= is imran@Devops:.../testcopy$ echo $file copyscript.sh digl dir2 2 “Invan@Devopat . . . /testcopys|Files=s (1s) imran8Devops:.../testeopys eho $files copyscript.sh dirl dir? Imran@Devops:.../bash$ myvar=$( 1s ete | we -1 ) imran@Devops -/bash$ echo There are $myvar entries in the directory /etc ‘There are 249 entries in the directory /ete Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.Exporting Variables Variable defined in the script leave with it and dies after the script dies or completes. If we want to define a variable that is accessible to all the scripts from your current shell we need to export it. Export a variable from bash shell as mentioned below. Imran@Devops:.../bashS vari=Foo imran@Devops:.../bash$ echo $varl £00 imran@bevops:.../bash$ export vari (Create a script which prints exported and local variable imran@Devops:.../bash§ vi scripti.sh #!/bin/bash # demonstrate variable scope var2=foobar echo "Printing exported varible from bash shell” echo $varl echo "Printing Varible defined in the script" echo $va¥2 Execute the script to see the results Inran@bevops:.../basnSygcripti sh Printing exported variable from bash shell foo Printing variable defined in the script foobar Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.Environment Variables or Exporting Variables Permanently To export variables permanently, you can add the export command in any of the following start-up files : ~/profile ~hbashre ‘etciprofile profile and .bashrc lives in the users home directory so they are accessible only for that user. Jetciprofile is for global access for all the variables. For example: imran@Devops:~$ cat .bashre | grep export fexport @cc_Cozor: error=01;31:warning=01;35:note=01;36:car@te01; 1 :quote=01" export tesla='Alternate currentyt export EDITOR=vim export name=Cassini Summary $1, $2, “The first, second, ete'command line argumenis tithe script. variable=value To set a value fora variable. Remember, no spaces on either side of = Quotes " Double will do variable substitution, single will not, variable=$(command ) Save the output of a command intova variable export varl Make the variable var available to child processes. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.3. User Input Interactive Scripts Ask the User for Input ‘Taking input from the user while executing the scrip, storing it into a variable and then using that variable in our script. We would be taking inputs from user like IP addresses, usernames, passwords or confirmation Y/N to do this we use command called read. This command takes the input and will save it into a variable. read vari Let's look at a simple example: input.sh T. ¥17bin/bash 2. # Ask the username 3. echo “Please enter the-username” 4, read varnane 5. echo “Welcome back $varname” imran@Devops: -/bashscriptss ./input 6h Please enter the username DevTestops Welcome back DeyTestops Explanation: Line 3 - a message asking the user for input, Line 4 - Run the command read and save the users response into the variable varname Line 5 — echo another message just to verify the read command worked. Note: I had to put a backslash (\ in front of the "So that it was escaped, You are able to alter the behaviour of read with a variety of command line options. (See the man page for read to see all of them.) Two commonly used options however are -p which allows you to specify a prompt and -8 which ‘makes the input silent. This can make it easy to ask for a username and password combination like the example below: Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.Loginsh #1 /bin/bash # Ask the user for login details zead -p 'Usernane: ' uservar read -sp ‘Password: ' passvar ebho echo Thankyou $uservar we now have your login details So far we have looked at a single word as input. We can do more than that however. ‘multiinput.sh #1 bin/bash # Demonstrate how read actually works echo Enter your Name, Profession & Ifter@Bts in same order seprated by a space? read name profession interest echo Your entered name is: Snané echo Your profession is: Sprofessiom echo Your are interested in: Sinterest imran@Devops: -/bashscripts$ ./muitiinput.sh Enter your Name,/ Profession & Interests HHJamé order seprated by a space? Iman Devops Hiking Your entered name is¥/ Imran Your professiohyis: Devops Your arewinterested in: Hiking Visualpath Training & Consulting, Flat no: 205, Nilgiri Block, Aditya Enclave, Ameerpet, Hyderabad, Phone No: -+91-970 445 5959, 961 824 5689 E- Mail ID : online. [email protected], Website : www.visualpath.in,4. If Statements Scripts making decisions. Basic If Statements If you use bash for scripting you will undoubtedly have to use conditions a lot. Based on a condition you decide if you should execute some commands on the system or not. A basic if statement effectively says, if a particular testis tr, then perform a given set of actions. If itis not true then don't perform those actions. If follows the format below: if [ fi Anything between then and fi (if backwards) will be executed only if the test (between the square brackets) is, true, condition is true condition is false rest of code [x ——! If statement low chart Let’ look at a simple example: if_example.sh ¥i/bin/bash # Basic if’ statement £ [ $1 -gt 100} then echo Hey that’s a large number. pwd Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 59: Mail ID : [email protected], Website : www.visualpath.in.fi date Explanation: Line 3 - Let's see ifthe first command line argument is greater than 100 Line 5 and 6 - Will only get run if the test on line 4 returns true. You can have as many commands here as you like. Line 7 - fi signals the end of the if statement, All commands after this will be run as normal. Line 8 - Because this command is outside the if statement it will be run regardless of the outcome of the if statement. imran@Devops:.../bash§ ./if_example.sh 150 Hey that’s a large number. /tmp/bash Sun Oct 30 16:19:28 IST 2016 Sample if/else condition script If condition is true we execute a block of code but if its false we can execute some other block of code by using else command. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.¥iJbin/bash echo "intA is equal to echo "Not Equal” £ [ -£ hosts J; then "File exists!" echo "Does not exist Test intB” ‘The square brackets ([] )in the if statement above are actually a reference to the command test. This means that all of the operators that test allows may be used here as Wells Look up the man page for test to see all of the possible operators (there are quite a few) but some of the more common ones are listed below. Operator 1! EXPRESSION -n STRING -z STRING STRING1 = STRING2 STRING1 STRING2 INTEGER -eq INTEGER2 INTEGERI -gt INTEGER2 INTEGERI -It INTEGER2 -d FILE -e FILE -r FILE -s FILE -w FILE -x FILE Visualpath Training & Consulting, Description ‘The EXPRESSION is false The length of STRING is greater than zero. The length of STRING is zero (ie it is empty). STRINGI is equal to STRING2 STRING1 is not equal to STRING2 INTEGER1 is numerically equal to INTEGER2 INTEGERL is numerically greater than INTEGER2 INTEGER1 is numerically less than INTEGER2 FILE exists and is a directory. FILE exists. FILE exists and the read permission is granted. FILE exists and it's size is greater than zero (ie. it is not empty). FILE exists and the write permission is granted. FILE exists and the execute permission is granted Flat no: 205, Nilgiri Block, Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.A few points to note: = is slightly different to -eq. [ 001 = 1 ] will return false as = does a string comparison (ie. character for character the same) whereas -eq does a numerical comparison meaning [ 001 -eq 1 J will return true. ‘When we refer to FILE above we are actually meaning a path. Remember that a path may be absolute or relative and may refer toa file or a directory. Because [ 5 is just a reference to the command test we may experiment and trouble shoot with test on the ‘command line to make sure our understanding of its behaviour is correct, Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.5. Loops! Execute it again and again and again.... Loops allow us to take a series of commands and keep re-running them until a particular situation is reached. They are useful for automating repetitive tasks. For Loop A ‘for oop’ is a bash programming language statement which allows code to be repeatedly executed A for loop is Classified as an iteration statement i.e. itis the repetition of a process within a bash script. For example you can run Linux command or task 5 times or read and process list of files usingja for loop. A for loop ¢an be used ata shell prompt or within a shell script itself for var in do done The for loop will take each item in the list (in order, one after the other), assign that item as the value of the variable var, execute the commands between do and done then go back to the top, grab the next item in the list and repeat over. The list is defined as a series of strings, separated by spaces. For loops iterate for as many arguments given: Example: ‘The contents of $Variable is printed three times. #17bin/bash for Variable in {1..3} do echo "Variable" done Or write it the "traditional for loop" way: for ((a=1; a <= 3; at+)) do echo $a done Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.They can also be used to act on files. This will run the command ‘cat’ on filel and file2 for Variable in file1 file2 dg cat "$Variable" done or the output from a command This will cat the output from Is For Output in $(1s) do cat "soutput" done While loop The bash while loop is a control flow statement that allows code or commands to be executed repeatedly based on a given condition. For example, run echo command 5 times or read text file line by line or evaluate. the options passed on the command line for a script. syntax ‘The syntax is as follows: while [ condition ] command? Command? commana command! to command3 will be executed repeatedly till condition is true. The argument for a while loop can be any Boolean expression. Infinite loops occur when the conditional never evaluates to false. For loop will print welcome 5 times on screen: example, following wl Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.#/ein/bazh while [$a -le 51 echo "Number Sa" Sees (( se 429) done Printing list of host IP addresses from hosts file. Create a file named hosts $ vi hosts 192.168.1.10 192.168.1.11 192.168.1.12 192.168.1.13 #!/bin/bash © for i in “cat hostsssdo © echo "Printing list of hosts." © echo 94 © done Explanation Line 1. For command substitution we are using backticks ” . It's different from single quote ** “cat hosts" will return the content of hosts file line by line, which will be stored in variable “i”. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.6. Real time use cases Bash script to install Apache, MYSQL and PHP for Ubuntu OS. Colouring your script. Check below mentioned link for information on colouring your echo output. http://misc.flogisoft.com/bash/tip_colors_and_formatting The ANSI/VT100 terminals and terminal emulators are not just able to display black and white text ; they can display colours and formatted texts thanks to escape sequences. Those sequences are composed of the Escape character (often represented by ”{” or””) followed by some other characters: "[FormatCodem”. In Bash, the character can be obtained with the following syntaxes: ° \e + (033 + \xIB #1/bin/bash #COLORS # Reset color_ofi \033f0m" # Text Reset # Regular Colors Red="\033(0;31n" # Red Green#"\03310732n" ¥ Green YeMow="\033/0;33m" #ire1100 Purple=¥\033(0;35m" # purple cyan="\033(0;36n" # cyan # Update packag@syand Upgrade system echo -e "SCyan \n Updating System.. $Co sudo apt-get update -y s& sudo apt-get upgrade -y ## Install aMP echo -e "$Cyan \n Installing Apache? $Color off" Visualpath Training & Consulting, Flat no: 205, Nilgiri Block, Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.sudo apt-get install apache2 apache2-doc apache2-mpm-prefork apache2-utils libexpati ssl-cert -y echo -e "SCyan \n Installing PHP § Requirements $Color_off" sudo apt-get install 1ibapache2-nod-phpS php phpS-conmon phpS-cur] phpS-dev hpS-gd phpS-idn php-pear phpS-inagick phpSencrypt phpsemysqi. PhpS-pe PhP5= a ea echo -e "$Cyan \n Installing MySQL $Color_off” sudo apt-get install mysql-server mysql-client, libmysqlclienti5.dew -y echo -e "SCyan \n Installing phpMyAdmin $Coloxorf™ sudo apt-get install phpmyadmin -y echo -e "$Cyan \n Verifying instalis$color_ Off" sudo apt-get install apache2 1iBapache2-mod-php5 php5_mysql-server php-pear phpS-mysql mysqi-client mysqi~server php5-mysq php5-gd -y ## TWEAKS and Settings # Permissions echo -e "$Cyan \n Permissions for /var/www $Colox Off" sudo chown -R wwadata:wwwsdata /var/ee echo ~e "SGreen \n Permissions have been set $Color_off" # Enabling Mod Rewrite, required for WordPress permalinks and .htaccess files echo -e "SCyan \mJEnabling Modules $color_off” sudo a2enmodlewrite sud®, phpSerimed merypt # Restart Apache echo -e "SCyan \n Restarting Apache $Color_off* sudo service apache? restart Visualpath Training & Consulting, Flat no: 205, Nilgiri Block, Aditya Enclave, Ameerpet, Hyderabad, Phone No: -+91-970 445 5959, 961 824 5689 E- Mail ID : online. [email protected], Website : www.visualpath.in,Backup scripts Being working with systems you may need to take backup of files, directories, log files etc. Below mention scenario shows you how you can automate the backup procedures. In this example, we will create a file and mention the name of log files that needs to be backup up swith tar command, Before taking backup, our script will also tell us if the log file exists or not. It Will skip the backup procedure if the file does not exists. After all there is no point running backup command if the log file does not exist. Create directory for storing log files $ mkdir -p /tmp/scripts/logs ut some files in the logs directory. $ cd /tmp/scripts/logs $ touch ansible.log apache.log mysql/log™ nagies. log ‘You can choose to put some content in the log files, couch will ust create empty files, $ ed /tmp/scripts Create a file where you place the name of the files that you want to backup. $ vi backup_files.txt apache. log mysql.log naglos.1og ansible. log chef. log ‘There is one extra filename chef.log which is not present in our logs directory /Amp/scripts/logs. We will see how we will handle it in our script $ vi backup. sh #!/bin/bash 10¢_b1 /tmp/scripts/logs? BACKUP_DIR='/ttp/scripts/logs_backup! mkdir -p $BACKUP_DIR for i in ‘cat backup files.txt’; do if [ -£ $L0G_DIR/$i J; then Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.echo "Copying $i to logs_backup directory." cp $LOG_DIR/Si $BACKUP_DIR echo "$i log file does exist, skipping.” echo echo "Zipping log files" tar -cxvf logs_backup.tgz logs_backup echo echo “Backup completed successfully.” Mysql Database Backup Script mysqldump command is used to take the db dump for mysqW. In the script, we're taking the dbdump and sending it to a target directory in a zipped format. We are also removing 8 days old GBbackup £i16 bylusing find command. This process is called a purging or putging old backup/log files. #1/bin/sh TIME_NOW="$(date +'%d_%n_%Y_RH_M_¥S')" BACKUPFILE="db_backup_STIME_NOW".gz BACKUP_DIR="/opt/db_backup_dir" PATHOFBACKUPFILE="$BACKUP_OIR/$BACKUPFILE” LOG_FILE="$BACKUP_DIR/"backup_log_"$(date +'%Y_%mn')". txt echo "mysqldump started at $(date +'%d-%n-Y %H:8M:%S')" >> "SLOG_FILE” mysqldump --user=dbuser--password=dbpass --default-character-set=utf8 mydatabase | gzip > "SPATHOFBACKUPFILE" echo "mysqldump finished at $(date +'%d-%n-%Y RH: 1)" >> "$LOG_FILE” chown myuser "$PATHOFBACKUPFILE” chown myuser "$LOG_FILE” echo "file permission changed" >> "$LOG_FILE" Find "$BACKUP_DIR" -name db_backup_* -mtime +8 -exec rm {} \; echo "old files deleted" >> "$LOG_FILE" Visualpath Training & Consulting. Flat no: 205, Nilgiti Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, Mail ID : [email protected], Website : www.visualpath.in.echo "operation finished at $(date +'%d-%m-®Y RH:RM:%S')"” >> "SLOG FILE” echo "Httstenesseenesten oy "$106 FILEM exit 0 Running command on remote servers/nodes ‘Sometimes we need to run a command or set of commands on multiple nodes/server. ‘We use ssh to login to these nodes and run that command individually and manually on all the nodes/servers, But it's a very time consuming and mundane work if you have to do it many times. ‘We will write a bash script to do that. For this exercise, we will choose to run “yum install ttpd” on three nodes Assumptions: 1. Three centos VMs 2. VMs have internet connection to download and install software 3, All VMs have same username to connect. We will create a file named “hosts-dew” and add ip address of all three nodes in that. $ vi hosts-dev 192,168.2.5 192. 168.2.6 192. 168.2.7 ‘We will write a script which will read the ip address from the hosts-dev file, do ssh to all of them one by one and run yum install httpd command over ssh. $ vi install.sh #1/bin/bash for hosts in ‘cat hosts-dev* do ssh vagrant@Shosts sudo yum install httpd -y done Let’s break it down Line 2: for loop will run over the content on hosts-dev file one by one, which are the ip addresses of the VMs. Notice ‘we have used backticks “> and not single quotes ‘? to read the hosts-dev file (“cat hosts-dev” ). Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 59: Mail ID : [email protected], Website : www.visualpath.in.Line 4: we are running sudo yum install httpd -y command over ssh, Shosts variable will hold the ip address and will establish ssh connection with vagrant user( ssh vagrant@Shosts). This loop will run until we exhaust all the entries in the hosts-dev file, if you have lets say 50 nodes you can add ip address ofall the 50 nodes in this file Every time it logs into the vm's/sever/nades it will ask you a password, it will be painful if you have lot of servers that you manage to enter password manually. In the next section, we will deal with this issue by doing ssh key exchange. The above script will install httpd package on all the three nodes but will also ask you password everytime it does a ssh login. To avoid this we can do key based login which is discussed in next chapter. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block, Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.7. How To Set Up SSH Keys About SSH Keys SSH keys provide a more secure way of logging into a virtual private server with SSH than using a password alone. While a password can eventually be cracked with a brute force attack, SSH keys ate nearly impossible to decipher by brute force alone. Generating a key pair provides you with two long string of characters: a public and a private key. You can place the public key on any server, and then unlock it by connecting to it with a client that already has the private key. When the two match up, the system unlocks without the need for a password, You can increase security even more by protecting the private key with a passphrase. Step One—Create the RSA Key Pair The first step is to create the key pair on the client machine (there isa good chance that this will just be your computer): § ssh-keygen -t rsa Step Two—Store the Keys and Passphrase Once you have entered the Gen Key command, you will get a few more questions: Enter file in which to save the key (/home/demo/.sshiid_rsa) ‘You can press enter here, saving the file to the user home (in this case, my example user is called demo). Enter passphrase (empty for no passphrase): Its up to you whether you want to use a passphrase. Entering a passphrase does have its benefits: the security of a key, no matter how encrypted, still depends on the fact that itis not visible to anyone else. Should a passphrase-protected private key fall into an unauthorized user’s possession, they will be unable to log in to its associated accounts until they figure out the passphrase, buying the hacked user some extra time. The only downside, of course, to having a passphrase, is then having to type it in each time you use the Key Pair. The entire key generation process looks like this: $ ssh-keygen -t rsa Generating public/pfivate req/key pair, Enter file in whi€WlEBlgave the key (/nome/demo/.ssh/id_rea)+ Enter passphrase (empty for no passphrase) Enter sane passphrase again: Your identification has been saved in /home/demo/.ssh/id_rsa. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block, Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.Your public key hae been saved in /home/demo/.eeh/id_rsa.pub. The key fingerprint is: 4ardd:Oarc6:35:4e:3£:ed:27 Bcr74:44:4d:93:67 denota ‘The key's randomart image is +--[ RSA 2048] ‘The public key is now located in ‘home/demo/.sshid_rsa.pub The private key (identification) is now located in ‘/home/clemo/.sshiid_rsa Step Three—Copy the Public Key Once the key pair is generated, it's time to place the public key on the virtual server that we want to use. You can copy the public key into the new machine's authorized keys file with the ssh-copy-id command, Make sure to replace the example username and IP address below. § ssh-copy-id/us@[email protected] Alternatively, you can paste in the keys using SSH: s cat ~/.ssh/id_rea.pub | ssh [email protected] “mkdir -p ~/.ssh 6& cat >> ssh/authorized_keys" Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 59: Mail ID : [email protected], Website : www.visualpath.in.No matter which command you chose, you should see something like: he authenticity of host '192.168.2.5 (192.168.2.5)' can't be established.RSA key fingerprint is bl:2d:33:67:ce:35:4d:5£:£3 48:86:12. Are you sure you want to continue connecting (yes/no)? Yes Warning: Permanently added '12.34.56.78' (RSA) to the 1i8t of known hosts. [email protected]'s password: Now try logging into the machine, with "ssh ‘[email protected]'", andljcheck in: ~/.ssh/authorized_keys to make sure we haven't added extra keys that you weren't expecting, Now you can go ahead and log into [email protected] and you will not be prompted for a password. However, if you set a passphrase, you will be asked to enter the passphrase at that time (and whenever else you log in in the future). Exercise: ~ Create 4 centos vm’s, One among these four vm one will be the automation box from where we will run our scripts. - Do SSH key exchange from automation box to rest of the three vm’s. - Write a script which will install Apache & mysql server, start and enable both services and check status of apache, mysql & iptables. Execute this script from automation box for all three vm and validate. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.8. Few Sample scripts A script to automate below mentioned task on Centos 1. Install mlocate search tool, update mlocate database 2, Install, start & enable httpd service. 3. Find files with permission 0777 and delete it. 4, Check hard disk free space and alerts if its running low on disk space. ¥1/bin/bash # Sample script to automate task: # -Update local file database: echo -e "\e[4;32mInstalling mlocate\e[0m" sudo yum install mlocate -y echo -e "\e[4;32mUPDATING LOCAL FILE DATABASE \e[0m™ sudo updatedb ($7 0 J; then echo "The local file database was updated correctly.” echo "PR Jocal file database wa3 not updated correctly.” fi echo # Installing and staring HITED service echo “e."\e[4;32mInstalling HTTPD package\e[0n" sudo yum install nttpdlby echo -e "\e[4;32mstarting and enabling HTTPD package\e[0m" sudo /etc/init.d/nttpd start s& chkconfig httpd on echo "™ # For Centos 7 #sudo systemet start httpd ¢& sudo systemct1 enable httpd Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.# -Find and / or delete files with 777 permissions. echo -e "\e/4;32mLOOKING FOR FILES WITH 777 PERMISSIONS\e[0m" # Enable either option (comment out the other line), but not both. # Option 1: Delete Files without prompting for confirmation. Assumes GNU version of find. find -type f -perm 0777 -delete # option 2: Ask for confirmation before deleting files. Mofeyportable across systems. find ~type f -perm 0777 exec rm -£ {} +7 echo *™ # “Alert when file system usage surpasses a deFined)iinit echo ~e "\e[4;32mCHECKING FILE SYSTEM USAGE\e [Om" ‘THRESHOLD=10 while read line; do # This variable stores the file system path as a string FILESYSTEMS (echo $line awk ' {print $1}") # This variable stores the/Use percentage (XX#) PERCENTAGE=$ (echo $line | awk "(print $5}') # Use percentage without the % signs -$ { PERCENTAGES?) Af [ SUSAGE.-gt $THRESHOLD J; then echo "The remaining availabl@yspace in $FILESYSTEM is critically low, Used: $PERCENTAGE™ otal | grep Avi filesystem) Automatic maintenance script for nginx service. This Script starts nginx service if its dead. Place it under cronjob and schedule to run every 2 minutes which acts as a monitoring and maintenance script. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.Assumptions: Nginx is already installed on the system. cat nginstart.sh #!/bin/bash af [ -f /var/run/nginx.pid J then echo "Nginx is running.” else echo "Starting nginx service." service nginx start Fi # crontab -e + ++ + * /opt/scripts/nginstart.sh Example Script for automating Jenkins setup. #1/bin/bash ~ 4 # Author: Pavan Kumar Ranjit ## LOGIC TO CHECK THE TYPE OF DISTRIBUTION (REDHAT OR DEBIAN) yum --help >> /tmp/log1 if [ 8? -eq 0] ## "$2" STORES THE EXIT CODE OF THE MOST RECENT COMMAND then echo "RPM Based 0S Detected” echo “Installing Java-JOK, Jenkins, Maven” sleep 3 sudo yum install java-1.8:0-openjak -y sudo yum install java-4.8.0-openjdk-devel -y sudo yum install wget -y sudo wget -0 /etc/yum.repos.d/jenkins.repo http://pkg.jenkins- ci.org/redhat/jenkins. repo sudo rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key sudo yum install Jenkins -y sudo yum install maven -y sudo yum install git -y Visualpath Training & Consulting, Flat no: 205, Nilgiri Block, Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.echo "Configuring services.... Please wait” sleep 5 sudo service iptables stop sudo service Jenkins start else echo "Debian Based 0S Detected” sleep 3 echo "Installing Java-JOK, Jenkins, Maven” sudo apt-get update sudo apt-get install openjdk-8-jdk -y sudo apt-get install openjdk-8-jre -y sudo apt-get install maven -y sudo apt-get install wget -y wget -q -0 - https://pkg. jenkins.io/debian/jenkins-ci.org.key | sudo apt-key add - sudo sh -c ‘echo deb http://pkg. jenkins. io/debian-stable binary/ > /etc/apt/sources. list.d/jenkins. list’ sudo apt-get update -y sudo apt-get install jenkins)-y sudo apt-get install git -y echo "Configuring services... Please Wait” sleep 5 sudo systemct] stop ufw sudo systemctl start jenkins Fi Summary: ‘* Scripting is required for doing system tasks automatically without manual intervention. ‘© Bash scripting is used by Linux System Admin for ages to automation Linux tasks. ‘© Variables, Condition’s, loops etc are important aspects of scripting language which helps us automate complex tasks. * Try all the sample and real-time use case scripts to get hold of system automation. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block, Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.Conclusion: There are so many advanced options in Bash Scripting like functions, list constructs, Regular Expressions etc, which you may be interested in and feel tempted to use them. ‘You can check Advanced Bash Scripting guide http://tldp.org/LDP/abs/html/ for advanced options. Learning all those advanced options are great and make you a Scripting Guru if practised, There are also so many limitations to Bash script, like its only for Linux systems and there is so much of syntax in it, Python should be your next choice of scripting language which is easier to read and write and is also versatile, it can be used to automate tasks on windows platform and cloud services also. Python has very less syntax for example check below “if condition” of bash vs python. Bash way #1 /bin/bash ass [ $a -1t 10} then echo "Variabléliis ‘smaller than 10" exit 1 Python way #1 /ust/bin/python 0 ifa< 10: print “variable is less than 10” Both have their upsides and downside, you should make your choices wisely. But making choices comes with experience and practice. There is no perfect way of doing anything. Practice makes improvement, Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 59: Mail ID : [email protected], Website : www.visualpath.in.III. Virtualization Virtualization is not a new thing in IT industry. Virtualization gained huge momentum because of ‘Vmware as it has solved the problem of running multiple OS and apps on one physical computer. But that is just the tip of the iceberg, Vmware and other virtualization vendors has given great féatures like Clustering of Virtualized Physical machines, pooling compute resources, storage pooling, High availability, Live migration of virtual machine and so much more. In this chapter we will understand virtualization, its benefits and how to use it for our DevOps day to day operations. 1. Life without virtualization In software industry, we create and deliver softwares or as we say nowadays Applications. Application runs businesses or vice versa. If application don’t run or underperform, business will suffer. Applications runs on servers. We deploy one application per server because we want our applications to be isolated. For example, if we need web app, db app and few backend apps. We may end up having multiple physical system each running a single instance of that app. “gad a8 So, every time we need a new app to,run we buy servers, install OS and setup our app on that. And most of the time nobody knew the performance requirements of the new application! This meant IT had to make guesses when choosing the model and size of servers to buy. Asa result, IT did the only reasonable thing - it bought big fast servers with lots of resiliency. After all, the last thing anyone wanted - including the business - was under-powered servers. Most part of the time these physical server’s computer resource will be underutilized as low as 5- 10% of their potential capacity. A tragic waste of company capital and resources. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block, Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.2. Enter the Vmware Amid all of this, VMware, Inc. gave the world the virtual machine (VM). And almost overnight the world changed into a much better place! Finally, we had a technology that would let us run multiple business applications on a single server safely and securely. . 3.The Virtual Machine A virtual computer system is known as a “virtual machine” (VM): a tightly isolated software container with an operating system and application inside. Each self-contained VM is completely independent. Putting multiple VMs on a single computer enables several operating systems and applications to run on just one physical server, or “host”. 4. Key Properties of Virtual Machines VMS have the following characteristics, which offer several benefits © Partitioning ‘* Run multiple operating systems on one physical machine * Divide system resources between virtual machines * Isolation ‘© Provide fault and security isolation at the hardware level ‘© Preserve performance with advanced resource controls © Encapsulation ‘© Save the entire state of a virtual machine to files ‘© Move and copy virtual machines as easily as moving and copying files © Hardware Independence Provision or migrate any virtual machine to any physical server Visualpath Training & Consulting, Flat no: 205, Nilgiri Block, Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.5. Hypervisors As generally we install OS on the physical server and then install our apps on that. By setting up Hypervisor on the physical server we can create multiple VMs each with their own OS. : Perey Application: Applications Em EE Tge NC LE ‘There are two types of hypervisors: Type 1 ‘Type 1 hypervisors run directly on the system hardware, They are often referred to as a "native" or “bare metal” hypervisors. ‘They get installed on a physical computer like an OS. So instead of installing OS we install Hypervisor to achieve virtualization. Example: © Microsoft Hyper-V * VMware ESX/ESXi, © Xen Hypervisors ‘These hypervisors are used to run production grade virtual machines that gives great performance. Type 1 hypervisors can be grouped together(clustered) and manged centrally by a software like Vmware Vcenter for Es) Visualpath Training & Consulting, Flat no: 205, Nilgiri Block, Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.‘Type 2. ‘Type 2 hypervisors run on a host operating system. In this kind, we install hypervisor on the OS like Linux or windows. Example + © Vmware server/workstation/player © Oracle virtualbox. Applications Died Lcud De cur eS uy De ea 6. Your choice of hypervisor. ‘Type 1 hypervisors are great and gives you amazing features like Clustering of Virtualized Physical machines, pooling compute resources; storage pooling, High availability, Live migration of virtual machine and so much more. It's used generally to virtualize entire datacentres or pool of servers. This is managed by very expensive softwares like Vmware Vcenter. As the focus of these tutorials is on DevOps tools we will not go into type 1 hypervisors. Our focus is going to be on type 2 hypervisors like oracle virtualbox which we can install on our Laptops/desktops and get going with VMs. We will need to run multiple VMs on our system to test/run our tools and scripts. We also need to setup web application softwares on multiple OS and we will do it by using Oracle virtualnox Hypervisor. Visualpath Training & Consulting, Flat no: 205, Nilgii Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - #91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpathi.7. Some terminologies When dealing with virtualization (and also for understanding the following chapters of this documentation), it helps to acquaint oneself with a bit of crucial terminology, especially the following terms: Host operating system (host OS). This is the operating system of the physical computer on which VirtualBox was installed. There are versions of VirtualBox for Windows, Mac OS X, Linux and Solaris hosts. Guest operating system (guest OS). This is the operating system that is running inside the virtual machine. 8. Installing virtualbox on windows. > Go to Virtualbox Download page, Click Windows hosts. > Open VirtualBox exe file and follow below screenshots. Bock ai venaonSt Sete x Welcome to the Oracle VM VirtualBox 5.1.18 Setup Wizard Iyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- »wisualpath.in Cee) Coe mwees [oar] [ete ee] eae 9 Oracle VM ViralBox 5.1.18 Setup stom Setup ‘Sect nay yu ant fate abe rae ene dose ton teaotare oe oes start mers ees ovate astertaston te destin Cloves srt fe ack ach Br Crepe te asccasone sed [rer] oeal Visualpath Training & Consulting, Flat no: 205, Nilgiri Block, Aditya Enclave, Ameerpet, Hyderabad, Phone No: -+91-970 445 5959, 961 824 5689 E- Mail ID : online. [email protected], Website : www.visualpath.in,3H) Oracle VM VirtualBox 5.1.18 Setup Ready to Install The Setup Wizard is ready to begin the Custom installation. (Click Install to begin the installation. If you want to review or change any of your installation settings, cick Back. Click Cancel to exit the wizard, When you start VirtualBox for the first time, a window like the following should come up: ee = pees DY pe arctan arrr G Just to give you an idea, what VirtualBox might look like later, after you have created many machines, here's another example: 22 a bt ja TEES eer Breer seen ES sot sear cn ener eae Bags Ear So Brssrgewrnene a beeper = Beem ae Sree i Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : online. visualpath@gmail,com, Website : www.visualpath.in.> Click on the "New" button at the top of the VirtualBox Manager window. A wizard will pop up to guide you through setting up a new virtual machine (VM): > On the following pages, the wizard will ask you for the bare minimum of information that is needed to create a VM, in particular: Name and operating system Please choose a descriptive name for the new virtual machine and select the type of operating system you intend to install on it. The name you choose will be used throughout VirtualBox to identify this machine. Name: | ‘Type: Microsoft Windows Version: Windows 7 (64-bit) Expert Mode) | Cancel 1. The VM name will later be shown in the VM list of the VirtualBox Manager window, and it will be used for the VM's files on disk. Even though any name could be used, keep in mind that once you have created a few VMs, you will appreciate if you have given your VMs rather informative names; “ubuntu” would thus be less useful than "Windows XP SP2 with OpenOffice’, 2. For“Operating System Type”, select the operating system that you want to install later. 3. On the next page, select the memory (RAM)that VirtualBox should allocate every time the virtual machine is started. The amount of memory given here will be taken away from your host machine and presented to the guest operating system, which will report this size as the (virtual) computer's installed RAM. Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.rina een) Name and operating system Please choose a descriptive name for the new virtual machine and select the type of operating system you intend to install on it. The | name you choose will be used throughout VirtualBox to identify | this machine. ubuntu-learn-devops Type: Linux a Version: Ubuntu (64-bit) Jan Expert Mode | MB 4MB 16384 MB Next, you must specify a virtual hard disk for your VM. Here you have the following options: Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 5959, 961 824 5689 E- Mail ID : [email protected], Website : www.visualpath.in.© To create a new, empty virtual hard disk, press the"New"button. ‘* You can pick an existing disk image file. ‘©The drop-down list presented in the window contains all disk images which are currently remembered by VirtualBox, probably because they are currently attached to a virtual machine (or have been in the past) ‘© Alternatively, you can click on the small folder button next to the drop-down list to bring up a standard file dialog, which allows you to pick any disk image file on your host disk. ‘© Most probably, if you are using VirtualBox for the first time, you will want to create a new disk image. Hence, press the "New" button, ‘© This brings up another window, the “Create New Virtual Disk Wizard”, which helps you create a new disk image file in the new virtual machine's folder. VirtualBox supports two types of image files: * Dynamically allocated file will only grow in size when the guest actually stores data on its virtual hard disk. It will therefore initially be small on the host hard drive and only later grow to the size specified as it is filled with data.. * A fixed-size file will immediately occupy the file specified, even if only a fraction of the virtual hard disk space is actually in use. While occupying much more space, a fixed-size file incurs less overhead and is therefore slightly faster than a dynamically allocated file. pena eee) Hard k IF you wish you can add a virtual hard disk to the new machine. You can either create a new hard disk file or select one from the list or From another location using the folder icon. IF you need a more complex storage set-up you can skip this step and make the changes to the machine settings once the machine is created. The recommended size of the hard disk is 10.00 GB. Do not add a virtual hard disk @ Create a virtual hard disk now Use an existing virtual hard disk File Live co/Dvo concel SRN Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 59: Mail ID : [email protected], Website : www.visualpath.in.General Perrier System | Look in: {/home/imran/1sO ‘i sae} Display | Computer Name + Size Ty Master Storage) 55 imran ty eabit Fo | ‘Audio 7601.17514.10111...KEVAL_EN_DVD.iso_ 2.95 GB isc | Network 9600.17050.WINBLU...EN-US_DV9(1).ISO 4.23 GB Ist caitttl kal-linux-2016.2-amd64.iso 2.87 GB isc crial ru-en_win7_sp1_x86...1_activated_v4iso 4.23 GB isc uss ubuntu-16.04.1-desktop-amd64.iso 1.41 GB isc Shared Ft vubuntu-16.04.2-server-amd64.iso 829.0 MB isc User inte File name: Files of type: All virtual optical disk files (*.dmg*.iso*.cdr) = Cancel in atete > Once the ISO is attached power on your vm by double clicking it or click on start button, ubuntu-learn-devops [Running] - Oracle VM VirtualBox fmboric | Francais MaxeaoncK Tamil bic Gaeilge Maleyalam Som Asturianu | Galego Morathi Thai Benapycxan | Gujerati Burmese Tagalog Semrapcxn | may Nepali Tirkce Bengali | Hindi Nederiands. Uyghur Tibetan | Hrvatski Norsk bokmal vxpaincexa Bosanski | Magyar Norsk nynorsk Ting viet Catala Bahasa Indonesia | Punjabi (Gurmukhi) | PYZCie Hk) Cestina islenska Polski yc SH) Dansk Italiano Portugués do Brasil Deutsch BAIR Portugués Dzongkha | deFozes Roman Exinvucir | kasax Pyccen khmer Samegillii Esperanto | 83*3e. 298 Espafiol | 2301 Slovenéina eesti Kurd Slovenééina euskera | Lao Shain Soe Lictuviskai Cenc Suomi Latviski Svenska Visualpat 7 . G08 20288008 Flat no: 205, Nilgiri Block, Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 45 5959, 961 824 5689 E~ Mail ID: [email protected], Website: www.visualpath.in,> Follow the installation procedure and setup the vm. Snapshots With snapshots, you can save a particular state of a virtual machine for later use. At any later time, you can revert to that state, even though you may have changed the VM considerably since then. To take snapshot of a machine right click on the vm and click on snapshot. ‘When you want to restore machine from snapshot, find the snapshot location on your filesystem and simply double click on it Virtual Machine configuration ‘When you select a virtual machine from the list in the Manager window, you will see a summary of that machine's settings on the right. Right click on the vm and open settings window. ‘You can configure various configuration of the vm’s like RAM size, number of CPU’s, Network etc. system Motherboard Processor Acceleration 1B Display GD storage Base Memory: + ems D Audio Me 63846 W Network poot order: Hard Disk © ® Serial Ports I Floppy o © optical a Network (Gl Shared Folders omue 1B User interface Pointing Device: PS/2 Mouse Extended Features: & Enable /0 APIC Enable EFl (special OSes only) @ Hardware Clock in UTC Time cancel SRS Visualpath Training & Consulting, Flat no: 205, Nilgiri Block,Aditya Enclave, Ameerpet, Hyderabad, Phone No: - +91-970 445 59: Mail ID : [email protected], Website : www.visualpath.in.Cloning virtual machines To experiment with a VM configuration, test different guest OS levels or to simply backup a VM, VirtualBox can create a full or a linked copy of an existing VML5] A wizard will guide you through the clone process: Grau reed New machine name Please choose a name for the new virtual machine. The new machine will be a clone of the machine Fabric- scripts_default_1496279304678_44633. Enea ee eT ECE ERS Reinitialize the MAC address of all network cards Expert Mode

You might also like

• Aws by Riaz SIR

  

  100% (3)

  Aws by Riaz SIR

  545 pages
• Hand Written DevOps Notes

  

  100% (3)

  Hand Written DevOps Notes

  241 pages
• DevOps 300 Interview Questions With Answers

  

  100% (1)

  DevOps 300 Interview Questions With Answers

  71 pages
• Docker Notes

  

  No ratings yet

  Docker Notes

  15 pages
• Venkat GCP DevOps

  

  No ratings yet

  Venkat GCP DevOps

  6 pages
• Docker Handwritten Notes

  

  100% (1)

  Docker Handwritten Notes

  61 pages
• Jenkins Notes

  

  100% (1)

  Jenkins Notes

  11 pages
• DevOps Tutorial For Begin.9093468.Powerpoint

  

  100% (2)

  DevOps Tutorial For Begin.9093468.Powerpoint

  7 pages
• DevOps Material

  

  100% (6)

  DevOps Material

  153 pages
• Google Cloud Sample MCQs

  

  No ratings yet

  Google Cloud Sample MCQs

  8 pages
• Vikram Kumar Devops Notes: Jenkins

  

  100% (3)

  Vikram Kumar Devops Notes: Jenkins

  15 pages
• Ansible Interview Questions and Answers

  

  No ratings yet

  Ansible Interview Questions and Answers

  7 pages
• Devops

  

  No ratings yet

  Devops

  172 pages
• Linux Commands MT

  

  0% (1)

  Linux Commands MT

  72 pages
• Diagnostic Questions Associate Cloud Engineer

  

  No ratings yet

  Diagnostic Questions Associate Cloud Engineer

  5 pages
• GCP Dump Questions

  

  100% (1)

  GCP Dump Questions

  62 pages
• Jenkins

  

  No ratings yet

  Jenkins

  39 pages
• Docker Notes by Bhupinder Sir

  

  100% (1)

  Docker Notes by Bhupinder Sir

  21 pages
• DevOps Interview Questions - 1 PDF

  

  No ratings yet

  DevOps Interview Questions - 1 PDF

  224 pages
• Edureka DevOps Ebook

  

  83% (6)

  Edureka DevOps Ebook

  21 pages
• Devops Project 2 Develop Jenkins Pipeline Project

  

  No ratings yet

  Devops Project 2 Develop Jenkins Pipeline Project

  6 pages
• TG Devops Bhupinder Sir Notes

  

  100% (4)

  TG Devops Bhupinder Sir Notes

  64 pages
• Jenkins Kubernetes PDF

  

  No ratings yet

  Jenkins Kubernetes PDF

  7 pages
• DevOps Exam

  

  No ratings yet

  DevOps Exam

  1 page
• Devops Notes

  

  No ratings yet

  Devops Notes

  55 pages
• DevOps Interview Questions

  

  100% (3)

  DevOps Interview Questions

  224 pages
• DevOps Interview Question IMP

  

  100% (2)

  DevOps Interview Question IMP

  41 pages
• DevOpsSyllabus MithunTechnologies 2020

  

  No ratings yet

  DevOpsSyllabus MithunTechnologies 2020

  10 pages
• Naresh It Course - Devops Chapter - SDLC

  

  No ratings yet

  Naresh It Course - Devops Chapter - SDLC

  25 pages
• Intelliq Devops Soft

  

  No ratings yet

  Intelliq Devops Soft

  245 pages
• NT-TRN-V1.0-Git & Github

  

  100% (1)

  NT-TRN-V1.0-Git & Github

  18 pages
• Project - CICD Using Jenkins and Docker

  

  No ratings yet

  Project - CICD Using Jenkins and Docker

  38 pages
• Devops Full Note by Bhupinder Sir

  

  100% (2)

  Devops Full Note by Bhupinder Sir

  240 pages
• Devops Full Notes

  

  No ratings yet

  Devops Full Notes

  101 pages
• DevOps Course Content by ImranTeli

  

  100% (1)

  DevOps Course Content by ImranTeli

  13 pages
• Aws + Azure + GCP - Devops Course Content V1.2

  

  No ratings yet

  Aws + Azure + GCP - Devops Course Content V1.2

  8 pages
• DevOps Project

  

  No ratings yet

  DevOps Project

  160 pages
• Certification Project: Problem Statement - I

  

  0% (1)

  Certification Project: Problem Statement - I

  3 pages
• DevOps Interview Questions PDF

  

  No ratings yet

  DevOps Interview Questions PDF

  224 pages
• Ci CD

  

  75% (4)

  Ci CD

  151 pages
• DevOps Basics

  

  100% (1)

  DevOps Basics

  28 pages
• Devops CI CD Project

  

  No ratings yet

  Devops CI CD Project

  18 pages
• Daily Notes DevOps

  

  No ratings yet

  Daily Notes DevOps

  53 pages
• Jenkins Interview Qns

  

  No ratings yet

  Jenkins Interview Qns

  4 pages
• AWS & DevOps MicroDegree Syllabus (New-2)

  

  No ratings yet

  AWS & DevOps MicroDegree Syllabus (New-2)

  8 pages
• Vikram Kumar Devops Notes: Build Management

  

  No ratings yet

  Vikram Kumar Devops Notes: Build Management

  6 pages
• Maven: Mr. Ashok

  

  100% (1)

  Maven: Mr. Ashok

  20 pages
• Devops Lab Manual

  

  67% (3)

  Devops Lab Manual

  33 pages
• Mastering Jenkins - Sample Chapter

  

  No ratings yet

  Mastering Jenkins - Sample Chapter

  35 pages
• Durgasoft Git For Devop Study Material

  

  100% (4)

  Durgasoft Git For Devop Study Material

  140 pages
• Docker Interview Questions

  

  100% (1)

  Docker Interview Questions

  6 pages
• Amazon: Questions & Answers

  

  No ratings yet

  Amazon: Questions & Answers

  279 pages
• Devops Links

  

  No ratings yet

  Devops Links

  1 page
• Top 50 AWS DevOps Interview Questions & Answers 2021 - Intellipaat

  

  100% (1)

  Top 50 AWS DevOps Interview Questions & Answers 2021 - Intellipaat

  9 pages
• Kubernetes Material - 1

  

  No ratings yet

  Kubernetes Material - 1

  50 pages
• Jenkins Class Notes

  

  No ratings yet

  Jenkins Class Notes

  6 pages
• AWS Learning Links

  

  No ratings yet

  AWS Learning Links

  11 pages
• DevOps Stack PDF

  

  No ratings yet

  DevOps Stack PDF

  3 pages
• 14.Dockers

  

  No ratings yet

  14.Dockers

  56 pages
• DevOps Master - 2020

  

  No ratings yet

  DevOps Master - 2020

  4 pages
• The Great Indian Kitchen

  

  No ratings yet

  The Great Indian Kitchen

  3 pages
• 70 Linux Questions Part2

  

  No ratings yet

  70 Linux Questions Part2

  148 pages
• Jenkins Automation

  

  100% (1)

  Jenkins Automation

  30 pages
• Azure Notes

  

  No ratings yet

  Azure Notes

  3 pages