Introduction to DevOps slides.pdf
- 1. Introduction to DevOps Agile Training Series Spring 2016
- 2. Course Description Course Description The simultaneous needs for IT to 1) deploy new features and 2) keep systems up and running creates a core conflict that challenges development, operations to the respond to business needs customers in a timely manner. DevOps represents practices, tools, and a culture that seeks to resolve this core conflict by enabling operations and development engineers to participate together in the entire service life cycle, from design through the development process to production support. This class will explore these practices, tools, and culture using Gene Kim's "Three Ways of DevOps" as guideposts. Course Audience Newcomers to Agile and DevOps will find this class a welcoming environment to learn the basics on the DevOps mindset, The Three Ways, automation pipelines, common DevOps systems and tools, and continuous integration / continuous delivery (CI/CD) Operations Engineers will appreciate the class focus on using DevOps to effectively manage the large quantity and frequency of changes demanded in modern IT operations while keeping systems stable Agile Teams already developing software using agile methods will find this class to be a logical extension toward achieving synchronicity with operations and business using DevOps 2
- 3. Learning Goals Today Experience the DevOps way of thinking Form beliefs about how DevOps can work for you Tomorrow Identify actions for your project Weeks/Months See improved results Create DevOps experiences for others Years Build a widespread DevOps Culture in our organization 3 Who are you? What are you working on? How do you plan to apply DevOps? Introductions
- 4. Let’s Review Our Progress with Agile So Far… 4 What results have we seen working this way? kanban USCIS Agile Projects/Portfolios
- 5. Let’s Review Our Progress with Agile So Far… Early and continuous delivery of valuable software Rapid feedback Empirical decision-making Satisfied customers Business people and technical people working together Measurement-based forecasting Harnessing change for competitive advantage 5 The Agile Manifesto and the agile methods that followed focused on software development – DevOps is a logical evolution of a maturing agile process Emergent design Technical excellence Empowered self-organizing teams Personal safety Sustainable pace High trust environments Lean processes Continuous improvement We applied the agile empirical mindset and agile methods and observed these results:
- 7. 7 Leave class able to confidently answer these questions: Who is Dev? Who is Ops? What is DevOps? “The beginning of wisdom is the definition of terms” - Socrates The Basics
- 8. Traditional Development The Inventors Create new features and functionality in “dev” environment Occasionally deliver new product to operators, along with instructions May incorporate feedback from operators in future deliveries Rewarded for delivering new features 8 The inventors are responsible for changing the system
- 9. Traditional Operations The Mechanics Receive new product from developers to be installed and operated Expected to keep production systems up and running Track problems, deployment failures, and system outages May provide feedback to the inventors for future consideration Penalized for downtime 9 The mechanics are responsible for keeping the system in operation
- 10. Differing Views on Change 10 Change Orientation Stability Orientation Logical extremes Alienate customers b/c system doesn’t change Alienate customers b/c system constantly changes Hero Obstacle
- 11. We Have A LOT of Changes USCIS needs to update IT capabilities to support field users AND USCIS needs to keep IT capabilities operational for use by field users AND USCIS needs to keep IT capabilities compliant with security, regulatory, and compatibility requirements Can we deploy new patch for the release? Can you deploy this one, small Change? Can you upgrade the database version? Can you deploy this one, small Change? Can you deploy this one, small Change? Prod is running slow, can you cycle the server? Can you deploy this one, small Change? Can you upgrade the operating system? Can we deploy latest version? Can you deploy this one, small Change? Can you stage this new environment? Can we apply this security patch? Production server is down, fix it now!! USCIS applied over 4,000 changes in 2014
- 12. Separation of Dev and Ops: A History As computers became more complex, dev and ops became necessarily specialized: Accelerating pace of technology Increased demand for turning around new features Huge amounts of data and number of calculations More and more specialized tools Increasingly abstract architectures and design patterns 12 Nobody can be an expert in everything – your enterprise can’t rely on Brent! Augusta Ada King, Countess of Lovelace (1815-1852) And these were the problems in 1945!
- 13. The Reunification of Dev + Ops 13
- 14. DevOps in a Nutshell DevOps is the practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support 14 Monitoring, Feedback, and Action Automated Systems People Collaborating Hmm… what would happen if we extend the core drivers of successful agile development to operations? What if we built a bunch of great tools to help us?
- 15. Breaking the Silos: Communication, Collaboration, Integration How can dev help system stability? How can ops help accelerate feature delivery? We can build cross-functional teams around “knowledge overlaps” – people with experience on both sides and “Ops Devs” 15 Communication Integration Collaboration Development Operations
- 16. Breaking the Silos: Dev and Ops 16 Development Operations Ops can anticipate how new functionality will effect production Dev can respond to bugs and deployment failures quickly Dev and Ops can work together to permanently remove root causes of bugs and failures Ops trusts dev will provide good code Dev trusts ops will put code in prod quickly Visibility enables “trust but verify”
- 17. Dev and Ops Working Together 17 Create feedback loops between inventors and mechanics Expose real-time metrics from ops enabling dev to learn from the system running under real world conditions Expose real-time metrics from dev enabling ops to anticipate production needs and provide early input Cross-functional teams collaborate to deliver whole working systems including all infrastructure, software code, and configurations Feature delivery + stability become shared goals
- 18. Matching IT Capacity to Business Demand 18
- 19. Breaking the Silos: Communication, Collaboration, Integration 19 Communication Integration Collaboration Development Business Operations
- 20. Breaking the Silos: Dev, Ops, and Business 20 Development Business Operations Business better understands capability for changes to features and functionality Dev can better incorporate needs of the business and customers into new development
- 21. Breaking the Silos: Dev, Ops, and Business 21 Development Business Operations Business better understands operational capabilities Ops understands better how to support business goals
- 22. Business Demand: Continuously Deliver Valuable Software Modern business is dependent on IT deploying new features Need very fast time-to-value in the face of change Immigration policy can change rapidly – IT capacity must keep up Immigration Executive Action Multiyear lead time no longer acceptable Expectations for delivery times continue to decrease
- 23. Software is increasingly customer- facing, rather than internally-facing Customers expect an interactive, self- service interface Customers expect deep, direct engagement with their data, not a paper system Customers expect to be able to get information immediately Customers can now identify problems in our systems directly – and they expect us to fix them Business Demand: Support Modern Norms of Customer Interaction
- 24. Business Demand: Rapidly Incorporate Latest Technology Modern web interfaces Mobile devices Social media Accessibility tools Live customer interaction tools Tools for online communities and user-generated content Amazing new features
- 25. Business Demand: “Lean Bureaucracy” Supporting Government Values 25 “Working in public” Governance – many, many stakeholders Transparency in how we work “Presentability” of what we produce Mission alignment Risk aversion Baked in support of values such as: • Contracting preferences • Hiring fairness • Procurement fairness DOES14 - Mark Schwartz
- 26. Business Demand: Respond to Feedback Very Quickly System operations increasingly yields insights that must be acted immediately to keep pace with demand Availability of ubiquitous automated data collection yields expectations that organizations will rapidly act on key data points to improve efficiency in mission and services With so many routes to innovation, organizations are expected to test and identify the best options very quickly Well run companies are expected to maintain very low MTTR (mean-time to repair) times – delays in fixing problems can be catastrophic
- 27. Beware! … we won’t judge Pain Ahead! If you turn back from the journey now…
- 28. The Not-Recommended, All-Too-Familiar, Pain-for-Everyone, To-Be-Avoided Approach 28 • Business makes even more audacious commitments to catch up • Developers see more and more urgent projects coming in • All effort is spent on features as opposed to non-functional requirements • More shortcuts, more technical debt, more fragility • Deployments become more difficult – what took a weekend now takes 3 days! • We try to fix this by doing less deployments, increasing batch size • More moving parts, more failures… we are consumed by unplanned work Business starts missing commitments to the outside world, and then… This approach preordains us to failure Creates a permanent wedge between making urgent business changes and maintaining stability Working here is a major drag
- 29. Results: Puppet Labs State of DevOps 2014 Report Scientific study of relationship between organizational performance, IT performance, and DevOps practices 9200 respondents representing 110 countries 29 Findings DevOps adoption is accelerating High-performing IT organizations deploy code 30 times more frequently with 50% fewer failures Strong IT performance is a competitive advantage DevOps improves IT performance Organizational culture matters Job satisfaction is the No. 1 predictor of organizational performance High performing companies are good at getting better – nobody starts out high performing
- 30. 1st Way Emphasize entire system performance versus a specific silo of work 2nd Way Creating feedback loops 3rd Way Culture of continual experimentation Understanding that mastery requires practice The Three Ways of DevOps by Gene Kim The Three Ways describe the values that frame the processes of DevOps and they provide prescriptive steps
- 31. DevOps: The First Way 31
- 32. The First Way: Systems Thinking 32 Use systems thinking to ensure work always flows forward “Work moving backwards, or standing still, is almost always indicative of problems that need to be solved, and will span people, process and technology.” –Gene Kim
- 33. What is a silo, really? Disconnection from other people No shared context Different management 33 Barriers build up Different incentives Different objectives Bad handoffs Lack of understanding Lack of empathy “The nature of a large, complex organization is to fall out of alignment without deliberate effort – inertia pulls it apart” –Damon Edwards
- 34. The First Way: Understand the Flow of Work 34 Work starts with a description of features needed by the business Work ends with the stable, secure and reliable delivery of services to the customer Additional sources of work: IT finds defects Help desk fields incident reports Security raises compliance requirements Enterprise architecture initiatives e.g. single sign-on Visualize work Measure the flow of work (cycle time, lead time, wait times) Think about software production as a value stream similar to a manufacturing value stream
- 35. Organizations are Complex Systems 35 Human complex system Communication patterns Locations Work styles Personalities Roles and responsibilities Skill sets Technology complex system Programming Languages Tools Networks Configurations Interconnections One complex system working on another complex system
- 36. The First Way: Always Seek to Increase Flow 36 Reduce work in progress (WIP) Reduce batch size of deliveries Reduce variation in size of work items Make policies explicit Eliminate inventory and other waste Maintain a steady, sustainable pace Deliver often – and get really good at it
- 37. The First Way: Optimize Flow Globally, Not Locally Focus on interactions between parts of the system Build controls into the system Local efficiencies are good, but should never jeopardize global goals Avoid tribal warfare! Know your bottlenecks … and elevate them The bottleneck is the lever of control for speed of flow through your process 37 Upstream Queues to be serviced Bottleneck Flow is restricted Downstream Starved of full flow
- 38. The First Way: Never Consciously Pass Defects Downstream Create quality at the source Make rework visible Understand the origination point of defects in order to avoid recurrence 38 “This is legacy code, I’ll just make the change for my story, I don’t have time to fix the rest of this” “That issue is a doozy… leave it to fix in the hardening sprint” “Just push this feature over to the testers… it’s their job to find defects, right?” “Call the story done. We know there are still a few problems so just open up some defects against it”
- 39. The First Way: The System of Profound Knowledge Organizations are systems of interrelated processes and people which form the system’s components Components of the system must reinforce, not compete with each other to accomplish the aim of the system Workers’ success of depends on managing the balance between each component to optimize the system 39 Understand business goals – how value is achieved Understand people, processes, and technologies Understand risks and risk controls Understand cause and effect Make informed decisions based on rich, accurate, and timely information Teach the organization how to fix and regulate itself
- 40. The First Way: Bringing It All Together 40 Business Dev End Users Ops What is the minimum viable product? Is it profitable? Do we have the capability to build it and maintain it?
- 41. DevOps: The First Way – Practices 41 Communication Integration Collaboration
- 42. DevOps Practice: Deploy Shippable Environments SHIPPABLE CODE AND SHIPPABLE ENVIRONMENT 42 Communication Integration Collaboration Traditionally, dev is responsible for applications while ops is responsible for environments In DevOps, we use a single repository for everything –functional code, test code, environment configurations, and tool configurations
- 43. DevOps Practice: Infrastructure as Code 43 “Programmable infrastructure” “Fully automated configuration management” Code to automate provisioning Code to manage configurations
- 44. DevOps Practice: One Step Environment Creation 44 Provision and configure environments at the touch of a button Make production-like environments available early in the dev process Build code & environment at the same time Create a common dev, testing, and prod environment creation process Everyone uses a consistent environment Communication Integration Collaboration
- 45. DevOps Practice: The Daily Build “Heartbeat of the project” and “clean room every day” Rebuild every line of code from scratch – be able to reconstitute the system from “bare metal” Run all the tests! Check all dependencies Verify no defects introduced yesterday Build all versions Automate with Continuous Integration (CI) server 45 Communication Integration Collaboration
- 46. DevOps Practice: Deploy Early, Often, and Quickly 46 Small deployments mean Fast deployments mean more deployments mean easier deployments mean lower cycle times means faster time to market Communication Integration Collaboration
- 47. DevOps Practice: Classify Ops Work by Four Types Business Projects Internal IT Projects Changes Unplanned Work Types of work 47 Systematically allocating time to the 4 types enables all the work to get done and becomes routine Communication Integration Collaboration Exercise: Classify real USCIS work according to the four types
- 48. Doing DevOps at USCIS – First Way Understand and measure your flow of work using visualizations such as a Kanban board and value stream map Use a single USCIS repository for source code, test code, and environment configuration scripts Builds done by automated, script-driven retrieval of source code by a Continuous Integration (CI) server Frequent deployments – no less than every two weeks Consistent record of successful deployments Baked in accessibility and security compliance – no compliance work flowing backwards 48 What is the concept of a “Team-Managed Deployment” at USCIS?
- 49. DevOps: The Second Way 49
- 50. The Second Way: Amplify Feedback Loops 50 • Shorten and amplify “right to left” feedback loops • Use feedback to create even higher quality at the source • Create and embed knowledge where it is needed to provide immediate feedback • Understand needs of all customers, internal and external, and respond to their feedback The goal of any process improvement is to shorten and amplify feedback loops
- 51. The Second Way: Shorten and Amplify Feedback Loops 51 Develop Commit Test Build Product Backlog Issue Tracker Manual tester overloaded due to end of sprint Operations End Users Help Desk Triage Product Owner/ Value Team 10 steps to get feedback & VERY long delay
- 52. The Second Way: Shorten and Amplify Feedback Loops (cont) 52 Develop Commit Test Build Manual Test Issue Tracker 5 steps to get feedback
- 53. Most users won’t call… some may just quit being customers Many defects remain latent for a long time By the time defects come back, dev forgets how the code works 53 The Second Way: Shorten and Amplify Feedback Loops (cont) Develop Commit Test Build Failed Automated Test 4 Steps to get feedback – automated and quick!
- 54. The Second Way: Use Feedback to Create Quality at the Source Development is the source of quality – or problems As applications evolve, changes must not negatively impact end user experiences Developers need access to deep diagnostics so they can incorporate latest operational concerns and understand impact of their changes Traces from slow transactions that suggest performance bottlenecks in distributed applications Service-oriented architecture issues spanning multiple application tiers Correlation of application response times on end-user satisfaction levels Browser performance metrics Application response times Server usage Performance data by technology component Runtime code diagnostics including database queries
- 55. The Second Way: Create and Embed Knowledge 55 • Ops and Security: • Become part of the agile process – especially planning and prioritization • Provide recommendations and requirements as new code developed • Ensure relevant metrics are monitored early in the dev process • Dev participates in incident handling to acquire knowledge to prevent future problems: • Incident escalation • Root cause analyses • Post-mortems Communication Integration Collaboration • Ops receives cross training by dev and security • Extend agile practices to all teams • Visible work • Open meetings • Working agreements • Explicit policies
- 56. The Second Way: Respond to Needs of All Customers Use a service model for both internal and external customers Agile encouraged dev and test to focus on customer collaboration with business stakeholders and end users DevOps extends the service model to Dev and Ops treating each other as customers 56 Change Orientation Stability Orientation Customer Service Provider
- 57. DevOps: The Second Way - Practices 57 Communication Integration Collaboration
- 58. DevOps Practice: Deployment Automation Problems in deployment procedure will be found quickly and can be permanently eliminated Runs fast “smoke test” to ensure system is running as expected Built-in automatic rollback and/or redeploy Build confidence through frequent repetition – the prospect of deployments and rollback no longer instill fear Create a virtuous cycle of successful deployments, smaller deployments, and lower risk 58 Communication Integration Collaboration
- 59. DevOps Practice: Operations Monitoring Monitoring gives us continuous, live feedback about how the system is running 59 User Feedback Approach Monitored Approach Field user calls Automatic alert about a problem when it happens Multiple people call Monitoring tools show me how widespread the problem is Users can’t tell me the real source of the problem I can see which component of the application is generating errors Communication Integration Collaboration “Tell me what is happening before the phone rings”
- 60. Operations Monitoring – Needs and Challenges Monitoring Challenges Operators typically responsible for numerous applications Environments can be complex with unique or complicated application stacks Visibility into different components can be vague or non-existent Quantity of logged data can be overwhelming Combining monitoring tools into a single view that provides insight Monitoring Needs Active production monitoring, not just reacting to downtime Easily monitor critical areas of application stability with minimal tooling Tune dashboard to display key database, network, server, and application performance measurements in a holistic view Ability to quickly share potential performance issues with your team 60
- 61. DevOps Practice: Operations Monitoring Dashboard Application Response Time Application Performance Index (User Satisfaction) Application Throughput Alerts Transaction Timings (drill down capability to code level, transaction level) Communication Integration Collaboration
- 62. DevOps Practice: Operations Monitoring Drives Dev & Ops Priorities Communication Integration Collaboration
- 63. DevOps Practice: Prioritize Fixing Production Defects Prioritize fixing defects very fast 63 • Assume incidents will occur • Ensure ops feedback will come back rapidly • Ensure developers will get the info they need to fix the problem • Add automated tests to ensure problem cannot reoccur • Get really good at fixing defects very fast Communication Integration Collaboration
- 64. DevOps Practice: Reusable Ops and Security User Stories 64 Communication Integration Collaboration As security I want cross-site scripting attacks prevented so that access controls cannot be bypassed Estimate Priority 5 points 1 (High) • Verify all input is filtered as potentially malicious • Verify all output of the page is encoded to the explicitly defined character set • Verify output is sanitized by escaping dynamic content to properly enforce separation of code and data Acceptance Criteria User Story On back… As an ops engineer I want to monitor how many people are listening to audio feeds so I can tune playback quality during spikes in demand Estimate Priority 3 points 2 (Med) • Verify the count of active audio sessions is displayed in the application’s admin dashboard • Verify the count is accurate as playback sessions are added or completed Acceptance Criteria User Story On back…
- 65. DevOps Practice: Dev & Ops Common Communication System 65 Communication Integration Collaboration Remove all barriers to internal communication, collaboration, and integration Use common, intuitive dashboards combining information from all groups Key operational metrics Visible dev, ops, and security workflow (e.g. Kanban boards) List of recent and upcoming system changes Stability of the system Security status Schedules, planned release dates, and critical business dates Integrated alert policies Common internal note system – histories of defects and incidents can be very useful Shared wikis, file repositories, chat spaces, specs, and documentation
- 66. DevOps Practice: Track Dev & Ops Business Impact MTTR – Mean Time To Repair – How long is the system down? MTBF – Mean Time Between Failures – How often is the system down? USCIS Example: Key Performance Parameter (KPP) Service Agreement Low Threshold Objective Actual Reliability – uninterrupted correct function 641 hours 712 hours 739 hours Exceeded Objective Availability – 24/7 operations 97.63 % 98.88% 99.32% Exceeded Objective Maintainability – prompt restoration of service after outage No more than 10 hours No more than 8 hours 5 hours Exceeded Objective Communication Integration Collaboration
- 67. Doing DevOps at USCIS – Second Way Demonstrated information integration and collaboration between dev, ops, security, and business Partial or completely automated deployment – rapid, reliable, testable, repeatable Operational Monitoring Plan – preferably a dashboard Defined business impact measurements and thresholds See Team-Managed Deployment Management Instruction for more information 67
- 69. Systems to Make Software 69 Communication System Monitoring System Deploy System U Documentation System Issue Tracking i l Version Control (CM) ^ Requirements A R Build System a Test System Code Review System A good method of enabling DevOps is to simply begin connecting and automation the systems you use to make software. • Start where you are • Identify possible interconnections • Research tools to automate • Create future state roadmap • Let pipeline emerge • Continue to improve the sequence of connections as systems change
- 70. Version Control 70 Communication System Monitoring System Deploy System U Documentation System Issue Tracking i l Version Control (CM) ^ Requirements A R Build System a Test System Code Review System Version Control Ensures you’re working on the right version of something
- 71. Requirements System 71 Communication System Monitoring System Deploy System U Documentation System Issue Tracking i l Version Control (CM) ^ Requirements A R Build System a Test System Code Review System Requirements System Houses project requirements in a prioritized list and allows for item allocation to sprint/team member; Allows for traceability of dependencies between stories
- 72. Build System 72 Communication System Monitoring System Deploy System U Documentation System Issue Tracking i l Version Control (CM) ^ Requirements A R Build System a Test System Code Review System Build System Software tools designed to automate the process of program compilation to create a deployable package
- 73. Test System 73 Communication System Monitoring System Deploy System U Documentation System Issue Tracking i l Version Control (CM) ^ Requirements A R Build System a Test System Code Review System Test System orchestrated set of tests, both manual and automated that ensure the functionality and accuracy of code
- 74. Code Review System 74 Communication System Monitoring System Deploy System U Documentation System Issue Tracking i l Version Control (CM) ^ Requirements A R Build System a Test System Code Review System Code Review System Ensures that code complies with standards and identifies low level bugs and coding errors; Identifies design and requirements compliance
- 75. Issue Tracking System 75 Communication System Monitoring System Deploy System U Documentation System Issue Tracking i l Version Control (CM) ^ Requirements A R Build System a Test System Code Review System Issue Tracking System Collects issues throughout the cycle of the project and track them through completion
- 76. Documentation System 76 Communication System Monitoring System Deploy System U Documentation System Issue Tracking i l Version Control (CM) ^ Requirements A R Build System a Test System Code Review System Documentation System Repository of system information throughout the lifecycle
- 77. Deploy System 77 Communication System Monitoring System Deploy System U Documentation System Issue Tracking i l Version Control (CM) ^ Requirements A R Build System a Test System Code Review System Deploy System Installs and configures the package created by the build system into appropriate environments
- 78. Monitoring 78 Communication System Monitoring System Deploy System U Documentation System Issue Tracking i l Version Control (CM) ^ Requirements A R Build System a Test System Code Review System Monitoring System Collects current-state data to determine health of all environments
- 79. Communication System 79 Communication System Monitoring System Deploy System U Documentation System Issue Tracking i l Version Control (CM) ^ Requirements A R Build System a Test System Code Review System Communication System Method for conveying information between systems
- 80. Automate All the Connections! 80 Communication System Monitoring System Deploy System U Documentation System Issue Tracking i l Version Control (CM) ^ Requirements A R Build System a Test System Code Review System
- 81. Orchestrating Integration with a Pipeline
- 82. Pipelines User Commits Merge code Build Unit test/coverage Code Review Log Issues Deploy 82 A Pipeline is a chain of tasks that can be automated Integration tools use pipelines to perform tasks repetitively and continuously The process is called Continuous Integration (CI) Pipelines keep work flowing forward in our DevOps system
- 83. 83 Communication System Monitoring System Deploy System U Documentation System Issue Tracking i l Version Control (CM) ^ Requirements A R Build System a Test System Code Review System Pipeline Orchestration We Need Something to Integrate the Systems
- 84. 84 Communication System Monitoring System Deploy System U Documentation System Issue Tracking i l Version Control (CM) ^ Requirements A R Build System a Test System Code Review System Pipeline Orchestration Development Pipeline Example
- 85. 85 Communication System Monitoring System Deploy System U Documentation System Issue Tracking i l Version Control (CM) ^ Requirements A R Build System a Test System Code Review System Code is committed and Merged Initiate Build Initiate Testing Development Pipeline Example with Integration System Commit code Pipeline Orchestration
- 86. Pipeline Stages Code Done Unit Tests Integrate Acceptance Testing Deploy to Production 86 Continuous Delivery Auto Auto Auto Manual Code Done Unit Tests Integrate Acceptance Testing Deploy to Production Continuous Deployment Auto Auto Auto Auto Code Done Unit Tests Integrate Continuous Integration Auto Auto
- 87. CI Pipeline Example 87 CI Pipeline CM Repository Staging Integration Master STOP STOP STOP Fail Fail Fail Success Success Success Commit Commit Commit New Feature (NF) Legacy Feature (LF) Bad Feature (BF) Build gate Compile Code Code Quality Gates Applied Smoke Tests If Successful, Merge with Staging Staging gate Compile Code Functional Tests If Successful, Merge with Integration Branch Integration gate Compile Code Merge with Master Fortify Scans Release is packaged
- 88. Using a CI/CD Pipeline for Team-Managed Deployments at USCIS RRR eRRR TMD Deploy Manual Test Auto. Test Auto. Build Development Operations OR OR Approval: CI/CD Pipeline: DevOps: Team Managed Deployment (TMD) provides the approval step for a CI/CD Pipeline. The CI/CD Pipeline provides the forward link from Development to Operations.
- 89. Using a CI/CD Pipeline for Team-Managed Deployments at USCIS (cont) RRR Documents CI/CD Pipeline Artifacts VDD Release Number Source Code File List List of Changes Deployment Instructions TAS Test Results Test History Pipeline Release Number Deployment Scripts Version Control Source Code File List List of Changes Test Tools Test Results Test History Automation used in a CI/CD pipeline allows data to be collected as true artifacts. In an RRR approach, the information is manually collected into documents.
- 90. DevOps: The Third Way 90
- 91. DevOps is Not… 91 A tool A role A team Something that can be purchased or simply switched on DevOps requires a culture of operations and development engineers participating together in the entire service lifecycle
- 92. • Continual experimentation, taking risks, and learning from failure • Understanding that repetition is the prerequisite to master The Third Way: Culture of Improvement 92
- 93. The Third Way: Experimentation, Risk-Taking, and Learning 93 Develop a culture that pushes into the danger zone Develop habits to survive danger Build experimentation, risk-taking, and learning into our way of doing business Break things early and often Intuit ran 165 experiments on their TurboTax product in the 3 main months of tax season – ideas made it to market a year earlier and they increased customer conversion rate by 50%
- 94. The Third Way: Repetition for Mastery 94 • Do the hard parts often • Work through pain points to make the process easier • Do painful things MORE frequently to make it less painful • Reduce anxiety about unexpected outcomes • Automate!!!
- 95. DevOps: The Third Way - Practices 95 Communication Integration Collaboration
- 96. DevOps Practice: Inject Failures 96 • Netflix services are hosted completely in Amazon Web Services cloud • Design each distributed system to expect and tolerate failure • Chaos Monkey randomly kills services within architecture in order to learn to tolerate and respond to failure DevOps Approach How does this system react if I do this? Can we continue operations without this server? Will the users prefer option A or option B? Traditional Approach Not in my system, you don’t Not in my system, you don’t Not in my system, you don’t Communication Integration Collaboration
- 97. DevOps Practice: Make Your Improvement Work Visible 97 Along with regular user stories, use colored cards to indicate: • Technical debt • Unplanned work • Experiments • Learning backlog Allocate time to improve daily work Track the work needed to maintain overall health of the system Communication Integration Collaboration
- 98. DevOps Practice: Regularly Improve Technical Debt 98 Allocate 20% of cycles to Technical Debt Reduction • Write tests to find misconfigurations – and fix them • Constantly run automated static code analysis during continuous integration and testing, and raise the bar in your quality gates • Enforce consistency in code, environments, and configurations • Repeatedly tackle the hard stuff Communication Integration Collaboration
- 99. DevOps Practice: Regularly Improve Tools 99 Good tools are key to enabling DevOps collaboration, automation, and visibility Provide teams the best tools available Regularly invest time researching and piloting new tools Provide expert support Communication Integration Collaboration
- 100. DevOps Practice: Reward Contributions to a DevOps Culture Incentivize DevOps practices and behaviors Recognize experimentation and risk-taking that leads to valuable learning Model honest self-assessment of organizational strengths and weaknesses and use of improvement techniques such as Toyota Kata Quantify and promote the link between DevOps practices and organizational performance 100 Communication Integration Collaboration
- 101. DevOps Practice: Conduct Deliberate Culture Change Experiments 101 Org Change Patterns Decentralized, emergent Protected, dedicated team Organizational baby steps Champions / advocates Boss’ orders Communication Integration Collaboration Discussion: What are our biggest cultural challenges? What experiments should we run?
- 102. DevOps Team Profiles DevOps Team Member End-to-end viewpoint Contributes to and uses visibility Automator Collaborative, cross-functional, friction reducer Participates in collective ownership of code and code delivery Personal success = team success Enjoys working this way 102 DevOps Expert Support Team Helps introduce DevOps- supportive processes and tools Works with teams to automate environment creation and deployment Helps teams use operational performance logs and dashboards Provides infrastructure support
- 103. Food for Thought – Maturing DevOps Practices Level 1 Level 2 Level 3 Level 4 Level 5 Culture & Processes Frequent commits Prioritized work Defined & documented process One backlog team and a master backlog Adopt basic agile methods Remove boundary of dev & test Team collaboration Remove boundary of dev & ops Act on metrics Common processes for all changes Dedicated tools team for automation Deploy disconnected from Release Continuous improvement Cross functional teams No rollbacks Architecture Define Context View, Logical Composition & Physical Composition Stage SDD wiki with Views in place Define related views Review process in place ? ? Build / Deploy Versioned code base Scripted builds Basic scheduled builds Dedicated build server Documented manual deploy Poling builds Build are stored Manual Tag & Versioning First step towards standardized deploys Auto triggered builds Automated tag & versioning Automated bulk of DB changes Basic pipeline with deploy to prod Scripted configuration changes Standard process for all environments Zero downtime deploy Multiple build machines Full automated DB deploys Zero touch continuous deployments Test & Verification Automated Unit Tests (Coverage <50%) Separate test environment Automated Unit Tests (Coverage >50% & < 80%) Automated Integration Tests (Coverage ??) Automated Functional tests Automated acceptance criteria (<40%) Automated acceptance criteria (80%) Automated performance tests Automated Security Tests Automated Section 508 tests All tests automated Coverage 100% Collaboration & Information Sharing Baseline process metrics Manual reporting Static code analysis Quality reports History of reports available Traceability built into pipeline Report trend analysis Graphing as a service Reports accessible via common dashboard Dynamic graphing 103
- 104. Wrap Up Questions 104 How much more productive, effective, and enjoyable might our work be? How much business value is left on the table due to unmatched demand and capacity? Can we afford not to do DevOps?