Data Loading...
Learning DevOps The complete guide to accelerate collaboration with Jenkins, Kubernetes, Terraform and Azure DevOps
Mikael Krief
www.packt.com
FOR SALE IN INDIA ONLY
Learning DevOps
The complete guide to accelerate collaboration with Jenkins, Kubernetes, Terraform and Azure DevOps
Mikael Krief
BIRMINGHAM - MUMBAI
Learning DevOps Copyright © 2019 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Commissioning Editor: Vijin Boricha Acquisition Editor: Meeta Rajani Content Development Editor: Drashti Panchal Senior Editor: Arun Nadar Technical Editor: Prachi Sawant Copy Editor: Safis Editing Project Coordinator: Vaidehi Sawant Proofreader: Safis Editing Indexer: Tejal Daruwale Soni Production Designer: Nilesh Mohite First published: October 2019 Production reference: 1251019 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-83864-273-0
www.packt.com
I would like to dedicate this book to my wife and children, who are my source of happiness.
Foreword Having discussed DevOps with Mikael Krief on several occasions, it is clear that he understands the importance of empowering both Dev and Ops in order to deliver value. DevOps is the union of people, processes, and products to enable the continuous delivery of value to our end users. Value is the most important word of that definition. DevOps is not about software, automation, shipping a feature, or getting to the bottom of your product backlog. It is about delivering value. To deliver value, you must measure your application while it is running in production and use the telemetry to guide what you deliver next. To deliver value, your team must fully embrace the culture of DevOps. The hardest part of DevOps is the people part: building the culture that is required to succeed. Learning DevOps does a great job of focusing on the culture behind DevOps. To succeed, you must change the way your team thinks about their roles. Everyone must have a common goal that encourages collaboration. Delivering value to the end user is the responsibility of everyone involved in the application. Our community tends to spend more time on the Dev side of DevOps. Learning DevOps, however, has invested considerable time on Infrastructure as Code. As more workloads move to the cloud, IaC becomes more valuable. The ability to provision and configure your infrastructure as part of your pipeline allows engineers to innovate. IaC can save companies money by shutting down environments when they are no longer in use or simply provisioning them on demand. Once your entire infrastructure is stored in version control and acted upon via your pipeline, recovering from a disaster is simply a deployment. The time to debate whether you should or should not implement DevOps is over. You either implement DevOps or you lose. Donovan Brown Principal Cloud Advocate Manager at Microsoft
Packt.com
Subscribe to our online digital library for full access to over 7,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.
Why subscribe? Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals Improve your learning with Skill Plans built especially for you Get a free eBook or video every month Fully searchable for easy access to vital information Copy and paste, print, and bookmark content Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.packt.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details. At www.packt.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.
Contributors About the author Mikael Krief lives in France and works as a DevOps engineer, and for 4 years he has worked as a DevOps consultant and DevOps technical officer at an expert consulting company in Microsoft technologies. He is passionate about DevOps culture and practices, ALM, and Agile methodologies. He loves to share his passion through various communities, such as the ALM | DevOps Rangers community, which he has been a member of since 2015. He also contributes to many open source projects, writes blogs and books, speaks at conferences, and publishes public tools such as extensions for Azure DevOps. For all his contributions and passion in this area, he has received the Microsoft© Most Valuable Professional (MVP) award for the last 4 years. I would like to extend my thanks to my family for accepting that I needed to work long hours on this book during family time. I would like to thank Meeta Rajani for giving me the opportunity to write this book, which was a very enriching experience. Special thanks to Drashti Panchal, Prachi Sawant, Arun Nadar for their valuable input and time reviewing this book and to the entire Packt team for their support during the course of writing this book.
About the reviewers Abhinav Krishna Kaiser manages in a leading consulting firm. He is a published author and has penned three books on DevOps, ITIL, and IT communication. Abhinav has transformed multiple programs into the DevOps ways of working and is one of the leading DevOps architects on the circuit today. He has assumed the role of an Agile Coach to set the course for Agile principles and processes in order to set the stage in development. Apart from DevOps and Agile, Abhinav is an ITIL expert and is a popular name in the field of IT service management. Abhinav's latest publication, on recasting ITIL with the DevOps processes, came out in 2018. Reinventing ITIL in the Age of DevOps transforms the ITIL framework to work in a DevOps project. His earlier publication, Become ITIL Foundation Certified in 7 Days, is one of the top guides for IT professionals looking to become ITIL Foundation certified and to those getting into the field of service management. Abhinav started consulting with clients 15 years ago on IT service management, where he created value by developing robust service management solutions. Moving with the times, he eventually went into DevOps and Agile consulting. He is one of the foremost authorities in the area of configuration management and his solutions have stood the test of time, rigor, and technological advancements. Abhinav blogs and writes guides and articles on DevOps, Agile, and ITIL on popular sites. While the life of a consultant is to go where the client is, currently he is based in London, UK. He is from Bangalore, India, and is happily married with a daughter and a son.
Ebru Cucen works as a technical principal consultant at Contino, and is also a public speaker and trainer on Serverless. She has a BSc in mathematics and started her journey as a .NET developer/trainer in 2004. She has over 10 years of experience in digital transformation of financial enterprise companies. She's spent the last 5 years working with the cloud, covering the full life cycle of feature development/deployment and CI/CD pipelines. Being a lifetime student, she loves learning, exploring, and experimenting with technology to understand and use it to make our lives better. She enjoys living in London with her 7-year-old son and her husband, Tolga Cucen, to whom she is thankful for supporting her during the nights/weekends she has worked on this book.
Packt is searching for authors like you If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.
Table of Contents Preface
1
Section 1: DevOps and Infrastructure as Code Chapter 1: DevOps Culture and Practices Getting started with DevOps Implementing CI/CD and continuous deployment Continuous integration (CI) Implementing CI
Continuous delivery (CD) Continuous deployment
Understanding IaC practices The benefits of IaC IaC languages and tools Scripting types Declarative types
The IaC topology
The deployment and provisioning of the infrastructure Server configuration Immutable infrastructure with containers Configuration and deployment in Kubernetes
IaC best practices
Summary Questions Further reading Chapter 2: Provisioning Cloud Infrastructure with Terraform Technical requirements Installing Terraform Manual installation Installation by script
Installing Terraform by script on Linux Installing Terraform by script on Windows Installing Terraform by script on macOS
Integrating Terraform with Azure Cloud Shell
Configuring Terraform for Azure
Creating the Azure SP Configuring the Terraform provider
Terraform configuration for local development and testing
Writing a Terraform script to deploy Azure infrastructure Following some Terraform good practices
9 10 13 13 14 16 18 20 20 21 21 21 23 23 23 25 25 26 28 29 29 31 31 32 32 33 33 34 37 37 39 39 41 42 43 47
Table of Contents
Better visibility with the separation of files Protection of sensitive data Dynamizing the code with variables and interpolation functions
Deploying the infrastructure with Terraform Initialization Previewing changes Applying the changes
Terraform command lines and life cycle Using destroy to better rebuild Formatting and validating the code Formatting the code Validating the code
Terraform's life cycle in a CI/CD process
Protecting tfstate in a remote backend Summary Questions Further reading Chapter 3: Using Ansible for Configuring IaaS Infrastructure Technical requirements Installing Ansible Installing Ansible with a script Integrating Ansible into Azure Cloud Shell Ansible artifacts Configuring Ansible
Creating an inventory for targeting Ansible hosts The inventory file Configuring hosts in the inventory Testing the inventory
Writing the first playbook
Writing a basic playbook Understanding Ansible modules Improving your playbooks with roles
Executing Ansible
Using the preview or dry run option Increasing the log level output
Protecting data with Ansible Vault
Using variables in Ansible for better configuration Protecting sensitive data with Ansible Vault
Using a dynamic inventory for Azure infrastructure Summary Questions Further reading Chapter 4: Optimizing Infrastructure Deployment with Packer [ ii ]
47 47 48 49 51 52 54 56 56 58 58 59 60 62 66 67 67
69 70 70 71 73 74 75 77 77 79 80 82 82 83 84 86 88 89 90 90 94 96 105 105 105 107
Table of Contents
Technical requirements An overview of Packer Installing Packer
Installing manually Installing by script
Installing Packer by script on Linux Installing Packer by script on Windows Installing Packer by script on macOS Integrating Packer with Azure Cloud Shell Checking the Packer installation
Creating Packer templates for Azure VMs with scripts The structure of the Packer template The builders section The provisioners section The variables section
Building an Azure image with the Packer template
Using Ansible in a Packer template
Writing the Ansible playbook Integrating an Ansible playbook in a Packer template
Executing Packer
Configuring Packer to authenticate to Azure Checking the validity of the Packer template Running Packer to generate our VM image
Using a Packer image with Terraform Summary Questions Further reading
108 109 109 109 110 110 111 112 112 113 114 114 115 116 118 120 123 123 124 125 126 127 127 130 131 132 132
Section 2: DevOps CI/CD Pipeline Chapter 5: Managing Your Source Code with Git Technical requirements Overviewing Git and its command lines Git installation Configuration Git Git vocabulary Git command lines
Retrieving a remote repository Initializing a local repository Configuring a local repository Adding a file for the next commit Creating a commit Updating the remote repository Synchronizing the local repository from the remote Managing branches
Understanding the Git process and GitFlow pattern Starting with the Git process
[ iii ]
135 136 136 139 144 144 145 146 146 146 146 147 147 148 148 149 150
Table of Contents
Creating and configuring a Git repository Committing the code Archiving on the remote repository Cloning the repository The code update Retrieving updates
Isolating your code with branches Branching strategy with GitFlow The GitFlow pattern GitFlow tools
Summary Questions Further reading Chapter 6: Continuous Integration and Continuous Delivery Technical requirements The CI/CD principles Continuous integration (CI) Continuous delivery (CD)
Using a package manager
Private NuGet and npm repository Nexus Repository OSS Azure Artifacts
Using Jenkins
Installing and configuring Jenkins Configuring a GitHub webhook Configuring a Jenkins CI job Executing the Jenkins job
Using Azure Pipelines
Versioning of the code with Git in Azure Repos Creating the CI pipeline Creating the CD pipeline: the release
Using GitLab CI
Authentication at GitLab Creating a new project and managing your code source Creating the CI pipeline Accessing the CI pipeline execution details
Summary Questions Further reading
150 154 155 157 158 158 159 162 163 164 166 166 167
169 170 170 171 171 172 174 174 175 177 177 179 181 185 186 188 190 200 207 208 209 214 215 217 218 218
Section 3: Containerized Applications with Docker and Kubernetes Chapter 7: Containerizing Your Application with Docker Technical requirements [ iv ]
221 222
Table of Contents
Installing Docker
Registering on Docker Hub Docker installation An overview of Docker's elements
Creating a Dockerfile
Writing a Dockerfile Dockerfile instructions overview
Building and running a container on a local machine Building a Docker image Instantiating a new container of an image Testing a container locally
Pushing an image to Docker Hub Deploying a container to ACI with a CI/CD pipeline The Terraform code for ACI Creating a CI/CD pipeline for the container
Summary Questions Further reading Chapter 8: Managing Containers Effectively with Kubernetes Technical requirements Installing Kubernetes Kubernetes architecture overview Installing Kubernetes on a local machine Installing the Kubernetes dashboard
First example of Kubernetes application deployment Using HELM as a package manager Using AKS Creating an AKS service Configuring kubectl for AKS Advantages of AKS
Creating a CI/CD pipeline for Kubernetes with Azure Pipelines The build and push of the image in the Docker Hub Automatic deployment of the application in Kubernetes
Summary Questions Further reading
222 223 224 229 229 230 231 232 232 234 235 235 239 240 241 250 250 251 253 254 254 255 256 257 261 265 269 270 271 272 273 274 280 283 283 284
Section 4: Testing Your Application Chapter 9: Testing APIs with Postman Technical requirements Creating a Postman collection with requests Installation of Postman Creating a collection
[v]
287 288 288 290 290
Table of Contents
Creating our first request
Using environments and variables to dynamize requests Writing Postman tests Executing Postman request tests locally Understanding the Newman concept Preparing Postman collections for Newman Exporting the collection Exporting the environments
Running the Newman command line Integration of Newman in the CI/CD pipeline process Build and release configuration Npm install Npm run newman Publish test results
The pipeline execution
Summary Questions Further reading Chapter 10: Static Code Analysis with SonarQube Technical requirements Exploring SonarQube Installing SonarQube Overview of the SonarQube architecture Installing SonarQube Manual installation of SonarQube Installation via Docker Installation in Azure
Real-time analysis with SonarLint Executing SonarQube in continuous integration
Configuring SonarQube Creating a CI pipeline for SonarQube in Azure Pipelines
Summary Questions Further reading Chapter 11: Security and Performance Tests Technical requirements Applying web security and penetration testing with ZAP Using ZAP for security testing Ways to automate the execution of ZAP
Running performance tests with Postman Summary Questions Further reading [ vi ]
292 296 298 301 305 307 307 309 310 313 314 316 317 318 319 321 321 321 323 324 324 325 325 326 327 327 328 332 335 335 337 341 341 341 343 344 344 345 348 350 352 353 353
Table of Contents
Section 5: Taking DevOps Further Chapter 12: Security in the DevOps Process with DevSecOps Technical requirements Testing Azure infrastructure compliance with Chef InSpec Overview of InSpec Installing InSpec Configuring Azure for InSpec Writing InSpec tests
Creating an InSpec profile file Writing compliance InSpec tests
Executing InSpec
Using the Secure DevOps Kit for Azure Installing the Azure DevOps Security Kit Checking the Azure security using AzSK Integrating AzSK in Azure Pipelines
Preserving data with HashiCorp's Vault Installing Vault locally Starting the Vault server Writing secrets in Vault Reading secrets in Vault Using the Vault UI web interface Getting Vault secrets in Terraform
Summary Questions Further reading Chapter 13: Reducing Deployment Downtime Technical requirements Reducing deployment downtime with Terraform Understanding blue-green deployment concepts and patterns
Using blue-green deployment to improve the production environment Understanding the canary release pattern Exploring the dark launch pattern
Applying blue-green deployments on Azure Using App Service with slots Using Azure Traffic Manager
Introducing feature flags Using an open source framework for feature flags Using the LaunchDarkly solution Summary Questions Further reading Chapter 14: DevOps for Open Source Projects [ vii ]
357 358 359 360 360 362 363 364 365 366 369 369 370 373 377 378 380 382 383 385 388 392 393 393 395 396 396 399 400 400 401 402 402 404 406 408 413 418 418 418 421
Learning DevOps The implementation of DevOps processes requires the efficient use of various tools, and the choice of these tools is crucial for the sustainability of projects and collaboration between development (Dev) and operations (Ops). This book presents the different patterns and tools that you can use to provision and configure an infrastructure in the cloud. You'll begin by understanding DevOps culture, the application of DevOps in cloud infrastructure, provisioning with Terraform, configuration with Ansible, and image building with Packer. You'll then be taken through source code versioning with Git and the construction of a DevOps CI/CD pipeline
using Jenkins, GitLab CI, and Azure Pipelines. This DevOps handbook will also guide you in containerizing and deploying your applications with Docker and Kubernetes. You'll learn how to reduce deployment downtime with blue-green deployment and the feature flags technique, and study DevOps practices for open source projects. Finally, you'll grasp some best practices for reducing the overall application lead time to ensure faster time to market. By the end of this book, you'll have built a solid foundation in DevOps, and developed the skills necessary to enhance a traditional software delivery process using modern software delivery tools and techniques.
Things you will learn: •
Become well versed with DevOps culture and its practices
•
Build a DevOps pipeline with Jenkins, Azure Pipelines, and GitLab CI
•
Use Terraform and Packer for cloud infrastructure provisioning
•
Containerize your applications with Docker and Kubernetes
•
Implement Ansible for infrastructure configuration
•
Check application quality with SonarQube and Postman
•
Use basic Git commands and understand the Git flow process
•
Protect DevOps processes and applications using DevSecOps tools
www.packt.com
FOR SALE IN INDIA ONLY