The DevOps 2.0 Toolkit: Automating the Continuous Deployment Pipeline with Containerized Microservices

This book is about different techniques that help us architect software in a better and more efficient way with microservices packed as immutable containers, tested and deployed continuously to servers that are automatically provisioned with configuration management tools. It's about fast, reliable and continuous deployments with zero-downtime and ability to roll-back. It's about scaling to any number of servers, design of self-healing systems capable of recuperation from both hardware and software failures and about centralized logging and monitoring of the cluster.In other words, this book envelops the whole microservices development and deployment lifecycle using some of the latest and greatest practices and tools. We'll use Docker, Kubernetes, Ansible, Ubuntu, Docker Swarm and Docker Compose, Consul, etcd, Registrator, confd, and so on. We'll go through many practices and even more tools. Finally, while there will be a lot of theory, this is a hands-on book. You won't be able to complete it by reading it in a metro on a way to work. You'll have to read this book while in front of the computer and get your hands dirty. Table of Contents 4 Preface 8 Overview 10 Audience 11 About the Author 12 The DevOps Ideal 13 Continuous Integration, Delivery, and Deployment 13 The Light at the End of the Deployment Pipeline 16 The Implementation Breakthrough: Continuous Deployment, Microservices, and Containers 18 Continuous Integration 18 Continuous Delivery and Deployment 25 Microservices 30 Containers 30 The Three Musketeers: Synergy of Continuous Deployment, Microservices, and Containers 32 System Architecture 34 Monolithic Applications 34 Services Split Horizontally 37 Microservices 38 Monolithic Applications and Microservices Compared 40 Deployment Strategies 42 Microservices Best Practices 51 Final Thoughts 56 Setting Up the Development Environment With Vagrant and Docker 58 Combining Microservice Architecture and Container Technology 58 Vagrant and Docker 60 Development Environment Setup 62 Implementation of the Deployment Pipeline: Initial Stages 70 Spinning Up the Continuous Deployment Virtual Machine 70 Deployment Pipeline Steps 71 The Checklist 84 Configuration Management in the Docker World 86 CFEngine 86 Puppet 86 Chef 87 Ansible 87 Final Thoughts 88 Configuring the Production Environment 89 Setting Up the Ansible Playbook 92 Implementation of the Deployment Pipeline: Intermediate Stages 96 Deploying Containers to the Production Server 97 Docker UI 102 The Checklist 103 Service Discovery: The Key to Distributed Services 105 Service Registry 107 Service Registration 107 Service Discovery 109 Service Discovery Tools 110 Manual Configuration 111 Zookeeper 111 etcd 112 Consul 128 Service Discovery Tools Compared 149 Proxy Services 151 Reverse Proxy Service 151 How Can Proxy Service Help Our Project? 153 nginx 154 HAProxy 166 Proxy Tools Compared 172 Implementation of the Deployment Pipeline: The Late Stages 175 Starting the Containers 177 Integrating the Service 179 Running Post-Deployment Tests 180 Pushing the Tests Container to the Registry 181 The Checklist 182 Automating Implementation of the Deployment Pipeline 184 Deployment Pipeline Steps 184 The Playbook and the Role 188 Pre-Deployment Tasks 189 Deployment Tasks 192 Post-Deployment Tasks 195 Running the Automated Deployment Pipeline 196 Continuous Integration (CI), Delivery and Deployment (CD) Tools 198 CI/CD Tools Compared 198 The Short History of CI/CD Tools 200 Jenkins 202 Final Thoughts 227 Blue-Green Deployment 229 The Blue-Green Deployment Process 230 Manually Running the Blue-Green Deployment 233 Automating the Blue-Green Deployment with Jenkins Workflow 250 Clustering And Scaling Services 260 Scalability 260 Axis Scaling 262 Clustering 265 Docker Clustering Tools Compared: Kubernetes vs Docker Swarm vs Mesos 266 Docker Swarm Walkthrough 271 Deploying with Docker Swarm Without Links 285 Scaling Services with Docker Swarm 292 Scheduling Containers Depending on Reserved CPUs and Memory 294 Automating Deployment With Docker Swarm and Ansible 299 Self-Healing Systems 307 Self-Healing Levels and Types 308 Self-Healing Architecture 313 Self-Healing with Docker, Consul Watches, and Jenkins 320 Combining On-Premise with Cloud Nodes 353 Self-Healing Summary (So Far) 353 Centralized Logging and Monitoring 355 The Need for Centralized Logging 356 Sending Log Entries to ElasticSearch 358 Parsing Log Entries 366 Sending Log Entries to a Central LogStash Instance 369 Sending Docker Log Entries to a Central LogStash Instance 375 Self-Healing Based on Software Data 388 Logging Hardware Status 394 Self-Healing Based on Hardware Data 400 Final Thoughts 401 Farewell 403