The Practice of Cloud System Administration: Devops and Sre Practices for Web Services, Volume 2

"There's an incredible amount of depth and thinking in the practicesdescribed here, and it's impressive to see it all in one place." " -Win Treese, coauthor of Designing Systems for Internet Commerce " " The Practice of Cloud System Administration, Volume 2, " focuses on "distributed" or "cloud" computing and brings a DevOps/SRE sensibility to the practice of system administration. Unsatisfied with books that cover either design or operations in isolation, the authors created this authoritative reference centered on a comprehensive approach. Case studies and examples from Google, Etsy, Twitter, Facebook, Netflix, Amazon, and other industry giants are explained in practical ways that are useful to all enterprises. The new companion to the best-selling first volume, "The Practice of System and Network Administration, Second Edition, " this guide offers expert coverage of the following and many other crucial topics: Designing and building modern web and distributed systems Fundamentals of large system design Understand the new software engineering implications of cloud administration Make systems that are resilient to failure and grow and scale dynamically Implement DevOps principles and cultural changes IaaS/PaaS/SaaS and virtual platform selection Operating and running systems using the latest DevOps/SRE strategies Upgrade production systems with zero down-time What and how to automate; how to decide what not to automate On-call best practices that improve uptime Why distributed systems require fundamentally different system administration techniques Identify and resolve resiliency problems before they surprise you Assessing and evaluating your team's operational effectiveness Manage the scientific process of continuous improvement A forty-page, pain-free assessment system you can start using today Cover......Page 1 Title Page......Page 4 Copyright Page......Page 5 Contents......Page 8 Preface......Page 24 About the Authors......Page 30 Introduction......Page 32 Part I: Design: Building It......Page 38 1 Designing in a Distributed World......Page 40 1.1 Visibility at Scale......Page 41 1.2 The Importance of Simplicity......Page 42 1.3 Composition......Page 43 1.4 Distributed State......Page 48 1.5 The CAP Principle......Page 52 1.6 Loosely Coupled Systems......Page 55 1.7 Speed......Page 57 1.8 Summary......Page 60 Exercises......Page 61 2.1 Operational Requirements......Page 62 2.2 Implementing Design for Operations......Page 76 2.3 Improving the Model......Page 79 2.4 Summary......Page 80 Exercises......Page 81 3 Selecting a Service Platform......Page 82 3.1 Level of Service Abstraction......Page 83 3.2 Type of Machine......Page 87 3.3 Level of Resource Sharing......Page 93 3.4 Colocation......Page 96 3.5 Selection Strategies......Page 97 Exercises......Page 99 4 Application Architectures......Page 100 4.1 Single-Machine Web Server......Page 101 4.2 Three-Tier Web Service......Page 102 4.3 Four-Tier Web Service......Page 108 4.5 Cloud-Scale Service......Page 111 4.6 Message Bus Architectures......Page 116 4.7 Service-Oriented Architecture......Page 121 4.8 Summary......Page 123 Exercises......Page 124 5 Design Patterns for Scaling......Page 126 5.1 General Strategy......Page 127 5.2 Scaling Up......Page 129 5.3 The AKF Scaling Cube......Page 130 5.4 Caching......Page 135 5.5 Data Sharding......Page 141 5.6 Threading......Page 143 5.7 Queueing......Page 144 5.8 Content Delivery Networks......Page 145 Exercises......Page 147 6 Design Patterns for Resiliency......Page 150 6.1 Software Resiliency Beats Hardware Reliability......Page 151 6.2 Everything Malfunctions Eventually......Page 152 6.3 Resiliency through Spare Capacity......Page 155 6.4 Failure Domains......Page 157 6.5 Software Failures......Page 159 6.6 Physical Failures......Page 162 6.7 Overload Failures......Page 169 6.8 Human Error......Page 172 6.9 Summary......Page 173 Exercises......Page 174 Part II: Operations: Running It......Page 176 7 Operations in a Distributed World......Page 178 7.1 Distributed Systems Operations......Page 179 7.2 Service Life Cycle......Page 186 7.3 Organizing Strategy for Operational Teams......Page 191 7.4 Virtual Office......Page 197 7.5 Summary......Page 198 Exercises......Page 199 8 DevOps Culture......Page 202 8.1 What Is DevOps?......Page 203 8.2 The Three Ways of DevOps......Page 207 8.3 History of DevOps......Page 211 8.4 DevOps Values and Principles......Page 212 8.5 Converting to DevOps......Page 217 8.6 Agile and Continuous Delivery......Page 219 8.7 Summary......Page 223 Exercises......Page 224 9 Service Delivery: The Build Phase......Page 226 9.1 Service Delivery Strategies......Page 228 9.2 The Virtuous Cycle of Quality......Page 231 9.3 Build-Phase Steps......Page 233 9.5 Continuous Integration......Page 236 9.6 Packages as Handoff Interface......Page 238 9.7 Summary......Page 239 Exercises......Page 240 10.1 Deployment-Phase Steps......Page 242 10.2 Testing and Approval......Page 245 10.4 Infrastructure Automation Strategies......Page 248 10.6 Infrastructure as Code......Page 252 10.8 Summary......Page 253 Exercises......Page 254 11.1 Taking the Service Down for Upgrading......Page 256 11.2 Rolling Upgrades......Page 257 11.3 Canary......Page 258 11.4 Phased Roll-outs......Page 260 11.7 Toggling Features......Page 261 11.8 Live Schema Changes......Page 265 11.10 Continuous Deployment......Page 267 11.11 Dealing with Failed Code Pushes......Page 270 11.12 Release Atomicity......Page 271 Exercises......Page 272 12 Automation......Page 274 12.1 Approaches to Automation......Page 275 12.2 Tool Building versus Automation......Page 281 12.3 Goals of Automation......Page 283 12.4 Creating Automation......Page 286 12.6 Language Tools......Page 289 12.7 Software Engineering Tools and Techniques......Page 293 12.8 Multitenant Systems......Page 301 12.9 Summary......Page 302 Exercises......Page 303 13.1 Design Documents Overview......Page 306 13.2 Design Document Anatomy......Page 308 13.4 Document Archive......Page 310 13.5 Review Workflows......Page 311 13.6 Adopting Design Documents......Page 313 13.7 Summary......Page 314 Exercises......Page 315 14.1 Designing Oncall......Page 316 14.2 Being Oncall......Page 325 14.3 Between Oncall Shifts......Page 330 14.4 Periodic Review of Alerts......Page 333 14.5 Being Paged Too Much......Page 335 14.6 Summary......Page 336 Exercises......Page 337 15 Disaster Preparedness......Page 338 15.1 Mindset......Page 339 15.2 Individual Training: Wheel of Misfortune......Page 342 15.3 Team Training: Fire Drills......Page 343 15.4 Training for Organizations: Game Day/DiRT......Page 346 15.5 Incident Command System......Page 354 15.6 Summary......Page 360 Exercises......Page 361 16 Monitoring Fundamentals......Page 362 16.1 Overview......Page 363 16.2 Consumers of Monitoring Information......Page 365 16.3 What to Monitor......Page 367 16.4 Retention......Page 369 16.5 Meta-monitoring......Page 370 16.6 Logs......Page 371 Exercises......Page 373 17 Monitoring Architecture and Practice......Page 376 17.1 Sensing and Measurement......Page 377 17.2 Collection......Page 381 17.3 Analysis and Computation......Page 384 17.4 Alerting and Escalation Manager......Page 385 17.5 Visualization......Page 389 17.7 Configuration......Page 393 17.8 Summary......Page 394 Exercises......Page 395 18 Capacity Planning......Page 396 18.1 Standard Capacity Planning......Page 397 18.2 Advanced Capacity Planning......Page 402 18.3 Resource Regression......Page 412 18.4 Launching New Services......Page 413 18.5 Reduce Provisioning Time......Page 415 18.6 Summary......Page 416 Exercises......Page 417 19 Creating KPIs......Page 418 19.1 What Is a KPI?......Page 419 19.2 Creating KPIs......Page 420 19.3 Example KPI: Machine Allocation......Page 424 19.4 Case Study: Error Budget......Page 427 Exercises......Page 430 20.1 What Does Operational Excellence Look Like?......Page 432 20.2 How to Measure Greatness......Page 433 20.3 Assessment Methodology......Page 434 20.4 Service Assessments......Page 438 20.5 Organizational Assessments......Page 442 20.6 Levels of Improvement......Page 443 20.7 Getting Started......Page 444 20.8 Summary......Page 445 Exercises......Page 446 Epilogue......Page 448 Part III: Appendices......Page 450 A: Assessments......Page 452 A.1 Regular Tasks (RT)......Page 454 A.2 Emergency Response (ER)......Page 457 A.3 Monitoring and Metrics (MM)......Page 459 A.4 Capacity Planning (CP)......Page 462 A.5 Change Management (CM)......Page 464 A.6 New Product Introduction and Removal (NPI/NPR)......Page 466 A.7 Service Deployment and Decommissioning (SDD)......Page 468 A.8 Performance and Efficiency (PE)......Page 470 A.9 Service Delivery: The Build Phase......Page 473 A.10 Service Delivery: The Deployment Phase......Page 475 A.11 Toil Reduction......Page 477 A.12 Disaster Preparedness......Page 479 B: The Origins and Future of Distributed Computing and Clouds......Page 482 Availability Requirements......Page 483 Technology......Page 484 Costs......Page 485 Technology......Page 486 Scaling......Page 487 High Availability......Page 488 B.3 The Dot-Bomb Era (2000–2003)......Page 490 Technology......Page 491 High Availability......Page 492 Scaling......Page 493 Costs......Page 495 Technology......Page 496 High Availability......Page 497 Scaling......Page 498 Costs......Page 499 Costs......Page 500 Scaling and High Availability......Page 502 B.6 Conclusion......Page 503 Exercises......Page 504 C.1 Constant, Linear, and Exponential Scaling......Page 506 C.2 Big O Notation......Page 507 C.3 Limitations of Big O Notation......Page 509 D.1 Design Document Template......Page 512 D.2 Design Document Example......Page 513 D.3 Sample Postmortem Template......Page 515 E: Recommended Reading......Page 518 Bibliography......Page 522 A......Page 530 B......Page 532 C......Page 533 D......Page 535 E......Page 538 G......Page 539 H......Page 540 J......Page 541 L......Page 542 M......Page 543 N......Page 544 O......Page 545 P......Page 546 R......Page 548 S......Page 549 T......Page 552 W......Page 554 Z......Page 555 “There's an incredible amount of depth and thinking in the practices described here, and it's impressive to see it all in one place.” —Win Treese, coauthor of Designing Systems for Internet Commerce The Practice of Cloud System Administration, Volume 2, focuses on “distributed” or “cloud” computing and brings a DevOps/SRE sensibility to the practice of system administration. Unsatisfied with books that cover either design or operations in isolation, the authors created this authoritative reference centered on a comprehensive approach. Case studies and examples from Google, Etsy, Twitter, Facebook, Netflix, Amazon, and other industry giants are explained in practical ways that are useful to all enterprises. The new companion to the best-selling first volume, The Practice of System and Network Administration, Second Edition, this guide offers expert coverage of the following and many other crucial topics: Designing and building modern web and distributed systems Fundamentals of large system design Understand the new software engineering implications of cloud administration Make systems that are resilient to failure and grow and scale dynamically Implement DevOps principles and cultural changes IaaS/PaaS/SaaS and virtual platform selection Operating and running systems using the latest DevOps/SRE strategies Upgrade production systems with zero down-time What and how to automate; how to decide what not to automate On-call best practices that improve uptime Why distributed systems require fundamentally different system administration techniques Identify and resolve resiliency problems before they surprise you Assessing and evaluating your team's operational effectiveness Manage the scientific process of continuous improvement A forty-page, pain-free assessment system you can start using today "The Practice of Cloud System Administration, Volume 2, focuses on 'distributed' or 'cloud' computing and brings a DevOps/SRE sensibility to the practice of system administration. Unsatisfied with books that cover either design or operations in isolation, the authors created this authoritative reference centered on a comprehensive approach. Case studies and examples from Google, Etsy, Twitter, Facebook, Netflix, Amazon, and other industry giants are explained in practical ways that are useful to all enterprises. The new companion to the best-selling first volume, The Practice of System and Network Administration, Second Edition, this guide offers expert coverage of the following and many other crucial topics: Designing and building modern web and distributed systems: Fundamentals of large system design; Understand the new software engineering implications of cloud administration; Make systems that are resilient to failure and grow and scale dynamically; Implement DevOps principles and cultural changes; IaaS/PaaS/SaaS and virtual platform selection; Operating and running systems using the latest DevOps/SRE strategies: Upgrade production systems with zero down-time; What and how to automate, how to decide what not to automate; On-call best practices that improve uptime; Why distributed systems require fundamentally different system administration techniques; Identify and resolve resiliency problems before they surprise you; Assessing and evaluating your team's operational effectiveness; Manage the scientific process of continuous improvement; A forty-page, pain-free assessment system you can start using today"--Publisher's description Thomas A. Limoncelli, Strata R. Chalup, Christina J. Hogan. Includes bibliographical references and index.