Stealing Scarlett: A Sapphic Succubus Story

The Overwhelming Majority Of A Software System’s Lifespan Is Spent In Use, Not In Design Or Implementation. So, Why Does Conventional Wisdom Insist That Software Engineers Focus Primarily On The Design And Development Of Large-scale Computing Systems? In This Collection Of Essays And Articles, Key Members Of Google’s Site Reliability Team Explain How And Why Their Commitment To The Entire Lifecycle Has Enabled The Company To Successfully Build, Deploy, Monitor, And Maintain Some Of The Largest Software Systems In The World. You’ll Learn The Principles And Practices That Enable Google Engineers To Make Systems More Scalable, Reliable, And Efficient - Lessons Directly Applicable To Your Organization. This Book Is Divided Into Four Sections: Introduction - Learn What Site Reliability Engineering Is And Why It Differs From Conventional It Industry Practices; Principles - Examine The Patterns, Behaviors, And Areas Of Concern That Influence The Work Of A Site Reliability Engineer (sre); Practices - Understand The Theory And Practice Of An Sre’s Day-to-day Work: Building And Operating Large Distributed Computing Systems; Management - Explore Google's Best Practices For Training, Communication, And Meetings That Your Organization Can Use.--publisher's Description. Introduction. The Production Environment At Google, From The Viewpoint Of An Sre -- Principles. Embracing Risk -- Service Level Objectives -- Eliminating Toil -- Monitoring Distributed Systems -- The Evolution Of Automation At Google -- Release Engineering -- Simplicity -- Practices. Practical Alerting From Time-series Data -- Being On-call -- Effective Troubleshooting -- Emergency Response -- Managing Incidents -- Postmortem Culture: Learning From Failure -- Tracking Outages -- Testing For Reliability -- Software Engineering In Sre -- Load Balancing At The Frontend -- Load Balancing In The Datacenter -- Handling Overload -- Addressing Cascading Failures -- Managing Critical State: Distributed Consensus For Reliability -- Distributed Periodic Scheduling With Cron --data Processing Pipelines -- Date Integrity: What You Read Is What Your Wrote -- Reliable Product Launches At Scale -- Management. Accelerating Sres To On-call And Beyond -- Dealing With Interrupts -- Embedding An Sre To Recover From Operational Overload -- Communication And Collaboration In Sre -- The Evolving Sre Engagement Model -- Conclusions. Lessons Learned From Other Industries. Edited By Betsy Beyer, Chris Jones, Jennifer Petoff, And Niall Richard Murphy. Includes Bibliographical References (pages 501-512) And Index. Copyright 6 Table of Contents 7 Foreword 15 Preface 17 Conventions Used in This Book 21 Using Code Examples 22 Safari® Books Online 22 How to Contact Us 23 Acknowledgments 23 Part I. Introduction 27 Chapter 1. Introduction 29 The Sysadmin Approach to Service Management 29 Google’s Approach to Service Management: Site Reliability Engineering 31 Tenets of SRE 33 Ensuring a Durable Focus on Engineering 33 Pursuing Maximum Change Velocity Without Violating a Service’s SLO 34 Monitoring 35 Emergency Response 36 Change Management 36 Demand Forecasting and Capacity Planning 37 Provisioning 37 Efficiency and Performance 38 The End of the Beginning 38 Chapter 2. The Production Environment at Google, from the Viewpoint of an SRE 39 Hardware 39 System Software That “Organizes” the Hardware 41 Managing Machines 41 Storage 42 Networking 43 Other System Software 44 Lock Service 44 Monitoring and Alerting 44 Our Software Infrastructure 45 Our Development Environment 45 Shakespeare: A Sample Service 46 Life of a Request 47 Job and Data Organization 48 Part II. Principles 49 Chapter 3. Embracing Risk 51 Managing Risk 51 Measuring Service Risk 52 Risk Tolerance of Services 54 Identifying the Risk Tolerance of Consumer Services 54 Identifying the Risk Tolerance of Infrastructure Services 57 Motivation for Error Budgets1An early version of this section appeared as an article in ;login: (August 2015, vol. 40, no. 4). 59 Forming Your Error Budget 60 Benefits 61 Chapter 4. Service Level Objectives 63 Service Level Terminology 63 Indicators 64 Objectives 64 Agreements 65 Indicators in Practice 66 What Do You and Your Users Care About? 66 Collecting Indicators 67 Aggregation 67 Standardize Indicators 69 Objectives in Practice 69 Defining Objectives 70 Choosing Targets 71 Control Measures 72 SLOs Set Expectations 72 Agreements in Practice 73 Chapter 5. Eliminating Toil 75 Toil Defined 75 Why Less Toil Is Better 77 What Qualifies as Engineering? 78 Is Toil Always Bad? 78 Conclusion 80 Chapter 6. Monitoring Distributed Systems 81 Definitions 81 Why Monitor? 82 Setting Reasonable Expectations for Monitoring 83 Symptoms Versus Causes 84 Black-Box Versus White-Box 85 The Four Golden Signals 86 Worrying About Your Tail (or, Instrumentation and Performance) 87 Choosing an Appropriate Resolution for Measurements 88 As Simple as Possible, No Simpler 88 Tying These Principles Together 89 Monitoring for the Long Term 90 Bigtable SRE: A Tale of Over-Alerting 91 Gmail: Predictable, Scriptable Responses from Humans 91 The Long Run 92 Conclusion 92 Chapter 7. The Evolution of Automation at Google 93 The Value of Automation 93 Consistency 93 A Platform 94 Faster Repairs 94 Faster Action 95 Time Saving 95 The Value for Google SRE 96 The Use Cases for Automation 96 Google SRE’s Use Cases for Automation 97 A Hierarchy of Automation Classes 98 Automate Yourself Out of a Job: Automate ALL the Things! 99 Soothing the Pain: Applying Automation to Cluster Turnups 101 Detecting Inconsistencies with Prodtest 102 Resolving Inconsistencies Idempotently 104 The Inclination to Specialize 105 Service-Oriented Cluster-Turnup 107 Borg: Birth of the Warehouse-Scale Computer 107 Reliability Is the Fundamental Feature 109 Recommendations 110 Chapter 8. Release Engineering 113 The Role of a Release Engineer 113 Philosophy 114 Self-Service Model 114 High Velocity 114 Hermetic Builds 115 Enforcement of Policies and Procedures 115 Continuous Build and Deployment 116 Building 116 Branching 116 Testing 116 Packaging 117 Rapid 117 Deployment 119 Configuration Management 119 Conclusions 121 It’s Not Just for Googlers 121 Start Release Engineering at the Beginning 121 Chapter 9. Simplicity 123 System Stability Versus Agility 123 The Virtue of Boring 124 I Won’t Give Up My Code! 124 The “Negative Lines of Code” Metric 125 Minimal APIs 125 Modularity 126 Release Simplicity 126 A Simple Conclusion 127 Part III. Practices 129 Chapter 10. Practical Alerting from Time-Series Data 133 The Rise of Borgmon 134 Instrumentation of Applications 135 Collection of Exported Data 136 Storage in the Time-Series Arena 137 Labels and Vectors 138 Rule Evaluation 140 Alerting 144 Sharding the Monitoring Topology 145 Black-Box Monitoring 146 Maintaining the Configuration 147 Ten Years On... 148 Chapter 11. Being On-Call 151 Introduction 151 Life of an On-Call Engineer 152 Balanced On-Call 153 Balance in Quantity 153 Balance in Quality 154 Compensation 154 Feeling Safe 154 Avoiding Inappropriate Operational Load 156 Operational Overload 156 A Treacherous Enemy: Operational Underload 158 Conclusions 158 Chapter 12. Effective Troubleshooting 159 Theory 160 In Practice 162 Problem Report 162 Triage 163 Examine 164 Diagnose 165 Test and Treat 168 Negative Results Are Magic 170 Cure 171 Case Study 172 Making Troubleshooting Easier 176 Conclusion 176 Chapter 13. Emergency Response 177 What to Do When Systems Break 177 Test-Induced Emergency 178 Details 178 Response 178 Findings 179 Change-Induced Emergency 179 Details 180 Response 180 Findings 180 Process-Induced Emergency 181 Details 182 Response 182 Findings 183 All Problems Have Solutions 184 Learn from the Past. Don’t Repeat It. 184 Keep a History of Outages 184 Ask the Big, Even Improbable, Questions: What If...? 185 Encourage Proactive Testing 185 Conclusion 185 Chapter 14. Managing Incidents 187 Unmanaged Incidents 187 The Anatomy of an Unmanaged Incident 188 Sharp Focus on the Technical Problem 188 Poor Communication 188 Freelancing 188 Elements of Incident Management Process 189 Recursive Separation of Responsibilities 189 A Recognized Command Post 190 Live Incident State Document 190 Clear, Live Handoff 190 A Managed Incident 191 When to Declare an Incident 192 In Summary 192 Chapter 15. Postmortem Culture: Learning from Failure 195 Google’s Postmortem Philosophy 195 Collaborate and Share Knowledge 197 Introducing a Postmortem Culture 198 Conclusion and Ongoing Improvements 201 Chapter 16. Tracking Outages 203 Escalator 204 Outalator 204 Aggregation 206 Tagging 206 Analysis 207 Unexpected Benefits 208 Chapter 17. Testing for Reliability 209 Types of Software Testing 211 Traditional Tests 211 Production Tests 213 Creating a Test and Build Environment 216 Testing at Scale 218 Testing Scalable Tools 219 Testing Disaster 221 The Need for Speed 222 Pushing to Production 224 Expect Testing Fail 225 Integration 227 Production Probes 228 Conclusion 230 Chapter 18. Software Engineering in SRE 231 Why Is Software Engineering Within SRE Important? 231 Auxon Case Study: Project Background and Problem Space 233 Traditional Capacity Planning 233 Our Solution: Intent-Based Capacity Planning 235 Intent-Based Capacity Planning 235 Precursors to Intent 236 Introduction to Auxon 237 Requirements and Implementation: Successes and Lessons Learned 239 Raising Awareness and Driving Adoption 241 Team Dynamics 244 Fostering Software Engineering in SRE 244 Successfully Building a Software Engineering Culture in SRE: Staffing and Development Time 245 Getting There 246 Conclusions 248 Chapter 19. Load Balancing at the Frontend 249 Power Isn’t the Answer 249 Load Balancing Using DNS 250 Load Balancing at the Virtual IP Address 253 Chapter 20. Load Balancing in the Datacenter 257 The Ideal Case 258 Identifying Bad Tasks: Flow Control and Lame Ducks 259 A Simple Approach to Unhealthy Tasks: Flow Control 259 A Robust Approach to Unhealthy Tasks: Lame Duck State 260 Limiting the Connections Pool with Subsetting 261 Picking the Right Subset 262 A Subset Selection Algorithm: Random Subsetting 263 A Subset Selection Algorithm: Deterministic Subsetting 264 Load Balancing Policies 266 Simple Round Robin 267 Least-Loaded Round Robin 269 Weighted Round Robin 271 Chapter 21. Handling Overload 273 The Pitfalls of “Queries per Second” 274 Per-Customer Limits 274 Client-Side Throttling 275 Criticality 277 Utilization Signals 279 Handling Overload Errors 279 Deciding to Retry 280 Load from Connections 283 Conclusions 284 Chapter 22. Addressing Cascading Failures 285 Causes of Cascading Failures and Designing to Avoid Them 286 Server Overload 286 Resource Exhaustion 287 Service Unavailability 290 Preventing Server Overload 291 Queue Management 292 Load Shedding and Graceful Degradation 293 Retries 294 Latency and Deadlines 297 Slow Startup and Cold Caching 300 Always Go Downward in the Stack 301 Triggering Conditions for Cascading Failures 302 Process Death 302 Process Updates 303 New Rollouts 303 Organic Growth 303 Planned Changes, Drains, or Turndowns 303 Testing for Cascading Failures 304 Test Until Failure and Beyond 304 Test Popular Clients 305 Test Noncritical Backends 306 Immediate Steps to Address Cascading Failures 306 Increase Resources 306 Stop Health Check Failures/Deaths 307 Restart Servers 307 Drop Traffic 307 Enter Degraded Modes 308 Eliminate Batch Load 308 Eliminate Bad Traffic 308 Closing Remarks 309 Chapter 23. Managing Critical State: Distributed Consensus for Reliability 311 Motivating the Use of Consensus: Distributed Systems Coordination Failure 314 Case Study 1: The Split-Brain Problem 314 Case Study 2: Failover Requires Human Intervention 315 Case Study 3: Faulty Group-Membership Algorithms 315 How Distributed Consensus Works 315 Paxos Overview: An Example Protocol 316 System Architecture Patterns for Distributed Consensus 317 Reliable Replicated State Machines 317 Reliable Replicated Datastores and Configuration Stores 318 Highly Available Processing Using Leader Election 318 Distributed Coordination and Locking Services 319 Reliable Distributed Queuing and Messaging 320 Distributed Consensus Performance 322 Multi-Paxos: Detailed Message Flow 323 Scaling Read-Heavy Workloads 324 Quorum Leases 325 Distributed Consensus Performance and Network Latency 326 Reasoning About Performance: Fast Paxos 327 Stable Leaders 328 Batching 328 Disk Access 329 Deploying Distributed Consensus-Based Systems 330 Number of Replicas 330 Location of Replicas 332 Capacity and Load Balancing 333 Monitoring Distributed Consensus Systems 338 Conclusion 339 Chapter 24. Distributed Periodic Scheduling with Cron 341 Cron 341 Introduction 341 Reliability Perspective 342 Cron Jobs and Idempotency 342 Cron at Large Scale 343 Extended Infrastructure 343 Extended Requirements 344 Building Cron at Google 345 Tracking the State of Cron Jobs 345 The Use of Paxos 346 The Roles of the Leader and the Follower 346 Storing the State 350 Running Large Cron 351 Summary 352 Chapter 25. Data Processing Pipelines 353 Origin of the Pipeline Design Pattern 353 Initial Effect of Big Data on the Simple Pipeline Pattern 354 Challenges with the Periodic Pipeline Pattern 354 Trouble Caused By Uneven Work Distribution 354 Drawbacks of Periodic Pipelines in Distributed Environments 355 Monitoring Problems in Periodic Pipelines 357 “Thundering Herd” Problems 357 Moiré Load Pattern 357 Introduction to Google Workflow 359 Workflow as Model-View-Controller Pattern 360 Stages of Execution in Workflow 361 Workflow Correctness Guarantees 361 Ensuring Business Continuity 363 Summary and Concluding Remarks 364 Chapter 26. Data Integrity: What You Read Is What You Wrote 365 Data Integrity’s Strict Requirements 366 Choosing a Strategy for Superior Data Integrity 367 Backups Versus Archives 369 Requirements of the Cloud Environment in Perspective 370 Google SRE Objectives in Maintaining Data Integrity and Availability 370 Data Integrity Is the Means; Data Availability Is the Goal 371 Delivering a Recovery System, Rather Than a Backup System 371 Types of Failures That Lead to Data Loss 372 Challenges of Maintaining Data Integrity Deep and Wide 373 How Google SRE Faces the Challenges of Data Integrity 375 The 24 Combinations of Data Integrity Failure Modes 375 First Layer: Soft Deletion 376 Second Layer: Backups and Their Related Recovery Methods 378 Overarching Layer: Replication 380 1T Versus 1E: Not “Just” a Bigger Backup 380 Third Layer: Early Detection 382 Knowing That Data Recovery Will Work 385 Case Studies 386 Gmail—February, 2011: Restore from GTape 386 Google Music—March 2012: Runaway Deletion Detection 388 General Principles of SRE as Applied to Data Integrity 393 Beginner’s Mind 393 Trust but Verify 393 Hope Is Not a Strategy 393 Defense in Depth 393 Conclusion 394 Chapter 27. Reliable Product Launches at Scale 395 Launch Coordination Engineering 396 The Role of the Launch Coordination Engineer 397 Setting Up a Launch Process 398 The Launch Checklist 399 Driving Convergence and Simplification 400 Launching the Unexpected 401 Developing a Launch Checklist 401 Architecture and Dependencies 401 Integration 402 Capacity Planning 402 Failure Modes 403 Client Behavior 403 Processes and Automation 404 Development Process 404 External Dependencies 405 Rollout Planning 405 Selected Techniques for Reliable Launches 406 Gradual and Staged Rollouts 406 Feature Flag Frameworks 407 Dealing with Abusive Client Behavior 408 Overload Behavior and Load Tests 409 Development of LCE 410 Evolution of the LCE Checklist 411 Problems LCE Didn’t Solve 412 Conclusion 413 Part IV. Management 415 Chapter 28. Accelerating SREs to On-Call and Beyond 417 You’ve Hired Your Next SRE(s), Now What? 417 Initial Learning Experiences: The Case for Structure Over Chaos 420 Learning Paths That Are Cumulative and Orderly 421 Targeted Project Work, Not Menial Work 423 Creating Stellar Reverse Engineers and Improvisational Thinkers 423 Reverse Engineers: Figuring Out How Things Work 424 Statistical and Comparative Thinkers: Stewards of the Scientific Method Under Pressure 424 Improv Artists: When the Unexpected Happens 425 Tying This Together: Reverse Engineering a Production Service 425 Five Practices for Aspiring On-Callers 426 A Hunger for Failure: Reading and Sharing Postmortems 426 Disaster Role Playing 427 Break Real Things, Fix Real Things 428 Documentation as Apprenticeship 429 Shadow On-Call Early and Often 431 On-Call and Beyond: Rites of Passage, and Practicing Continuing Education 432 Closing Thoughts 432 Chapter 29. Dealing with Interrupts 433 Managing Operational Load 434 Factors in Determining How Interrupts Are Handled 434 Imperfect Machines 435 Cognitive Flow State 435 Do One Thing Well 436 Seriously, Tell Me What to Do 438 Reducing Interrupts 439 Chapter 30. Embedding an SRE to Recover from Operational Overload 443 Phase 1: Learn the Service and Get Context 444 Identify the Largest Sources of Stress 444 Identify Kindling 445 Phase 2: Sharing Context 446 Write a Good Postmortem for the Team 446 Sort Fires According to Type 446 Phase 3: Driving Change 447 Start with the Basics 447 Get Help Clearing Kindling 447 Explain Your Reasoning 448 Ask Leading Questions 449 Conclusion 449 Chapter 31. Communication and Collaboration in SRE 451 Communications: Production Meetings 452 Agenda 453 Attendance 455 Collaboration within SRE 456 Team Composition 457 Techniques for Working Effectively 457 Case Study of Collaboration in SRE: Viceroy 458 The Coming of the Viceroy 458 Challenges 460 Recommendations 461 Collaboration Outside SRE 463 Case Study: Migrating DFP to F1 463 Conclusion 466 Chapter 32. The Evolving SRE Engagement Model 467 SRE Engagement: What, How, and Why 467 The PRR Model 468 The SRE Engagement Model 469 Alternative Support 469 Production Readiness Reviews: Simple PRR Model 470 Engagement 471 Analysis 471 Improvements and Refactoring 472 Training 473 Onboarding 473 Continuous Improvement 473 Evolving the Simple PRR Model: Early Engagement 474 Candidates for Early Engagement 475 Benefits of the Early Engagement Model 475 Evolving Services Development: Frameworks and SRE Platform 477 Lessons Learned 477 External Factors Affecting SRE 478 Toward a Structural Solution: Frameworks 478 New Service and Management Benefits 480 Conclusion 482 Part V. Conclusions 483 Chapter 33. Lessons Learned from Other Industries 485 Meet Our Industry Veterans 486 Preparedness and Disaster Testing 488 Relentless Organizational Focus on Safety 488 Attention to Detail 489 Swing Capacity 489 Simulations and Live Drills 489 Training and Certification 490 Focus on Detailed Requirements Gathering and Design 490 Defense in Depth and Breadth 491 Postmortem Culture 491 Automating Away Repetitive Work and Operational Overhead 493 Structured and Rational Decision Making 495 Conclusions 496 Chapter 34. Conclusion 499 Appendix A. Availability Table 503 Appendix B. A Collection of Best Practices for Production Services 505 Fail Sanely 505 Progressive Rollouts 506 Define SLOs Like a User 506 Error Budgets 507 Monitoring 507 Postmortems 508 Capacity Planning 508 Overloads and Failure 509 SRE Teams 509 Appendix C. Example Incident State Document 511 Appendix D. Example Postmortem 513 Appendix E. Launch Coordination Checklist 519 Appendix F. Example Production Meeting Minutes 523 Bibliography 527 Index 537 About the Authors 549 Colophon 549