Software Engineering at Google : Lessons Learned From Programming Over Time

Today, software engineers need to know not only how to program effectively but also how to develop proper engineering practices to make their codebase sustainable and healthy. This book emphasizes this difference between programming and software engineering. How can software engineers manage a living codebase that evolves and responds to changing requirements and demands over the length of its life? Based on their experience at Google, software engineers Titus Winters and Hyrum Wright, along with technical writer Tom Manshreck, present a candid and insightful look at how some of the world’s leading practitioners construct and maintain software. This book covers Google’s unique engineering culture, processes, and tools and how these aspects contribute to the effectiveness of an engineering organization. You’ll explore three fundamental principles that software organizations should keep in mind when designing, architecting, writing, and maintaining code: • How time affects the sustainability of software and how to make your code resilient over time • How scale affects the viability of software practices within an engineering organization • What trade-offs a typical engineer needs to make when evaluating design and development decisions Cover Copyright Table of Contents Foreword Preface Programming Over Time Google’s Perspective What This Book Isn’t Parting Remarks Conventions Used in This Book O’Reilly Online Learning How to Contact Us Acknowledgments Part I. Thesis Chapter 1. What Is Software Engineering? Time and Change Hyrum’s Law Example: Hash Ordering Why Not Just Aim for “Nothing Changes”? Scale and Efficiency Policies That Don’t Scale Policies That Scale Well Example: Compiler Upgrade Shifting Left Trade-offs and Costs Example: Markers Inputs to Decision Making Example: Distributed Builds Example: Deciding Between Time and Scale Revisiting Decisions, Making Mistakes Software Engineering Versus Programming Conclusion TL;DRs Part II. Culture Chapter 2. How to Work Well on Teams Help Me Hide My Code The Genius Myth Hiding Considered Harmful Early Detection The Bus Factor Pace of Progress In Short, Don’t Hide It’s All About the Team The Three Pillars of Social Interaction Why Do These Pillars Matter? Humility, Respect, and Trust in Practice Blameless Post-Mortem Culture Being Googley Conclusion TL;DRs Chapter 3. Knowledge Sharing Challenges to Learning Philosophy Setting the Stage: Psychological Safety Mentorship Psychological Safety in Large Groups Growing Your Knowledge Ask Questions Understand Context Scaling Your Questions: Ask the Community Group Chats Mailing Lists YAQS: Question-and-Answer Platform Scaling Your Knowledge: You Always Have Something to Teach Office Hours Tech Talks and Classes Documentation Code Scaling Your Organization’s Knowledge Cultivating a Knowledge-Sharing Culture Establishing Canonical Sources of Information Staying in the Loop Readability: Standardized Mentorship Through Code Review What Is the Readability Process? Why Have This Process? Conclusion TL;DRs Chapter 4. Engineering for Equity Bias Is the Default Understanding the Need for Diversity Building Multicultural Capacity Making Diversity Actionable Reject Singular Approaches Challenge Established Processes Values Versus Outcomes Stay Curious, Push Forward Conclusion TL;DRs Chapter 5. How to Lead a Team Managers and Tech Leads (and Both) The Engineering Manager The Tech Lead The Tech Lead Manager Moving from an Individual Contributor Role to a Leadership Role The Only Thing to Fear Is...Well, Everything Servant Leadership The Engineering Manager Manager Is a Four-Letter Word Today’s Engineering Manager Antipatterns Antipattern: Hire Pushovers Antipattern: Ignore Low Performers Antipattern: Ignore Human Issues Antipattern: Be Everyone’s Friend Antipattern: Compromise the Hiring Bar Antipattern: Treat Your Team Like Children Positive Patterns Lose the Ego Be a Zen Master Be a Catalyst Remove Roadblocks Be a Teacher and a Mentor Set Clear Goals Be Honest Track Happiness The Unexpected Question Other Tips and Tricks People Are Like Plants Intrinsic Versus Extrinsic Motivation Conclusion TL;DRs Chapter 6. Leading at Scale Always Be Deciding The Parable of the Airplane Identify the Blinders Identify the Key Trade-Offs Decide, Then Iterate Always Be Leaving Your Mission: Build a “Self-Driving” Team Dividing the Problem Space Always Be Scaling The Cycle of Success Important Versus Urgent Learn to Drop Balls Protecting Your Energy Conclusion TL;DRs Chapter 7. Measuring Engineering Productivity Why Should We Measure Engineering Productivity? Triage: Is It Even Worth Measuring? Selecting Meaningful Metrics with Goals and Signals Goals Signals Metrics Using Data to Validate Metrics Taking Action and Tracking Results Conclusion TL;DRs Part III. Processes Chapter 8. Style Guides and Rules Why Have Rules? Creating the Rules Guiding Principles The Style Guide Changing the Rules The Process The Style Arbiters Exceptions Guidance Applying the Rules Error Checkers Code Formatters Conclusion TL;DRs Chapter 9. Code Review Code Review Flow How Code Review Works at Google Code Review Benefits Code Correctness Comprehension of Code Code Consistency Psychological and Cultural Benefits Knowledge Sharing Code Review Best Practices Be Polite and Professional Write Small Changes Write Good Change Descriptions Keep Reviewers to a Minimum Automate Where Possible Types of Code Reviews Greenfield Code Reviews Behavioral Changes, Improvements, and Optimizations Bug Fixes and Rollbacks Refactorings and Large-Scale Changes Conclusion TL;DRs Chapter 10. Documentation What Qualifies as Documentation? Why Is Documentation Needed? Documentation Is Like Code Know Your Audience Types of Audiences Documentation Types Reference Documentation Design Docs Tutorials Conceptual Documentation Landing Pages Documentation Reviews Documentation Philosophy WHO, WHAT, WHEN, WHERE, and WHY The Beginning, Middle, and End The Parameters of Good Documentation Deprecating Documents When Do You Need Technical Writers? Conclusion TL;DRs Chapter 11. Testing Overview Why Do We Write Tests? The Story of Google Web Server Testing at the Speed of Modern Development Write, Run, React Benefits of Testing Code Designing a Test Suite Test Size Test Scope The Beyoncé Rule A Note on Code Coverage Testing at Google Scale The Pitfalls of a Large Test Suite History of Testing at Google Orientation Classes Test Certified Testing on the Toilet Testing Culture Today The Limits of Automated Testing Conclusion TL;DRs Chapter 12. Unit Testing The Importance of Maintainability Preventing Brittle Tests Strive for Unchanging Tests Test via Public APIs Test State, Not Interactions Writing Clear Tests Make Your Tests Complete and Concise Test Behaviors, Not Methods Don’t Put Logic in Tests Write Clear Failure Messages Tests and Code Sharing: DAMP, Not DRY Shared Values Shared Setup Shared Helpers and Validation Defining Test Infrastructure Conclusion TL;DRs Chapter 13. Test Doubles The Impact of Test Doubles on Software Development Test Doubles at Google Basic Concepts An Example Test Double Seams Mocking Frameworks Techniques for Using Test Doubles Faking Stubbing Interaction Testing Real Implementations Prefer Realism Over Isolation How to Decide When to Use a Real Implementation Faking Why Are Fakes Important? When Should Fakes Be Written? The Fidelity of Fakes Fakes Should Be Tested What to Do If a Fake Is Not Available Stubbing The Dangers of Overusing Stubbing When Is Stubbing Appropriate? Interaction Testing Prefer State Testing Over Interaction Testing When Is Interaction Testing Appropriate? Best Practices for Interaction Testing Conclusion TL;DRs Chapter 14. Larger Testing What Are Larger Tests? Fidelity Common Gaps in Unit Tests Why Not Have Larger Tests? Larger Tests at Google Larger Tests and Time Larger Tests at Google Scale Structure of a Large Test The System Under Test Test Data Verification Types of Larger Tests Functional Testing of One or More Interacting Binaries Browser and Device Testing Performance, Load, and Stress testing Deployment Configuration Testing Exploratory Testing A/B Diff Regression Testing UAT Probers and Canary Analysis Disaster Recovery and Chaos Engineering User Evaluation Large Tests and the Developer Workflow Authoring Large Tests Running Large Tests Owning Large Tests Conclusion TL;DRs Chapter 15. Deprecation Why Deprecate? Why Is Deprecation So Hard? Deprecation During Design Types of Deprecation Advisory Deprecation Compulsory Deprecation Deprecation Warnings Managing the Deprecation Process Process Owners Milestones Deprecation Tooling Conclusion TL;DRs Part IV. Tools Chapter 16. Version Control and Branch Management What Is Version Control? Why Is Version Control Important? Centralized VCS Versus Distributed VCS Source of Truth Version Control Versus Dependency Management Branch Management Work in Progress Is Akin to a Branch Dev Branches Release Branches Version Control at Google One Version Scenario: Multiple Available Versions The “One-Version” Rule (Nearly) No Long-Lived Branches What About Release Branches? Monorepos Future of Version Control Conclusion TL;DRs Chapter 17. Code Search The Code Search UI How Do Googlers Use Code Search? Where? What? How? Why? Who and When? Why a Separate Web Tool? Scale Zero Setup Global Code View Specialization Integration with Other Developer Tools API Exposure Impact of Scale on Design Search Query Latency Index Latency Google’s Implementation Search Index Ranking Selected Trade-Offs Completeness: Repository at Head Completeness: All Versus Most-Relevant Results Completeness: Head Versus Branches Versus All History Versus Workspaces Expressiveness: Token Versus Substring Versus Regex Conclusion TL;DRs Chapter 18. Build Systems and Build Philosophy Purpose of a Build System What Happens Without a Build System? But All I Need Is a Compiler! Shell Scripts to the Rescue? Modern Build Systems It’s All About Dependencies Task-Based Build Systems Artifact-Based Build Systems Distributed Builds Time, Scale, Trade-Offs Dealing with Modules and Dependencies Using Fine-Grained Modules and the 1:1:1 Rule Minimizing Module Visibility Managing Dependencies Conclusion TL;DRs Chapter 19. Critique: Google’s Code Review Tool Code Review Tooling Principles Code Review Flow Notifications Stage 1: Create a Change Diffing Analysis Results Tight Tool Integration Stage 2: Request Review Stages 3 and 4: Understanding and Commenting on a Change Commenting Understanding the State of a Change Stage 5: Change Approvals (Scoring a Change) Stage 6: Commiting a Change After Commit: Tracking History Conclusion TL;DRs Chapter 20. Static Analysis Characteristics of Effective Static Analysis Scalability Usability Key Lessons in Making Static Analysis Work Focus on Developer Happiness Make Static Analysis a Part of the Core Developer Workflow Empower Users to Contribute Tricorder: Google’s Static Analysis Platform Integrated Tools Integrated Feedback Channels Suggested Fixes Per-Project Customization Presubmits Compiler Integration Analysis While Editing and Browsing Code Conclusion TL;DRs Chapter 21. Dependency Management Why Is Dependency Management So Difficult? Conflicting Requirements and Diamond Dependencies Importing Dependencies Compatibility Promises Considerations When Importing How Google Handles Importing Dependencies Dependency Management, In Theory Nothing Changes (aka The Static Dependency Model) Semantic Versioning Bundled Distribution Models Live at Head The Limitations of SemVer SemVer Might Overconstrain SemVer Might Overpromise Motivations Minimum Version Selection So, Does SemVer Work? Dependency Management with Infinite Resources Exporting Dependencies Conclusion TL;DRs Chapter 22. Large-Scale Changes What Is a Large-Scale Change? Who Deals with LSCs? Barriers to Atomic Changes Technical Limitations Merge Conflicts No Haunted Graveyards Heterogeneity Testing Code Review LSC Infrastructure Policies and Culture Codebase Insight Change Management Testing Language Support The LSC Process Authorization Change Creation Sharding and Submitting Cleanup Conclusion TL;DRs Chapter 23. Continuous Integration CI Concepts Fast Feedback Loops Automation Continuous Testing CI Challenges Hermetic Testing CI at Google CI Case Study: Google Takeout But I Can’t Afford CI Conclusion TL;DRs Chapter 24. Continuous Delivery Idioms of Continuous Delivery at Google Velocity Is a Team Sport: How to Break Up a Deployment into Manageable Pieces Evaluating Changes in Isolation: Flag-Guarding Features Striving for Agility: Setting Up a Release Train No Binary Is Perfect Meet Your Release Deadline Quality and User-Focus: Ship Only What Gets Used Shifting Left: Making Data-Driven Decisions Earlier Changing Team Culture: Building Discipline into Deployment Conclusion TL;DRs Chapter 25. Compute as a Service Taming the Compute Environment Automation of Toil Containerization and Multitenancy Summary Writing Software for Managed Compute Architecting for Failure Batch Versus Serving Managing State Connecting to a Service One-Off Code CaaS Over Time and Scale Containers as an Abstraction One Service to Rule Them All Submitted Configuration Choosing a Compute Service Centralization Versus Customization Level of Abstraction: Serverless Public Versus Private Conclusion TL;DRs Part V. Conclusion Afterword Index About the Authors Colophon The approach to and understanding of software engineering at Google is unlike any other company. With this book, you’ll get a candid and insightful look at how software is constructed and maintained by some of the world’s leading practitioners. Titus Winters, Tom Manshreck, and Hyrum K. Wright, software engineers and a technical writer at Google, reframe how software engineering is practiced and taught: from an emphasis on programming to an emphasis on software engineering, which roughly translates to programming over time. You’ll learn: • Fundamental differences between software engineering and programming • How an organization effectively manages a living codebase and efficiently responds to inevitable change • Why culture (and recognizing it) is important, and how processes, practices, and tools come into play