عملکرد پایگاه داده در مقیاس بزرگ: یک راهنمای عملی

Discover critical considerations and best practices for improving database performance based on what has worked, and failed, across thousands of teams and use cases in the field. This book provides practical guidance for understanding the database-related opportunities, trade-offs, and traps you might encounter while trying to optimize data-intensive applications for high throughput and low latency. Whether you are building a new system from the ground up or trying to optimize an existing use case for increased demand, this book covers the essentials. The book begins with a look at the many factors impacting database performance at the extreme scale that today’s game changing applications face—or at least hope to achieve. You’ll gain insight into the performance impact of both technical and business requirements, and how those should influence your decisions around database infrastructure and topology. The authors share an inside perspective on often-overlooked engineering details that could be constraining—or helping—your team’s database performance. The book also covers benchmarking and monitoring practices by which to measure and validate the outcomes from the decisions that you make. The ultimate goal of the book is to help you discover new ways to optimize database performance for your team’s specific use cases, requirements, and expectations. What You Will Learn Understand often overlooked factors that impact database performance at scale Recognize data-related performance and scalability challenges associated with your project Select a database architecture that’s suited to your workloads, use cases, and requirements Avoid common mistakes that could impede your long-term agility and growth Jumpstart teamwide adoption of best practices for optimizing database performance at scale Who This Book Is For Individuals and teams looking to optimize distributed database performance for an existing project or to begin a new performance-sensitive project with a solid and scalable foundation. This will likely include software architects, database architects, and senior software engineers who are either experiencing or anticipating pain related to database latency and/or throughput. You are most likely: • Experiencing or anticipating some pain related to database latency and/or throughput • Working primarily on a use case with terabytes to petabytes of raw (unreplicated) data, over 10K operations per second, and with P99 latencies measured in milliseconds • At least somewhat familiar with scalable distributed databases such as Apache Cassandra, ScyllaDB, Amazon DynamoDB, Google Cloud Bigtable, CockroachDB, and so on • A software architect, database architect, software engineer, VP of engineering, or technical CTO/founder working with a data-intensive application What This Book Is NOT: A few things that this book is not attempting to be: • A reference for infrastructure engineers building databases. We focus on people working with a database. • A “definitive guide” to distributed databases, NoSQL, or data-intensive applications. We focus on the top database considerations most critical to performance. • A guide on how to configure, work with, optimize, or tune any specific database. We focus on broader strategies you can “port” across databases. Table of Contents 5 About the Authors 12 About the Technical Reviewers 14 Acknowledgments 16 Introduction 18 Chapter 1: A Taste of What You’re Up Against: Two Tales 24 Joan Dives Into Drivers and Debugging 24 Joan’s Diary of Lessons Learned, Part I 26 The Tuning 26 Joan’s Diary of Lessons Learned, Part II 28 Patrick’s Unlucky Green Fedoras 29 Patrick’s Diary of Lessons Learned, Part I 30 The First Spike 31 Patrick’s Diary of Lessons Learned, Part II 31 The First Loss 32 Patrick’s Diary of Lessons Learned, Part III 32 The Spike Strikes Again 33 Patrick’s Diary of Lessons Learned, Part IV 34 Backup Strikes Back 34 Patrick’s Diary of Lessons Learned, Part V 35 Summary 36 Chapter 2: Your Project, Through the Lens of Database Performance 37 Workload Mix (Read/Write Ratio) 37 Write-Heavy Workloads 38 Read-Heavy Workloads 39 Mixed Workloads 41 Delete-Heavy Workloads 42 Competing Workloads (Real-Time vs Batch) 43 Item Size 45 Item Type 46 Dataset Size 48 Throughput Expectations 49 Latency Expectations 51 Concurrency 53 Connected Technologies 54 Demand Fluctuations 55 ACID Transactions 56 Consistency Expectations 58 Geographic Distribution 60 High-Availability Expectations 61 Summary 62 Chapter 3: Database Internals: Hardware and Operating System Interactions 63 CPU 64 Share Nothing Across Cores 64 Futures-Promises 65 Execution Stages 67 Frontend 67 Branch Speculation 67 Backend 67 Retiring 68 Implications for Databases 68 Memory 69 Allocation 69 Cache Control 72 I/O 73 Traditional Read/Write 73 mmap 74 Direct I/O (DIO) 74 Asynchronous I/O (AIO/DIO) 75 Understanding the Tradeoffs 76 Copying and MMU Activity 77 I/O Scheduling 77 Thread Scheduling 78 I/O Alignment 78 Application Complexity 79 Choosing the Filesystem and/or Disk 79 Filesystems vs Raw Disks 79 Appending Writes 80 How Modern SSDs Work 80 Networking 83 DPDK 84 IRQ Binding 84 Summary 85 Chapter 4: Database Internals: Algorithmic Optimizations 87 Optimizing Collections 88 To B- or Not to B-Tree 88 Linear Search on Steroids 90 Scanning the Tree 91 When the Tree Size Matters 92 The Secret Life of Separation Keys 94 Summary 96 Chapter 5: Database Drivers 98 Relationship Between Clients and Servers 99 Workload Types 100 Interactive Workloads 101 Batch (Analytical) Workloads 102 Mixed Workloads 102 Throughput vs Goodput 102 Timeouts 104 Client-Side Timeouts 104 Server-Side Timeouts 105 A Cautionary Tale 106 Contextual Awareness 107 Topology and Metadata 107 Current Load 108 Request Caching 109 Query Locality 112 Retries 115 Error Categories 115 Idempotence 116 Retry Policies 118 Paging 121 Concurrency 122 Modern Hardware 123 Modern Software 125 What to Look for When Selecting a Driver 126 Summary 128 Chapter 6: Getting Data Closer 129 Databases as Compute Engines 129 User-Defined Functions and Procedures 130 Determinism 132 Latency 133 Just-in-Time Compilation (JIT) 133 Examples 134 Best Practices 136 User-Defined Aggregates 137 Built-In Aggregates 137 Components 138 Initial Value 138 State Transition Function 138 Final Function 138 Reduce Function 139 Examples 139 State Transition Function 140 Final Function 140 Aggregate Definition 140 Distributed User-Defined Aggregate 141 Best Practices 143 WebAssembly for User-Defined Functions 144 Runtime 145 Back to Latency 146 Edge Computing 146 Performance 147 Conflict-Free Replicated Data Types 147 G-Counter 148 PN-Counter 148 G-Set 148 LWW-Set 148 Summary 149 Chapter 7: Infrastructure and Deployment Models 150 Core Hardware Considerations for Speed at Scale 151 Identifying the Source of Your Performance Bottlenecks 151 Achieving Balance 152 Setting Realistic Expectations 153 Recommendations for Specific Hardware Components 154 Storage 154 Disk Types 154 Disk Setup 159 Disk Size 160 Raw Devices and Custom Drivers 162 Maintaining Disk Performance Over Time 162 Tiered Storage 163 CPUs (Cores) 163 Memory (RAM) 164 Network 166 Considerations in the Cloud 167 Fully Managed Database-as-a-Service 169 Serverless Deployment Models 170 Containerization and Kubernetes 171 Summary 174 Chapter 8: Topology Considerations 175 Replication Strategy 175 Rack Configuration 176 Multi-Region or Global Replication 176 Multi-Availability Zones vs. Multi-Region 177 Scaling Up vs Scaling Out 178 Workload Isolation 180 More on Workload Prioritization for Logical Isolation 181 Abstraction Layers 185 Load Balancing 187 External Caches 188 An External Cache Adds Latency 188 An External Cache Is an Additional Cost 189 External Caching Decreases Availability 189 Application Complexity: Your Application Needs to Handle More Cases 190 External Caching Ruins the Database Caching 190 External Caching Might Increase Security Risks 190 External Caching Ignores the Database Knowledge and Database Resources 190 Summary 191 Chapter 9: Benchmarking 193 Latency or Throughput: Choose Your Focus 194 Less Is More (at First): Taking a Phased Approach 198 Benchmarking Do’s and Don’ts 200 Know What’s Under the Hood of Your Database (Or Find Someone Who Knows) 200 Choose an Environment That Takes Advantage of the Database’s Potential 201 Use an Environment That Represents Production 201 Don’t Overlook Observability 202 Use Standardized Benchmarking Tools Whenever Feasible 202 Use Representative Data Models, Datasets, and Workloads 203 Data Models 203 Dataset Size 204 Workloads 204 Exercise Your Cache Realistically 205 Look at Steady State 205 Watch Out for Client-Side Bottlenecks 206 Also Watch Out for Networking Issues 207 Document Meticulously to Ensure Repeatability 207 Reporting Do’s and Don’ts 207 Be Careful with Aggregations 208 Don’t Assume People Will Believe You 209 Take Coordinated Omission Into Account 211 Special Considerations for Various Benchmarking Goals 212 Preparing for Growth 212 Comparing Different Databases 213 Comparing the Same Database on Different Infrastructure 213 Assessing the Impact of a Data Modeling or Database Configuration Change 213 Beyond the Usual Benchmark 214 Benchmarking Admin Operations 214 Testing Disaster Recovery 214 Benchmarking at Extreme Scale 215 Summary 217 Chapter 10: Monitoring 218 Taking a Proactive Approach 218 Tracking Core Database KPIs 220 Database Cluster KPIs 220 What to Look for at Different Levels (Datacenter, Node, CPU/Shard) 223 Three Industry-Specific Examples 224 Application KPIs 224 Infrastructure/Hardware KPIs 226 Creating Effective Custom Alerts 227 Walking Through Sample Scenarios 228 One Replica Is Lagging in Acknowledging Requests 228 Disappointing P99 Read Latencies 230 Monitoring Options 234 The Database Vendor’s Monitoring Stack 234 Build Your Own Dashboards and Alerting (Grafana, Grafana Loki) 235 Third-Party Database Monitoring Tools 235 Full Stack Application Performance Monitoring (APM) Tool 235 Summary 236 Chapter 11: Administration 238 Admin Operations and Performance 238 Looking at Admin Operations Through the Lens of Performance 239 Backups 241 Impacts 242 Optimization 243 Compaction 244 Impacts 244 Optimization 246 Summary 248 Appendix A: A Brief Look at Fundamental Database Design Decisions 250 Sharding and Replication 250 Sharding 251 Replication 252 Learning More 254 Consensus Algorithms 255 Raft 256 Paxos 257 Comparing Leaderless and “Leader-Based” Classes 258 Learning More 258 B-Tree vs LSM Tree 258 Learning More 260 Record Storage Approach 261 Row-Oriented Databases 262 Column-Oriented Databases 263 Learning More 264 Index 265 Discover critical considerations and best practices for improving database performance based on what has worked, and failed, across thousands of teams and use cases in the field. This open access book provides practical guidance for understanding the database-related opportunities, trade-offs, and traps you might encounter while trying to optimize data-intensive applications for high throughput and low latency. Whether you are building a new system from the ground up or trying to optimize an existing use case for increased demand, this book covers the essentials. The book begins with a look at the many factors impacting database performance at the extreme scale that today's game changing applications face—or at least hope to achieve. You'll gain insight into the performance impact of both technical and business requirements, and how those should influence your decisions around database infrastructure and topology. The authors share an inside perspective on often-overlooked engineering details that could be constraining—or helping—your team's database performance. The book also covers benchmarking and monitoring practices by which to measure and validate the outcomes from the decisions that you make. The ultimate goal of the book is to help you discover new ways to optimize database performance for your team's specific use cases, requirements, and expectations. What You Will Learn Understand often overlooked factors that impact database performance at scale Recognize data-related performance and scalability challenges associated with your project Select a database architecture that's suited to your workloads, use cases, and requirements Avoid common mistakes that could impede your long-term agility and growth Jumpstart teamwide adoption of best practices for optimizing database performance at scale Who This Book Is For Individuals and teams looking to optimize distributed database performance for an existing project or to begin a new performance-sensitive project with a solid and scalable foundation. This will likely include software architects, database architects, and senior software engineers who are either experiencing or anticipating pain related to database latency and/or throughput.