MongoDB Performance Tuning : Optimizing MongoDB Databases and Their Applications

He is the author of Next Generation Databases (Apress) and many other books and articles on database technology. Guy writes monthly columns for Database Trends on Applications (dbta.com) on MongoDB and emerging technologies. He can be found on the internet at http://guyharrison.net. Michael Harrison is the lead developer at ProvenDB, working intimately with MongoDB from both an application and a database perspective. He is a coauthor of The MongoDB Workshop as well as senior developer of dbKoda, an open source development environment for MongoDB. Table of Contents 5 About the Authors 15 About the Technical Reviewer 16 Acknowledgments 17 Introduction 18 Part I: Methods and Tools 22 Chapter 1: Methodical Performance Tuning 23 A Cautionary Tale 23 Symptomatic Performance Tuning 24 Systematic Performance Tuning 25 Anatomy of a Database Request 25 The Layers of a MongoDB Database 26 Minimizing the Application Workload 28 Reducing Physical IO 29 Optimizing Disk IO 30 Cluster Tuning 31 Summary 31 Chapter 2: MongoDB Architecture and Concepts 33 The MongoDB Document Model 33 JSON 34 Binary JSON (BSON) 34 Collections 35 MongoDB Schemas 36 The MongoDB Protocol 38 Wire Protocol 38 MongoDB Drivers 39 MongoDB Commands 40 The find Command 40 The aggregate Command 41 Data Manipulation Commands 43 Consistency Mechanisms 43 Read Preference and Write Concern 44 Transactions 45 Query Optimization 45 MongoDB Architecture 46 Mongod 46 Storage Engines 46 Replica Sets 48 Sharding 50 Sharding Mechanisms 51 Cluster Balancing 52 Conclusion 52 Chapter 3: Tools of the Trade 53 Introduction to explain() 53 Getting Started with explain() 55 Alternate Plans 58 Execution Statistics 58 Using explain() to Tune a Query 61 Visual Explain Utilities 62 The Query Profiler 64 The system.profile Collection 66 Analyzing Profiling Data 67 Tuning with MongoDB Logs 70 Server Statistics 74 Examining Current Operations 78 Operating System Monitoring 81 MongoDB Compass 82 Summary 84 Part II: Application and Database Design 85 Chapter 4: Schema Modelling 86 The Guiding Principles 87 Linking vs. Embedding 88 A Case Study 88 Getting All the Data for a Customer 92 Fetching All Open Orders 93 Top Products 95 Inserting New Orders 96 Updating Products 97 Deleting a Customer 98 Case Study Summary 99 Advanced Patterns 100 Subsetting 100 Vertical Partitioning 103 The Attribute Pattern 105 Summary 106 Chapter 5: Indexing 107 B-Tree Indexes 107 Index Selectivity 109 Unique Indexes 109 Index Scans 110 Case-Insensitive Searches 112 Compound Indexes 113 Compound Index Performance 114 Compound Index Key Order 115 Guidelines for Compound Indexes 116 Covering Indexes 116 Index Merges 117 Partial and Sparse Indexes 118 Partial Indexes 118 Sparse Indexes 119 Using Indexes for Sorting and Joining 119 Sorting 119 Using Indexes for Joins 120 Index Overhead 120 Wildcard Indexes 120 Text Indexes 123 Text Index Performance 128 Geospatial Indexes 131 Geospatial Index Performance 135 Geospatial Index Limitations 136 Summary 137 Part III: Tuning MongoDB Code 138 Chapter 6: Query Tuning 139 Caching Results 139 Optimizing Network Round Trips 142 Projections 142 Batch Processing 144 Avoiding Excessive Network Round Trips in Code 145 Bulk Inserts 147 Application Architecture 147 Choosing an Index vs. a Scan 148 Overriding the Optimizer with Hints 150 Optimizing Sort Operations 152 Picking or Creating the Right Index 155 Filter Strategies 157 Not Equals Conditions 157 Range Queries 160 $OR or $IN Operations 161 Array Queries 163 Regular Expressions 164 $exists Queries 166 Optimizing Collection Scans 168 Summary 169 Chapter 7: Tuning Aggregation Pipelines 170 Tuning Aggregation Pipelines 171 Optimizing Aggregation Ordering 174 Automatic Pipeline Optimizations 176 Optimizing Multi-collection Joins 181 Join Order 182 Optimizing Graph Lookups 184 Aggregation Memory Utilization 186 Sorting in Aggregation Pipelines 189 Indexed Aggregation Sorts 189 Disk Sorts 191 Optimizing Views 193 Materialized Views 195 Summary 198 Chapter 8: Inserts, Updates, and Deletes 199 Fundamentals 199 Filter Optimizations 200 Explaining a Data Manipulation Statement 200 Index Overhead 201 Finding Unused Indexes 202 Write Concern 203 Inserts 204 Batch Processing 204 Cloning Data 206 Loading from Files 209 Updates 209 Dynamic Value Bulk Updates 209 The multi:true Flag 211 Upserts 212 Bulk Upsert with $merge 213 Delete Optimizations 214 Summary 215 Chapter 9: Transactions 216 Transaction Theory 216 MongoDB Transactions 219 Transaction Limits 219 TransientTransactionErrors 220 Transactions in the MongoDB Drivers 222 The Performance Implications of TransientTransactionErrors 225 Transaction Optimization 226 Avoiding Transactions 227 Ordering of Operations 229 Partitioning Hot Documents 231 Conclusion 233 Chapter 10: Server Monitoring 234 Host-Level Monitoring 235 Network 236 CPU 238 Memory 240 Disk IO 240 MongoDB Server Monitoring 241 Compass 241 Free Monitoring 242 Ops Manager 243 MongoDB Atlas 244 Third-Party Monitoring Tools 245 Summary 246 Part IV: Server Tuning 247 Chapter 11: Memory Tuning 248 MongoDB Memory Architecture 248 Host Memory 250 Measuring Memory 251 WiredTiger Memory 253 Cache Size 253 Determining the Optimum Cache Size 254 The Database Cache "Hit" Ratio 254 Evictions 257 Blocking Evictions 257 Checkpoints 259 WiredTiger Concurrency 262 Reducing Application Memory Demand 263 Document Design 263 Indexing 264 Transactions 264 Summary 265 Chapter 12: Disk IO 266 IO Fundamentals 266 Latency and Throughput 266 Queuing 267 Sequential and Random IO 269 Disk Hardware 270 Magnetic Disks (HDD) 270 Solid State Drives 272 SSD Storage Hierarchy 272 Write Performance 273 Write Endurance 273 Garbage Collection and Wear Levelling 274 SATA vs. PCI 274 Recommendations for SSDs 276 Storage Arrays 277 RAID Levels 277 The RAID 5 Write Penalty 280 Non-volatile Caches in RAID 5 Devices 280 Do It Yourself Arrays 281 Hardware Storage Arrays 282 Cloud Storage 283 Disk Devices in MongoDB Atlas 284 MongoDB IO 284 Temporary File IO 285 The Journal 287 Moving the Journal to a Dedicated Device 290 Datafile IO 292 Datafile Writes 293 Splitting Up Datafiles Across Multiple Devices 294 Detecting and Solving IO Problems 296 Increasing IO Subsystem Bandwidth 299 Dedicated Server with Dedicated Disks 299 Storage Arrays 300 Cloud Storage 300 MongoDB Atlas 301 Summary 302 Chapter 13: Replica Sets and Atlas 304 Replica Set Fundamentals 304 Using Read Preference 305 Setting Read Preference 307 maxStalenessSeconds 308 Tag Sets 309 Write Concern 311 Journaling 311 The Write Concern w Option 312 Write Concern and Secondary Reads 314 MongoDB Atlas 314 Atlas Search 315 Atlas Data Lake 321 Summary 324 Chapter 14: Sharding 326 Sharding Fundamentals 326 Scaling and Sharding 327 Sharding Concepts 327 To Shard or Not to Shard? 328 Shard Key Selection 330 Range- vs. Hash-Based Sharding 330 Zone Sharding 333 Shard Balance 335 Rebalancing Shards 336 Modifying the Balancer Window 339 Disabling the Balancer 339 Changing the Chunk Size 340 Changing Shard Keys 341 Sharded Queries 343 Sharded Explain Plans 343 Shard Key Lookups 345 Accidental Shard Merge 346 Shard Key Range 347 Sorting 348 Non-Shard Key Lookups 349 Aggregations and Sorts 350 Sharded $lookup Operations 352 Summary 352 Index 354 Use this fast and complete guide to optimize the performance of MongoDB databases and the applications that depend on them. You will be able to turbo-charge the performance of your MongoDB applications to provide a better experience for your users, reduce your running costs, and avoid application growing pains. MongoDB is the world's most popular document database and the foundation for thousands of mission-critical applications. This book helps you get the best possible performance from MongoDB. MongoDB Performance Tuning takes a methodical and comprehensive approach to performance tuning that begins with application and schema design and goes on to cover optimization of code at all levels of an application. The book also explains how to configure MongoDB hardware and cluster configuration for optimal performance. The systematic approach in the book helps you treat the true causes of performance issues and get the best return on your tuning investment. Even when you're under pressure and don't know where to begin, simply follow the method in this book to set things right and get your MongoDB performance back on track. What You Will Learn Apply a methodical approach to MongoDB performance tuning Understand how to design an efficient MongoDB application Optimize MongoDB document design and indexing strategies Tune MongoDB queries, aggregation pipelines, and transactions Optimize MongoDB server resources: CPU, memory, disk Configure MongoDB Replica sets and Sharded clusters for optimal performance Who This Book Is For Developers and administrators of high-performance MongoDB applications who want to be sure they are getting the best possible performance from their MongoDB system. For developers who wish to create applications that are fast, scalable, and cost-effective. For administrators who want to optimize their MongoDB server and hardware configuration.