KAFKA TROUBLESHOOTING IN PRODUCTION : stabilizing kafka clusters in the cloud and on-premises

Table of Contents About the Author About the Technical Reviewer Acknowledgments Introduction Chapter 1: Storage Usage in Kafka: Challenges, Strategies, and Best Practices How Kafka Runs Out of Disk Space A Retention Policy Can Cause Data Loss Configuring a Retention Policy for Kafka Topics Managing Consumer Lag and Preventing Data Loss Handling Bursty Data Influx from Producers Balancing Consumer Throttling and Avoiding Unintended Lag Understanding Daily Traffic Variations and Their Impact on Data Retention Ensuring Batch Duration Compliance with Topic Retention to Avoid Data Loss Adding Storage to Kafka Clusters When the Cluster Is On-Prem Scaling Up Scaling Out When the Cluster Is in the Cloud EBS Disks NVME Disks (Ephemeral Disks) Strategies and Considerations for Extended Retention in Kafka Clusters Calculating Storage Capacity Based on Time-Based Retention Retention Monitoring Data Skew in Partitions Message Rate Into Topic Don’t Write to the / Mount Point Summary Chapter 2: Strategies for Aggregation, Data Cardinality, and Batching Balancing Message Distribution and Aggregation for Optimal Kafka Performance Tuning Parameters to Increase Throughput and Reduce Latency Optimizing Producer and Broker Performance: The Impact of Tuning linger.ms and batch.size in Kafka Understanding Compression Rate The Effect of Data Cardinality on Producers, Consumers, and Brokers Defining Data Cardinality Effects of High Data Cardinality Reducing Cardinality Level and Distribution Duplicating Data to Reduce Latency Summary Chapter 3: Understanding and Addressing Partition Skew in Kafka Skew of Partition Leaders vs. Skew of Partition Followers Potential Problems with Brokers That Host Many Partition Leaders Message Rate (or Incoming Bytes Rate) Number of Consumers Consuming the Topic Number of Producers Producing to the Topic Follower (Replica) Skew in the Broker Number of In-Sync Replicas in the Broker Checking for an Imbalance of Partition Leaders Reassigning Partitions to Achieve an Even Distribution Data Distribution Among Disks Summary Chapter 4: Dealing with Skewed and Lost Leaders When Partitions Lose Their Leadership ZooKeeper The Network Interface Card (NIC) Should Leader Skew Always Be Solved? When There Is High Traffic When There Is a Large Number of Consumers/Producers Understanding Leader Skew Summary Chapter 5: CPU Saturation in Kafka: Causes, Consequences, and Solutions CPU Saturation CPU Usage Types Causes of High CPU User Times Causes of High CPU System Times Example of Kafka Brokers with High CPU %us and %sy Causes of High CPU Wait Times Causes of High CPU System Interrupt Times The Effect of Compacted Topics with High Retention on Disk and CPU Use What Is Log Compaction? Real Production Issues Due to Log Compaction The Number of Consumers per Topic vs. CPU Use Summary Chapter 6: RAM Allocation in Kafka Clusters: Performance, Stability, and Optimization Strategies Adding RAM to a Kafka Cluster The Strategic Role of RAM Over CPU and Disks Cloud vs. On-Prem RAM Expansion: Considerations and Constraints Adding RAM to the Cloud Adding RAM to On-Prem Kafka Clusters Enhancing Kafka’s Performance: The Benefits of Increasing Broker RAM Performance Boost Disk I/O Reduction Throughput Enhancement Latency Reduction Understanding the Linux Page Cache Page Cache in Kafka: Accelerating Writes and Reads Balancing Performance and Reliability: Kafka’s Page Cache Utilization Monitoring Page Cache Usage Using the Cachestat Tool Lack of RAM and its Effect on Disks Optimize Kafka Disks When the Cluster Lacks RAM Use SSDs Instead of HDDs Distribute Logs Across Disks Tune OS Disk Scheduling Algorithm Adjust Kafka’s Disk Flush Policies Enable Log Compression Enable OS Page Cache Monitor Disk Usage and I/O A Lack of RAM Can Cause Disks to Reach IOPS Saturation Optimize Kafka in Terms of RAM Allocation Set vm.swappiness to the Minimum Possible Value Increase the File Descriptor Limits Increase the Limit of Memory-Mapped Files GIVE at Least 32GB RAM to Your Kafka Brokers Monitor Garbage Collection Times Closely Tuning JVM Options Using Appropriate Instance Types When Deploying on a Cloud Platform Balancing Topics and Partitions Across Brokers Dealing with Garbage Collection (GC) and Out-Of-Memory (OOM) Latency Spikes Resource Utilization System Stability Impact on ZooKeeper Heartbeat Measuring Kafka Memory Usage The Crucial Role of RAM: Lessons from a Non-Kafka Cluster Summary Chapter 7: Disk I/O Overload in Kafka: Diagnosing and Overcoming Challenges Disk Performance Metrics Detecting Whether Disks Cause Latency in Kafka Brokers, Consumers, or Producers How Kafka Reads and Writes to the Disks Writes Reads Disk Performance Detection Data Skew in the Scope of a Single Broker Data Skew in the Scope of a Kafka Cluster Consumer Lag from a Specific Broker Slow (Faulty) Disk Real Production Issue: Detecting a Faulty Broker Using Disk Performance Metrics Discussion The Effect of Too Many disk.io Threads Discussion Looking at Disk Performance the Whole Time vs. During Peak Time Only Discussion The Effect of disk.io Threads on Broker, Producer, and Consumer Performance Request Queue Size Produce Latency Number of JVM Threads Number of Context Switches CPU User Time, System Time, and Normalized Load Average Discussion Summary Chapter 8: Disk Configuration: RAID 10 vs. JBOD RAID 10 and JBOD Terminology RAID 0 (aka Stripe Set) RAID 1 (aka Mirror Set) RAID 1+0 (aka RAID 10) Comparing RAID 10 and JBOD Disk Failure Data Skew Storage Use Pros and Cons of RAID 10 and JBOD Performance of Write Operations Storage Usage Disk Failure Tolerance Considering the Maintenance Burden of Disk Failure in On-Premises Clusters Disk Health Monitoring Frequency of Replacing Disks Kafka Availability During Disk Replacement Balancing the Data Between the Disks in the Broker JBOD RAID 10 Managing Disk Health in Kafka Clusters with JBOD Configuration Summary Chapter 9: A Deep Dive Into Producer Monitoring Producer Metrics Network I/O Rate Metric When Network I/O Rate Is High When Network I/O Rate Is Low Importance of the Network I/O Rate Metric Record Queue Time Metric When Record Queue Time Is High When Record Queue Time Is Low Mitigating a High Record Queue Time Importance of the Record Queue Time Metric Output Bytes Metric When Output Bytes Is High When Output Bytes Is Low Mitigating the Output Bytes Value Importance of the Output Bytes Metric Input Bytes Metric The Difference Between Output Bytes and Input Bytes Average Batch Size Metric When Average Batch Size Is High When Average Batch Size Is Low Mitigating the Average Batch Size Metric Importance of the Average Batch Size Metric Buffer Available Bytes Metric When Buffer Available Bytes Is High When Buffer Available Bytes Is Low Mitigating the Buffer Available Bytes Metric Importance of the Buffer Available Bytes Metric Request Latency (Avg/Max) Metrics When Request Latency Is High When Request Latency Is Low Mitigating the Request Latency Metric Importance of the Request Latency Metric Understanding the Impact of Multiple Producers and Consumers on the Kafka Cluster Compression Rate: A Special Kind of Producer Metric Configuring Compression on the Producer and Broker Levels Compression Rate When Compression Rate Is High When Compression Rate Is Low Mitigating Compression Rate Importance of the Compression Rate Metric Summary Chapter 10: A Deep Dive Into Consumer Monitoring Consumer Metrics Consumer Lag Metrics When the Consumer Lag Metric Is High When the Consumer Lag Metric Is Low Mitigating the Consumer Lag Metric Importance of the Consumer Lag Metric Fetch Request Rate Metric When the Fetch Request Rate Is High When the Fetch Request Rate Is Low Mitigating the Fetch Request Rate Importance of the Fetch Request Rate Fetch Request Size (Avg/Max) Metrics When the Fetch Request Size Is High When the Fetch Request Size Is Low Mitigating the Fetch Request Size Importance of the Fetch Request Size Metrics Consumer I/O Wait Ratio Metric When the Consumer I/O Wait Ratio Is High When the Consumer I/O Wait Ratio Is Low Mitigating the Consumer I/O Wait Ratio Importance of the Consumer I/O Wait Ratio Records per Request Avg Metric When the Records per Request Metric Is High When the Records per Request Metric Is Low Mitigating the Records per Request Metric Importance of the Records per Request Metric Fetch Latency Avg/Max Metrics When the Fetch Latency Metrics Are High When the Fetch Latency Metrics Are Low Mitigating the Fetch Latency Metrics Importance of the Fetch Latency Metrics Consumer Request Rate Metric When the Consumer Request Rate Metric Is High When the Consumer Request Rate Metric Is Low Mitigating the Consumer Request Rate Metric Importance of the Consumer Request Rate Metric Bytes Consumed Rate Metric When the Bytes Consumed Rate Metric Is High When the Bytes Consumed Rate Metric Is Low Mitigating the Bytes Consumed Rate Metric Importance of the Bytes Consumed Rate Metric The Relationship Between Data Skew in Partitions and Consumer Lag An Example of Correlating Between Consumer and Producer Metrics Summary Chapter 11: Stability Issues in On-Premises Kafka Data Centers Common Failures in Hardware Components The Effect of Hardware Failures on the Stability of On-Prem Kafka Clusters HDD Disk Failures Common Reasons That Disks Fail Potential Impacts of Disk Failure on a Kafka Broker HDD Disks Lose Their Write Capability and Become Read-Only Monitoring Disk Health Remedying Failed Disks RAM DIMMs Failures Potential Causes of DIMMs Failures Monitoring DIMMs Failures Network Interface Cards (NICs) Failures Potential Causes of NIC Failures Implications of NIC Failures on a Kafka Cluster Detecting NIC Failures Resolving NIC Failures Power Supply Failures Potential Causes of Power Supply Failures Implications of Power Supply Failures Resolving Power Supply Failures Motherboard Failures Potential Causes of Motherboard Failures Implications of Motherboard Failures Resolving Motherboard Failures Disk Drawer and Rack Failures Potential Causes of Disk Drawer and Rack Failures Implications of Disk Drawer and Rack Failures Resolving Disk Drawer and Rack Failures Potential Negative Effects of Enabling Firewalls and Antivirus on Kafka Brokers ZooKeeper Best Practices in On-Premises Kafka Data Centers Disk Performance Dedicated Machines Monitoring ZooKeeper Developing a Dedicated Smoke Test for Kafka and ZooKeeper Stability Summary Chapter 12: Cost Reduction of Kafka Clusters Determining If You Can Reduce Costs in Your Kafka Clusters Lack of RAM Cloud-Based Cluster On-Prem Cluster Discussion Replace the DIMMs with Larger DIMMs That Have More RAM Scale Out by Adding More Brokers with the Same Amount of RAM Lack of CPU Cores Cloud-Based Cluster On-Prem Cluster Discussion Lack of Disk Storage Cloud-Based Cluster On-Prem Cluster Discussion Lack of Disk IOPS Cost Optimization Strategies for Kafka Clusters in the Cloud Additional Considerations Cloud vs. On-Prem Hyper-Threading CPU Type Normalized Load Average (NLA) Scale In Example 1 Findings Options for Reducing Costs Scale Down Scale In Recommendation Example 2 Findings Options for Reducing Costs Recommendation Example 3 Findings Options for Reducing Costs Scaling Down Scaling In Example 4 Findings Options for Reducing Costs Scale Down Scale In Recommendation Example 5 Findings Options for Reducing Costs Example 6 Findings Options for Reducing Costs Scale Down Scale In Recommendation Summary Index This book provides Kafka administrators, site reliability engineers, and DataOps and DevOps practitioners with a list of real production issues that can occur in Kafka clusters and how to solve them. The production issues covered are assembled into a comprehensive troubleshooting guide for those engineers who are responsible for the stability and performance of Kafka clusters in production, whether those clusters are deployed in the cloud or on-premises. This book teaches you how to detect and troubleshoot the issues, and eventually how to prevent them. Kafka stability is hard to achieve, especially in high throughput environments, and the purpose of this book is not only to make troubleshooting easier, but also to prevent production issues from occurring in the first place. The guidance in this book is drawn from the author's years of experience in helping clients and internal customers diagnose and resolve knotty production problems and stabilize their Kafka environments. The book is organized into recipe-style troubleshooting checklists that field engineers can easily follow when under pressure to fix an unstable cluster. This is the book you will want by your side when the stakes are high, and your job is on the line. What You Will Learn Monitor and resolve production issues in your Kafka clusters Provision Kafka clusters with the lowest costs and still handle the required loads Perform root cause analyses of issues affecting your Kafka clusters Know the ways in which your Kafka cluster can affect its consumers and producers Prevent or minimize data loss and delays in data streaming Forestall production issues through an understanding of common failure points Create checklists for troubleshooting your Kafka clusters when problems occur Who This Book Is For Site reliability engineers tasked with maintaining stability of Kafka clusters, Kafka administrators who troubleshoot production issues around Kafka, DevOps and DataOps experts who are involved with provisioning Kafka (whether on-premises or in the cloud), developers of Kafka consumers and producers who wish to learn more about Kafka