Practical Debugging at Scale : Cloud Native Debugging in Kubernetes and Production

Overhaul your debugging techniques and master the theory and tools needed to debug and troubleshoot cloud applications in production environments. This book teaches debugging skills that universities often avoid, but that typically consume as much as 60% of our time as developers. The book covers the use of debugger features such as tracepoints, object marking, watch renderers, and more. Author Shai Almog presents a scientific approach to debugging that is grounded in theory while being practical enough to help you to chase stubborn bugs through the maze of a Kubernetes deployment. Practical Debugging at Scale assumes a polyglot environment as is common for most enterprises, but focuses on JVM environments. Most of the tooling and techniques described are applicable to Python, Node, and other platforms, as well as to Java and other JVM languages. The book specifically covers debugging in production, an often-neglected discipline but an all too painful reality. You’ll learn modern techniques around observability, monitoring, logging, and full stack debugging that you can put to immediate use in troubleshooting common ailments in production environments. You Will Learn: The scientific method underlying the process of debugging Debugger capabilities such as tracepoints and marker objects The correct use of less understood features such as exception breakpoints Techniques for tracing issues in production Kubernetes environments Observability and monitoring to resolve production problems Industry best practices for common tooling such as logging Profiling to understand performance and memory problems Who This Book Is For Developers in Java and JVM-related languages who want to improve their debugging skills and production reliability; and developers of cloud applications who are facing the pain of production bugs that are hard to replicate and fix Table of Contents About the Author About the Technical Reviewer Introduction Part I: Basics Chapter 1: Know Your Debugger Basics Watch and Variable Areas Threads Breakpoints Line Breakpoint Method Breakpoint Field Watchpoint Exception Breakpoint Conditional Breakpoints Tracepoints and Suspend State Asynchronous Stack Trace Control Flow Return Immediately and Force Throw Drop Frame Jump to Line and Set Value Watch and Inspection Rendering Custom Rendering Mute Renderers Show Objects Memory Debugger Object Marking Filtering Collections and Arrays Stream Debugger Debugger Annotations Dependency Custom Renderers in Code Stack Trace Hinting Summary Chapter 2: The Checklist Documentation Why Is It Completely Wrong? You Should Still Read the Documentation Classification and Game Plan “It Works on My Machine” Bug That’s Inconsistent Give Up Isolating Assumptions Everybody Lies Double Verification Defining the Assumptions The Right Direction Assumption Jump Figure Working Code Common Problems and Solutions Exceptions or Faults Bad Application State Concurrency, Timing, and Races Performance and Resource Starvation Elimination Unit Testing and Mocking Eliminating in Flaky Environments Understand the Tongs Process Process Issue Daily Meetings Test Tips and Tricks Rubber Ducking Redefine the Bug Change the Angle Disrupt Your Environment Switch Environments Use Developer Extension Tooling Walk Away/Sleep It’s like a Game The Checklist Summary Chapter 3: The Auxiliary Tools DTrace Running DTrace Basic Usage Simple Usage strace Running Strace Strace and Java Track Regressions with Git Bisect In the Old Days Find the Bug. Automatically! Can We Automate This? JMXTerm How Does JMX Work? JMXTerm Basics jhsdb Basics of jhsdb debugd jstack jmap jinfo jsnap GUI Debugger Wireshark Getting Started with Wireshark Using Wireshark As a Debugging Tool tcpdump Summary Chapter 4: Logging, Testing, and Fail-Fast Logging Too Much of a “Good Thing” Use MDC (Mapped Diagnostic Context) Should I Log? Testing Role of Tests in the Debug – Fix Cycle Using the Debugger to Compose Unit Tests Writing Good Tests Writing Debuggable Code Fail-Fast vs. Fail-Safe Which One Should We Pick? Hybrid Environment in the Cloud Defining Failure Don’t Fix the Bug Debugging Failure Hybrid Fail-Safe Summary Chapter 5: Time Travel Debugging Why Haven’t They Won? Their Usage Is Awkward No One Ruled Them All The Different Types Getting Started Frontend Debugging with Replay Low-Level Debugging with rr Summary Part II: The Modern Production Environment Chapter 6: Debugging Kubernetes Short Kubernetes Introduction History Enter Kubernetes It’s Not Just About Scale The Problem with Kubernetes Ephemeral Containers Source Debugging Using jdb Remote Debugging Basics The Problems with JDWP Legal and Privacy Issues Breakpoints Break Scale: Breadth and Depth Configuration Troubleshooting CrashLoopBackOff Image Error Node Not Ready CreateContainerConfigError Debugging the Dockerfile Summary Chapter 7: Serverless Debugging Basics Idempotency Staging and Dev Local Debugging Feature Flags Staged Rollouts and Canaries Summary Chapter 8: Fullstack Debugging Frontend Debugging Obfuscation Launch Debugger DOM Breakpoints XHR Breakpoints Faking Browser Settings Frontend Logging Networking Storage DOM and CSS Database Layer Database Logic Layering Database Connectivity Invalid Data Consistent Databases Summary Chapter 9: Observability and Monitoring Theory Logs Metrics Distributed Tracing Prometheus Grafana OpenTelemetry An Observability Culture Coverage Deciding Developer Dashboard Summary Chapter 10: Developer Observability What Is Developer Observability? Remote Debugging? Basics Dynamic Logging Conditions Collaboration Snapshots Metrics Under the Hood Security Segregation Accountability Agent Summary Part III: In Practice Chapter 11: Tools of Learning Before the Debugger Moving to a Debugger Standard Debugger Techniques Who Changes This Variable? Why Do We Need This Code? Keeping Track What’s Going On? Learning with Observability Dashboard Exploration Log Pinpointing Learning with Developer Observability Summary Chapter 12: Performance and Memory Performance Measuring Basics JMeter Overview Working with JMeter Basics of Profilers Debugging Performance A Hacker Profiler Pinpoint Precision Error Rates Debugging Cache Misses Debugging Memory Issues Developer Observability and Performance Summary Chapter 13: Security Security Basics An Example Tips and Best Practices Security Review Injection ID Scanning Serialization Issues Internal Security Don’t Open Holes Refine Logging Dealing with a Zero-Day Summary Chapter 14: Bug Strategies The Impossible Database Entries Lessons Learned The Seven-Minute Process Lessons Learned Going Bankrupt in PaaS Lessons Learned The Fast Network Lessons Learned The Missing Wing Lessons Learned The Microservice Lessons Learned Long-Run Fail Lessons Learned Summary Appendix A: Resources Other People My Resources Index