Cloud Observability in Action

Don’t fly blind. Observability gives you actionable insights into your cloud native systems—from pinpointing errors, to increasing developer productivity, to tracking compliance. Observability is the difference between an error message and an error explanation with a recipe how to resolve the error! You know exactly which service is affected, who’s responsible for its repair, and even how it can be optimized in the future. Cloud Observability in Action teaches you how to set up an observability system that learns from a cloud application’s signals, logging, and monitoring, all using free and open source tools. In Cloud Observability in Action you will learn how to: • Apply observability in cloud native systems • Understand observability signals, including their costs and benefits • Apply good practices around instrumentation and signal collection • Deliver dashboarding, alerting, and SLOs/SLIs at scale • Choose the correct signal types for given roles or tasks • Pick the right observability tool for any given function • Communicate the benefits of observability to management A well-designed observability system provides insight into bugs and performance issues in cloud native applications. They help your development team understand the impact of code changes, measure optimizations, and track user experience. Best of all, observability can even automate your error handling so that machine users apply their own fixes—no more 3AM calls for emergency outages. About the technology Cloud native systems are made up of hundreds of moving parts. When something goes wrong, it’s not enough to know there is a problem—you need to know where it is, what it is, and how to fix it. This book takes you beyond traditional monitoring, explaining observability systems that turn application telemetry into actionable insights. About the book Cloud Observability in Action gives you the background and techniques you need to successfully introduce observability into cloud-based serverless and Kubernetes environments. In it, you’ll learn to use open standards and tools like OpenTelemetry, Prometheus, and Grafana to build your own observability system and end reliance on proprietary software. You’ll discover insights from different telemetry signals, including logs, metrics, traces, and profiles. Plus, the book’s rigorous cost-benefit analysis ensures you’re getting a real return on your observability investment. What's inside • Observability in and of cloud native systems • Dashboarding, alerting, and SLOs/SLIs at scale • Signal types for any role or task • State-of-the-art open source observability tools About the reader For application developers, platform owners, DevOps, and SREs. About the author Michael Hausenblas is a Product Owner in the AWS open source observability team. Cloud Observability in Action brief contents contents preface acknowledgments about this book Who should read this book How this book is organized About the code liveBook discussion forum Online resources about the author about the cover illustration 1 End-to-end observability 1.1 What is observability? 1.2 Observability use cases 1.3 Roles and goals 1.4 Example microservices app 1.5 Challenges and how observability helps 1.5.1 Return on investment 1.5.2 Signal correlation 1.5.3 Portability Summary 2 Signal types 2.1 Reference example 2.2 Assessing instrumentation costs 2.3 Logs 2.3.1 Instrumentation 2.3.2 Telemetry 2.3.3 Costs and benefits 2.3.4 Observability with logs 2.4 Metrics 2.4.1 Instrumentation 2.4.2 Telemetry 2.4.3 Costs and benefits 2.4.4 Observability with metrics 2.5 Traces 2.5.1 Instrumentation 2.5.2 Telemetry 2.5.3 Costs and benefits 2.5.4 Observability with traces 2.6 Selecting signals Summary 3 Sources 3.1 Selecting sources 3.2 Compute-related sources 3.2.1 Basics 3.2.2 Containers 3.2.3 Kubernetes 3.2.4 Serverless compute 3.3 Storage-related sources 3.3.1 Relational databases and NoSQL data stores 3.3.2 File systems and object stores 3.4 Network-related sources 3.4.1 Network interfaces 3.4.2 Higher-level network sources 3.5 Your code 3.5.1 Instrumentation 3.5.2 Proxy sources Summary 4 Agents and instrumentation 4.1 Log routers 4.1.1 Fluentd and Fluent Bit 4.1.2 Other log routers 4.2 Metrics collection 4.2.1 Prometheus 4.2.2 Other metrics agents 4.3 OpenTelemetry 4.3.1 Instrumentation 4.3.2 Collector 4.4 Other agents 4.5 Selecting an agent 4.5.1 Security for and of the agent 4.5.2 Agent performance and resource usage 4.5.3 Agent nonfunctional requirements Summary 5 Backend destinations 5.1 Backend destination terminology 5.2 Backend destinations for logs 5.2.1 Cloud providers 5.2.2 Open source log backends 5.2.3 Commercial offerings for log backends 5.3 Backend destinations for metrics 5.3.1 Cloud providers 5.3.2 Open source metrics backends 5.3.3 Commercial offerings for metrics backends 5.4 Backend destinations for traces 5.4.1 Cloud providers 5.4.2 Open source traces backends 5.4.3 Commercial offerings for trace backends 5.5 Columnar data stores 5.6 Selecting backend destinations 5.6.1 Costs 5.6.2 Open standards 5.6.3 Back pressure 5.6.4 Cardinality and queries Summary 6 Frontend destinations 6.1 Frontends 6.1.1 Grafana 6.1.2 Kibana and OpenSearch Dashboards 6.1.3 Other open source frontends 6.1.4 Cloud providers and commercial frontends 6.2 All-in-ones 6.2.1 CNCF Jaeger 6.2.2 CNCF Pixie 6.2.3 Zipkin 6.2.4 Apache SkyWalking 6.2.5 SigNoz 6.2.6 Uptrace 6.2.7 Commercial offerings 6.3 Selecting frontends and all-in-ones Summary 7 Cloud operations 7.1 Incident management 7.1.1 Health and performance monitoring 7.1.2 Handling the incident 7.1.3 Learning from the incident after the fact 7.2 Alerting 7.2.1 Prometheus alerting 7.2.2 Using Grafana for alerting 7.2.3 Cloud providers 7.3 Usage tracking 7.3.1 Users 7.3.2 Costs Summary 8 Distributed tracing 8.1 Intro and terminology 8.1.1 Motivational example 8.1.2 Terminology 8.1.3 Use cases 8.2 Using distributed tracing in a microservices app 8.2.1 Example app overview 8.2.2 Implementing the example app 8.2.3 The “happy path” 8.2.4 Exploring a failure in the example app 8.3 Practical considerations 8.3.1 Sampling 8.3.2 Observability tax 8.3.3 Traces vs. metrics vs. logs Summary 9 Developer observability 9.1 Continuous profiling 9.1.1 The humble beginnings 9.1.2 Common technologies 9.1.3 Open source CP tooling 9.1.4 Commercial continuous profiling offerings 9.1.5 Using continuous profiling to assess continuous profiling 9.2 Developer productivity 9.2.1 Challenges 9.2.2 Tooling 9.3 Tooling considerations 9.3.1 Symbolization 9.3.2 Storing profiles 9.3.3 Querying profiles 9.3.4 Correlation 9.3.5 Standards 9.3.6 Using tooling in production Summary 10 Service level objectives 10.1 The fundamentals of SLOs 10.1.1 Types of services 10.1.2 Service level indicator 10.1.3 Service level objective 10.1.4 Service level agreement 10.2 Implementing SLOs 10.2.1 High-level example 10.2.2 Using Prometheus to implement SLOs 10.2.3 Commercial SLO offerings 10.3 Considerations Summary 11 Signal correlation 11.1 Correlation fundamentals 11.1.1 Correlation with OpenTelemetry 11.1.2 Correlating traces 11.1.3 Correlating metrics 11.1.4 Correlating logs 11.1.5 Correlating profiles 11.2 Using Prometheus, Jaeger, and Grafana for correlation 11.2.1 Metrics–traces correlation example setup 11.2.2 Using metrics–traces correlation 11.3 Signal correlation support in commercial offerings 11.4 Considerations 11.4.1 Early days 11.4.2 Signals 11.4.3 User experience 11.5 Conclusion Summary appendix—A Kubernetes end-to-end example A.1 Overview A.2 Prerequisites A.3 Demo walk-through A.3.1 Installing the demo A.3.2 Using the demo index A B C D E F G H I J K L M N O P Q R S T U V W Z Generate actionable insights about your cloud native systems. This book teaches you how to set up an observability system that learns from a cloud applications signals, logging, and monitoring using free and open source tools. In Cloud Observability in Action you will learn how Cloud native, serverless, and containerized applications are made up of hundreds of moving parts. When something goes wrong, its not enough to just know there is a problemyou need to know where it is, what it is, and even how to fix it. Cloud Observability in Action shows you how to go beyond the traditional monitoring and build observability systems that turn application telemetry into actionable insight. Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications. About the technology A well-designed observability system provides insight into bugs and performance issues in cloud native applications. Often, observability is the difference between an error message and an explanation! You know exactly which service is affected, whos responsible for its repair, and even how it can be optimized in the future. Best of all, observability allows you to easily automate your error handling with machine users applying fixes without any human help. About the book Cloud Observability in Action teaches you to apply observability practices to cloud-based serverless and Kubernetes environments. In this one-of-a-kind guide, author Michael Hausenblas shares insights from his extensive experience building, monitoring, and improving cloud native systems. Youll use open source tools like Prometheus and Grafana to build your own observability system without having to rely on proprietary software. Learn how to use telemetry and destinations to continuously generate and discover insights from different signals, including logs, metrics, traces, and profiles. Throughout, use cases and rigorous cost-benefit analysis make sure youre getting a real return on your investment in observability. About the reader For developers and SREs who have worked with cloud native applications. This book can be used with any public cloud. About the author Michael Hausenblas is a Solution Engineering Lead in the AWS open source observability service team. He covers Prometheus, Grafana, and OpenTelemetry upstream and in managed services. Before Amazon, Michael worked at Red Hat, Mesosphere (now D2iQ), and MapR. Don’t fly blind. Observability gives you actionable insights into your cloud native systems—from pinpointing errors, to increasing developer productivity, to tracking compliance. In Cloud Observability in Action you will learn how to:• Apply observability in cloud native systems• Understand observability signals, including their costs and benefits• Apply good practices around instrumentation and signal collection• Deliver dashboarding, alerting, and SLOs/SLIs at scale• Choose the correct signal types for given roles or tasks• Pick the right observability tool for any given function• Communicate the benefits of observability to management ----------Generate actionable insights about your cloud native systems. This book teaches you how to set up an observability system that learns from a cloud application’s signals, logging, and monitoring using free and open source tools. Cloud Observability in Action teaches you to apply observability practices to cloud-based serverless and Kubernetes environments. In this one-of-a-kind guide, author Michael Hausenblas shares insights from his extensive experience building, monitoring, and improving cloud native systems. You’ll use open source tools like Prometheus and Grafana to build your own observability system without having to rely on proprietary software. Learn how to use telemetry and destinations to continuously generate and discover insights from different signals, including logs, metrics, traces, and profiles. Throughout, use cases and rigorous cost-benefit analysis make sure you’re getting a real return on your investment in observability.