Monitoring Cloud-Native Applications : Lead Agile Operations Confidently Using Open Source Software

Table of Contents 5 About the Authors 8 About the Technical Reviewer 9 Introduction 10 Part I: Architecture of Modern Monitoring Systems for Cloud Native Applications 12 Chapter 1: Introduction to Modern Monitoring 13 Microservices 14 Containers 16 Monitoring 18 Importance of Monitoring 18 DevOps—Roles and User Personas 19 Monitoring Domains 21 Reactive vs. Proactive Monitoring 25 Monitoring System 26 Need for a Monitoring System 26 Features of a Modern Monitoring System 28 Push vs. Pull Monitoring 30 Advent of AIOps 31 Kubernetes 32 Summary 34 Chapter 2: Observability 35 Observability 35 Three Pillars of Observability 36 Metrics 38 Time Series 39 Granularity/Resolution 39 Metadata Tags 40 Categories of Metrics 41 Work Metrics 42 Resource Metrics 43 Methods of Measurement 44 USE Method 44 RED Method 45 Four Golden Signals 47 Types of Metrics 48 Gauges 48 Counters 49 Timers 51 Histograms 51 Summary 52 Statistical Functions 53 Events 54 Observability of Kubernetes 55 Sources of Metrics in Kubernetes 57 Kubernetes API Server 58 cAdvisor 59 Kubernetes Metrics Server 60 Kube-State-Metrics 60 Kubernetes Internal Metrics 61 Operations in a Kubernetes Environment 63 Chapter 3: Architecture of a Modern Monitoring System 65 Modern Monitoring System Architecture 66 Data Collection 67 Instrumenting Your Code 67 Exposition 70 Bridges 71 Parsers 71 Third-Party Exporters 71 Agents 71 collectd 71 StatsD 72 Telegraf 72 fluentd 72 OpenTelemetry 73 OpenTelemetry Collector 75 OpenTelemetry Collector Architecture 75 Agent 76 Gateway 76 Open Telemetry Agent 77 Data Storage 77 LSM-Tree 78 Time-Series Data 79 TSDB 81 Time-Series Data Models 82 Modeling by Data Source 83 Modeling by Metrics 83 Data Analysis and Query Engine 84 Filter 84 Aggregation 86 Grouping 87 Downsampling 87 Auto-rollup 88 Data Visualization 89 Line Graphs 90 Stacked Graphs 91 Bar Graphs 92 Heatmaps 93 Summary Graphs 93 Aggregation 94 Alerting Engine 95 Alerting Strategy 97 Features of a Modern Alerting System 99 Log Aggregation 100 Log Aggregation Architecture 101 Ingestion Pipeline 102 Logging in a Kubernetes Environment 103 Best Practices for Logging 104 Summary 105 Part II: Open Source Monitoring Tools 107 Chapter 4: Prometheus 108 Installation 109 Architecture 119 Data Collection 119 Exposition Format 120 Service Discovery 124 Targets 125 Data Storage 127 Visualization 130 Alerting 138 Grouping 138 Routing 138 Inhibition 139 Silences 140 Chapter 5: TICK Stack 141 Installation 142 Key Concepts in InfluxDB 150 Organization 150 Creating an Organization 150 Renaming an Organization 153 Deleting an Organization 156 Members 156 Creating a New User 157 Updating a Username 159 Removing a Member 160 Deleting a User 161 Buckets 162 Data Collection 162 Scraper 162 Telegraf 164 Telegraf Installation 164 Manual Telegraf Configuration 170 Data Storage 175 InfluxDB Data Elements and Schema 176 Tables 180 Line Protocol 181 Influx Query Language 181 Alerting 183 Configuring a Check 184 Setting Thresholds 187 Connecting to a Notification Endpoint 190 Visualization 190 Summary 194 Chapter 6: Grafana 195 Installation 196 Adding a Data Source 203 Creating a Dashboard 210 Panels 210 Panel Editor 211 Visualization 219 Stat 219 Gauge 220 Bar Gauge 220 Display Settings 220 Dashboards 224 Dashboard Header 229 Rows 234 Manage Dashboards 235 Dashboard Folder 236 Playlists 237 Explore 239 Alerting 240 State History 243 Test Rule 244 Adding a Notification Channel 246 Summary 248 Index 249 Introduce Yourself To The Nuances Of Modern Monitoring For Cloud-native Applications Running On Kubernetes Clusters. This Book Will Help You Get Started With The Concepts Of Monitoring, Introduce You To Popular Open-source Monitoring Tools, And Help With Finding The Correct Set Of Use Cases For Their Implementation. It Covers The In-depth Technical Details Of Open-source Software Used In Modern Monitoring Systems That Are Tailor Made For Environments Running Microservices. Monitoring Cloud-native Applications Is Divided Into Two Parts. Part 1 Starts With An Introduction To Cloud-native Applications And The Foundational Concepts Of Monitoring. It Then Walks You Through The Various Aspects Of Monitoring Containerized Workloads Using Kubernetes As The De-facto Orchestration Platform. You Will Dive Deep Into The Architecture Of A Modern Monitoring System And Look At Its Individual Components In Detail. Part 2 Introduces You To Popular Open-source Tools Which Are Used By Enterprises And Startups Alike And Are Well Established As The Tools Of Choice For Industry Stalwarts. First Off, You Will Look At Prometheus And Understand Its Architecture And Usage. You Will Also Learn About Influxdb, Formerly Called Tick Stack (telegraf, Influxdb, Chronograf, And Kapacitor). You Will Explore The Technical Details Of Its Architecture And The Use Cases Which It Solves. Your Next Stop Will Be Elastic Stack, Formerly Known As The Elk Stack (elasticsearch, Logstash, And Kibana) And You Will Examine The Scenarios Where You Can Use It Effectively. In The Next Chapter, You Will Be Introduced To Grafana, A Multi-platform Open Source Analytics And Interactive Visualization Tool That Can Help You With Visualization Of Data And Dashboards. After Reading This Book, You Will Have A Much Better Understanding Of Key Terminologies And General Concepts Around Monitoring And Observability. You Will Also Be Able To Select A Suitable Monitoring Solution From The Bouquet Of Open-source Monitoring Solutions Available For Applications, Microservices, And Containers. Armed With This Knowledge, You Will Be Better Prepared To Design And Lead A Successful Agile Operations Team. What You Will Learn Monitor And Observe Of Metrics, Events, Logs, And Traces Carry Out Infrastructure And Application Monitoring For Microservices Architecture Analyze And Visualize Collected Data Use Alerting, Reporting, And Automated Actions For Problem Resolution Who This Book Is For Devops Administrators, Cloud Administrators, And Site Reliability Engineers (sres) Who Manage And Monitor Applications And Cloud Infrastructure On A Day-to-day Basis Within Their Organizations.