Splunk Best Practices

Cover 1 Copyright 3 Credits 4 About the Author 5 About the Reviewer 8 www.PacktPub.com 9 Table of Contents 10 Preface 15 Chapter 1: Application Logging 22 Loggers 24 Anatomy of a log 24 Log4* 25 Pantheios 25 Logging ? logging facility for Python 25 Example of a structured log 25 Data types 27 Structured data ? best practices 27 Log events 28 Common Log Format 28 Automatic Delimited Value Extraction (IIS/Apache) ? best practice 30 Manual Delimited Value Extraction with REGEX 32 Step 1 ? field mapping ? best practice 32 Step 2 ? adding the field map to structure the data (props/transforms) 33 Use correlation IDs? best practice 34 Correlation IDs and publication transactions ? best practice 36 Correlation IDs and subscription transactions ? best practices 37 Correlation IDs and database calls ? best practices 38 Unstructured data 38 Event breaking ? best practice 39 Best practices 40 Configuration transfer ? best practice 43 Summary 47 Chapter 2: Data Inputs 48 Agents 48 Splunk Universal Forwarder 48 Splunk Heavy Forwarder 48 Search Head Forwarder 49 Data inputs 49 [API inputs] 49 API inputs 49 Database inputs 50 Monitoring inputs 51 Scripted inputs 51 Custom or not 52 Modular inputs 54 Windows inputs 60 Windows event logs / Perfmon 61 Deployment server 61 Know your data 68 Long delay intervals with lots of data 69 Summary 70 Chapter 3: Data Scrubbing 71 Heavy Forwarder management 72 Managing your Heavy Forwarder 77 Manual administration 77 Deployment server 77 Important configuration files 78 Even data distribution 78 Common root cause 81 Knowledge management 83 Handling single- versus multi-line events 84 Manipulating raw data (pre-indexing) 85 Routing events to separate indexes 85 Black-holing unwanted events (filtering) 87 Masking sensitive data 88 Pre-index data masking 89 Post-index data masking 89 Setting a hostname per event 91 Summary 91 Chapter 4: Knowledge Management 93 Anatomy of a Splunk search 93 [Root search] 93 Root search 93 Calculation/evaluation 94 Presentation/action 94 Best practices with search anatomy 95 The root search 95 Calculation/evaluation 95 Presentation/action 96 Knowledge objects 97 Eventtype Creation 98 Creation through the Splunk UI 98 Creation through the backend shell 100 Field extractions 104 Performing field extractions 104 Pre-indexing field extractions (index time) 105 Post-indexing field extractions (search time) 105 Creating index time field extractions 105 Creating search time field extractions 107 Creating field extractions using IFX 107 Creation through CLI 110 Summary 112 Chapter 5: Alerting 113 Setting expectations 114 Time is literal, not relative 115 To quickly summarize 117 Be specific 117 To quickly summarize 119 Predictions 119 To quickly summarize 120 Anatomy of an alert 120 Search query results 121 Alert naming 121 The schedule 121 The trigger 121 The action 122 Throttling 122 Permissions 122 Location of action scripts 122 Example 122 Custom commands/automated self-healing 130 A word of warning 132 Summary 133 Chapter 6: Searching and Reporting 134 General practices 135 Core fields (root search) 135 _time 136 Index 136 Sourcetype 136 Host 136 Source 136 Case sensitivity 137 Inclusive versus exclusive 138 Search modes 138 Fast Mode 139 Verbose Mode 140 Smart Mode (default) 141 Advanced charting 142 Overlay 143 Host CPU / MEM utilization 143 Xyseries 146 Appending results 148 timechart 148 stats 151 The Week-over-Week-overlay 153 Day-over-day overlay 154 SPL to overlay (the hard way) 154 Timewrap (the easy way) 155 Summary 155 Chapter 7: Form-Based Dashboards 156 Dashboards versus reports 157 Reports 157 Dashboards 161 [Form-based] 161 Form-based 161 Drilldown 162 Report/data model-based 162 Search-based 162 Modules 162 Data input 163 Chart 163 Table 163 Single value 163 Map module 163 Tokens 163 Building a form-based dashboard 165 Summary 173 Chapter 8: Search Optimization 174 Types of dashboard search panel 174 Raw data search panel 175 Shared search panel (base search) 175 Report reference panel 175 Data model/pivot reference panels 175 Raw data search 175 Shared searching using a base search 179 Creating a base search 181 Referencing a base search 182 Report referenced panels 183 Data model/pivot referenced panels 188 Special notes 192 Summary 193 Chapter 9: App Creation and Consolidation 194 Types of apps 195 Search apps 195 Deployment apps 195 Indexer/cluster apps 195 Technical add-ons 196 Supporting add-ons 196 Premium apps 196 Consolidating search apps 197 Creating a custom app 197 App migrations 198 Knowledge objects 199 Dashboard consolidation 200 Search app navigation 207 Consolidating indexing/forwarding apps 210 Forwarding apps 211 Indexer/cluster apps 213 Summary 215 Chapter 10: Advanced Data Routing 216 Splunk architecture 217 Clustering 217 Search head clustering 217 Indexer cluster 217 Multi-site redundancy 218 Leveraging load balancers 218 Failover methods 218 Putting it all together 220 Network segments 221 Production 221 Standard Integration Testing (SIT) 221 Quality assurance 222 Development 222 The DMZ (App Tier) 223 The data router 224 Building roads and maps 225 Building the UF input/output paths 227 Building the HF input/output paths 228 If you build it, they will come 231 Summary 231 Index 233

Design, implement, and publish custom Splunk applications by following best practices

About This Book

• This is the most up-to-date guide on the market and will help you finish your tasks faster, easier, and more efficiently.
• Highly practical guide that addresses common and not-so-common pain points in Splunk.
• Want to explore shortcuts to perform tasks more efficiently with Splunk? This is the book for you!

Who This Book Is For

This book is for administrators, developers, and search ninjas who have been using Splunk for some time. A comprehensive coverage makes this book great for Splunk veterans and newbies alike.

What You Will Learn

• Use Splunk effectively to gather, analyze, and report on operational data throughout your environment
• Expedite your reporting, and be empowered to present data in a meaningful way
• Create robust searches, reports, and charts using Splunk
• Modularize your programs for better reusability.
• Build your own Splunk apps and learn why they are important
• Learn how to integrate with enterprise systems
• Summarize data for longer term trending, reporting, and analysis

In Detail

This book will give you an edge over others through insights that will help you in day-to-day instances. When you're working with data from various sources in Splunk and performing analysis on this data, it can be a bit tricky. With this book, you will learn the best practices of working with Splunk.

You'll learn about tools and techniques that will ease your life with Splunk, and will ultimately save you time. In some cases, it will adjust your thinking of what Splunk is, and what it can and cannot do.

To start with, you'll get to know the best practices to get data into Splunk, analyze data, and package apps for distribution. Next, you'll discover the best practices in logging, operations, knowledge management, searching, and reporting. To finish off, we will teach you how to troubleshoot Splunk searches, as well as deployment, testing, and development with Splunk.

Style and approach

If you're stuck or want to find a better way to work with Splunk environment, this book will come handy. This easy-to-follow, insightful book contains step-by-step instructions and examples and scenarios that you will connect to.

**Design, implement, and publish custom Splunk applications by following best practices** About This Book * This is the most up-to-date guide on the market and will help you finish your tasks faster, easier, and more efficiently. * Highly practical guide that addresses common and not-so-common pain points in Splunk. * Want to explore shortcuts to perform tasks more efficiently with Splunk? This is the book for you! Who This Book Is For This book is for administrators, developers, and search ninjas who have been using Splunk for some time. A comprehensive coverage makes this book great for Splunk veterans and newbies alike. What You Will Learn * Use Splunk effectively to gather, analyze, and report on operational data throughout your environment * Expedite your reporting, and be empowered to present data in a meaningful way * Create robust searches, reports, and charts using Splunk * Modularize your programs for better reusability. * Build your own Splunk apps and learn why they are important * Learn how to integrate with enterprise systems * Summarize data for longer term trending, reporting, and analysis In Detail This book will give you an edge over others through insights that will help you in day-to-day instances. When you're working with data from various sources in Splunk and performing analysis on this data, it can be a bit tricky. With this book, you will learn the best practices of working with Splunk. You'll learn about tools and techniques that will ease your life with Splunk, and will ultimately save you time. In some cases, it will adjust your thinking of what Splunk is, and what it can and cannot do. To start with, you'll get to know the best practices to get data into Splunk, analyze data, and package apps for distribution. Next, you'll discover the best practices in logging, operations, knowledge management, searching, and reporting. To finish off, we will teach you how to troubleshoot Splunk searches, as well as deployment, testing, and development with Splunk. Style and approach If you're stuck or want to find a better way to work with Splunk environment, this book will come handy. This easy-to-follow, insightful book contains step-by-step instructions and examples and scenarios that you will connect to. About This BookThis is the most up-to-date guide on the market and will help you finish your tasks faster, easier, and more efficiently.Highly practical guide that addresses common and not-so-common pain points in Splunk.Want to explore shortcuts to perform tasks more efficiently with Splunk? This is the book for you!Who This Book Is ForThis book is for administrators, developers, and search ninjas who have been using Splunk for some time. A comprehensive coverage makes this book great for Splunk veterans and newbies alike.What You Will LearnUse Splunk effectively to gather, analyze, and report on operational data throughout your environmentExpedite your reporting, and be empowered to present data in a meaningful wayLearn how to integrate with enterprise systemsCreate robust searches, reports, and charts using SplunkModularize your programs for better reusabilityBuild your own Splunk apps and learn why they are importantSummarize data for longer term trending, reporting, and analysisIn DetailThis book will give you an edge over others through insights that will help you in day-to-day instances. When you're working with data from various sources in Splunk and performing analysis on this data, it can be a bit tricky. With this book, you will learn the best practices of working with Splunk.You'll learn about tools and techniques that will ease your life with Splunk, and will ultimately save you time. In some cases, it will adjust your thinking of what Splunk is, and what it can and cannot do. To start with, you'll get to know the best practices to get data into Splunk, analyze data, and package apps for distribution. Next, you'll discover the best practices in logging, operations, knowledge management, searching, and reporting. To finish off, we will teach you how to troubleshoot Splunk searches, as well as deployment, testing, and development with Splunk. About This BookThis is the most up-to-date guide on the market and will help you finish your tasks faster, easier, and more efficiently.Highly practical guide that addresses common and not-so-common pain points in Splunk.Want to explore shortcuts to perform tasks more efficiently with Splunk? This is the book for you!Who This Book Is For This book is for administrators, developers, and search ninjas who have been using Splunk for some time. A comprehensive coverage makes this book great for Splunk veterans and newbies alike. What You Will LearnUse Splunk effectively to gather, analyze, and report on operational data throughout your environmentExpedite your reporting, and be empowered to present data in a meaningful wayLearn how to integrate with enterprise systemsCreate robust searches, reports, and charts using SplunkModularize your programs for better reusabilityBuild your own Splunk apps and learn why they are importantSummarize data for longer term trending, reporting, and analysisIn Detail This book will give you an edge over others through insights that will help you in day-to-day instances. When you're working with data from various sources in Splunk and performing analysis on this data, it can be a bit tricky. With this book, you will learn the best practices of working with Splunk. You'll learn about tools and techniques that will ease your life with Splunk, and will ultimately save you time. In some cases, it will adjust your thinking of what Splunk is, and what it can and cannot do. To start with, you'll get to know the best practices to get data into Splunk, analyze data, and package apps for distribution. Next, you'll discover the best practices in logging, operations, knowledge management, searching, and reporting. To finish off, we will teach you how to troubleshoot Splunk searches, as well as deployment, testing, and development