Elastic Stack 8.x Cookbook: Over 80 recipes to perform ingestion, search, visualization, and monitoring for actionable insights

Unlock the full potential of Elastic Stack for search, analytics, security, and observability and manage substantial data workloads in both on-premise and cloud environments Key Features • Explore the diverse capabilities of the Elastic Stack through a comprehensive set of recipes • Build search applications, analyze your data, and observe cloud-native applications • Harness powerful machine learning and AI features to create data science and search applications Book Description Learn how to make the most of the Elastic Stack (ELK Stack) products—including Elasticsearch, Kibana, Elastic Agent, and Logstash—to take data reliably and securely from any source, in any format, and then search, analyze, and visualize it in real-time. This cookbook takes a practical approach to unlocking the full potential of Elastic Stack through detailed recipes step by step. Starting with installing and ingesting data using Elastic Agent and Beats, this book guides you through data transformation and enrichment with various Elastic components and explores the latest advancements in search applications, including semantic search and Generative AI. You'll then visualize and explore your data and create dashboards using Kibana. As you progress, you'll advance your skills with machine learning for data science, get to grips with natural language processing, and discover the power of vector search. The book covers Elastic Observability use cases for log, infrastructure, and synthetics monitoring, along with essential strategies for securing the Elastic Stack. Finally, you'll gain expertise in Elastic Stack operations to effectively monitor and manage your system. What you will learn • Discover techniques for collecting data from diverse sources • Visualize data and create dashboards using Kibana to extract business insights • Explore machine learning, vector search, and AI capabilities of Elastic Stack • Handle data transformation and data formatting • Build search solutions from the ingested data • Leverage data science tools for in-depth data exploration • Monitor and manage your system with Elastic Stack Who this book is for This book is for Elastic Stack users, developers, observability practitioners, and data professionals ranging from beginner to expert level. If you’re a developer, you’ll benefit from the easy-to-follow recipes for using APIs and features to build powerful applications, and if you’re an observability practitioner, this book will help you with use cases covering APM, Kubernetes, and cloud monitoring. For data engineers and AI enthusiasts, the book covers dedicated recipes on vector search a Cover Title Page Copyright and Credits Dedication Foreword Contributors Acknowledgments Table of Contents Preface Chapter 1: Getting Started – Installing the Elastic Stack Deploying the Elastic Stack on Elastic Cloud How to do it... How it works... There’s more... Installing the Elastic Stack with ECK Technical requirements Getting ready How to do it... How it works... There’s more... See also Installing a self-managed Elastic Stack Getting ready How to do it... How it works... There’s more... Creating and setting up data tiering Getting ready How to do it on your local machine... How it works (on self-managed)... How to do it on Elastic Cloud... How to do it on ECK... There’s more... See also Creating and setting up additional Elasticsearch nodes Getting ready How to do it... How it works... How to do it on Elastic Cloud... How to do it on ECK... There’s more... See also Creating and setting up Fleet Server Getting ready How to do it on a self-managed Elastic Stack... How it works... Setting up on Elastic Cloud See also Setting up snapshot repository Getting ready How to do it... How it works... There’s more... Chapter 2: Ingesting General Content Data Introducing the Wikipedia Movie Plots dataset Technical requirements Adding data from the Elasticsearch client Getting ready How to do it... How it works... There’s more... Updating data in Elasticsearch Getting ready How to do it... How it works... There’s more... Deleting data in Elasticsearch Getting ready How to do it... How it works... There’s more... See also Using an analyzer Getting ready How to do it... How it works... There’s more... Defining index mapping Getting ready How to do it... How it works... There’s more... See also Using dynamic templates in document mapping Getting ready How to do it... How it works... There’s more... See also Creating an index template Getting ready How to do it... How it works... There’s more... Indexing multiple documents using Bulk API Getting ready How to do it... How it works... There’s more... See also Chapter 3: Building Search Applications Technical requirements Searching with Query DSL Getting ready How to do it... How it works... There’s more... Building advanced search queries with Query DSL Getting ready How to do it... How it works... There’s more... See also Using search templates to pre-render search requests Getting ready How to do it... How it works... There’s more... See also Getting started with Search Applications for your Elasticsearch index Getting ready How to do it... How it works... Building a search experience with the Search Application client Getting ready How to do it... How it works... There’s more... See also Measuring the performance of your Search Applications with Behavioral Analytics Getting ready How to do it... How it works... There’s more... See also Chapter 4: Timestamped Data Ingestion Technical requirements Deploying Elastic Agent with Fleet Getting ready How to do it... How it works... There’s more... See also Monitoring Apache HTTP logs and metrics using the Apache integration Getting ready How to do it... How it works... There’s more... See also Deploying standalone Elastic Agent Getting ready How to do it... How it works... There’s more... See also Adding data using Beats Getting ready How to do it... How it works... There’s more... See also Setting up a data stream manually Dataset Getting ready How to do it... How it works... There’s more... See also Setting up a time series data stream manually Getting ready How to do it... How it works... There’s more... See also Chapter 5: Transform Data Technical requirements Creating an ingest pipeline Getting ready How to do it... How it works... There’s more... See also Enriching data with a custom ingest pipeline for an existing Elastic Agent integration Getting ready How to do it... How it works... There’s more... Using a processor to enrich your data before ingesting with Elastic Agent Getting ready How to do it... How it works... There’s more... See also Installing self-managed Logstash Getting ready How to do it... How it works... There’s more... See also Creating a Logstash pipeline Getting ready How to do it... How it works... There’s more... See also Setting up pivot data transform Getting ready How to do it... How it works... There’s more... See also Setting up the latest data transform Getting ready How to do it... How it works... There’s more... See also Downsampling your time series data Getting ready How to do it... How it works... There’s more... See also Chapter 6: Visualize and Explore Data Technical requirements Exploring your data in Discover Getting ready How to do it... How it works... There’s more... See also Exploring your data with ES|QL Getting ready How to do it... How it works... There’s more... See also Creating visualizations with Kibana Lens Getting ready How to do it... How it works... There’s more... See also Creating visualizations from runtime fields Getting ready How to do it... How it works... There’s more... See also Creating Kibana maps Getting ready How to do it... How it works... There’s more... See also Creating and using Kibana dashboards Getting ready How to do it... How it works... There’s more... See also Creating Canvas workpads Getting ready How to do it... How it works... There’s more... See also Chapter 7: Alerting and Anomaly Detection Technical requirements Creating alerts in Kibana Getting ready How to do it... How it works... There’s more... See also Monitoring alert rules Getting ready How to do it... How it works... There’s more... See also Investigating data with log rate analysis Getting ready How to do it... How it works... There’s more... See also Investigating data with log pattern analysis Getting ready How to do it... How it works... There’s more... Investigating data with change point detection Getting ready How to do it... How it works... There’s more... See also Detecting anomalies in your data with unsupervised machine learning jobs Getting ready How to do it... How it works... There’s more... See also Creating anomaly detection jobs from a Lens visualization Getting ready How to do it... How it works... There’s more... Chapter 8: Advanced Data Analysis and Processing Technical requirements Finding deviations in your data with outlier detection Getting ready How to do it... How it works... See also Building a model to perform regression analysis Getting ready How to do it... How it works... There’s more... See also Building a model for classification Getting ready How to do it... How it works... There’s more... See also Using a trained model for inference Getting ready How to do it... How it works... There’s more... See also Deploying third-party NLP models and testing via the UI Getting ready How to do it... How it works... There’s more... See also Running advanced data processing with trained models Getting ready How to do it... How it works... There’s more... See also Chapter 9: Vector Search and Generative AI Integration Technical requirements Implementing semantic search with dense vectors Getting ready How to do it... How it works... There’s more... See also Implementing semantic search with sparse vectors Getting ready How to do it... How it works... There’s more... See also Using hybrid search to build advanced search applications Getting ready How to do it... How it works... There’s more... See also Developing question-answering applications with Generative AI Getting ready How to do it... How it works... There’s more... See also Using advanced techniques for RAG applications Getting ready How to do it... How it works... There’s more... See also Chapter 10: Elastic Observability Solution Technical requirements Instrumenting your application with Elastic APM Getting ready How to do it... How it works... There’s more... See also Setting up RUM Getting ready How to do it... How it works... There’s more... See also Instrumenting and monitoring with OpenTelemetry Getting ready How to do it... How it works... There’s more... See also Monitoring Kubernetes environments with Elastic Agent Getting ready How to do it... How it works... There’s more... See also Managing synthetics monitoring Getting ready How to do it... How it works... There’s more... See also Gaining comprehensive system visibility with Elastic Universal Profiling Getting ready How to do it... How it works... There’s more... See also Detecting incidents with alerting and machine learning Getting ready How to do it... How it works... There’s more... See also Gaining insights with the AI Assistant Getting ready How to do it... How it works... There’s more... See also Chapter 11: Managing Access Control Technical requirements Using built-in roles Getting ready How to do it... How it works... See also Defining custom roles Getting ready How to do it... How it works... There’s more... See also Granting additional privileges Getting ready How to do it... How it works... There’s more... See also Managing and securing access to Kibana spaces Getting ready How to do it... How it works... There’s more... See also Managing access with API keys Getting ready How to do it... How it works... There’s more... See also Configuring single sign-on Getting ready How to do it... How it works... There’s more... See also Mapping users and groups to roles Getting ready How to do it... How it works... There’s more... Chapter 12: Elastic Stack Operation Technical requirements Setting up an index lifecycle policy Getting ready How to do it... How it works... There’s more... See also Optimizing time series data streams with downsampling Getting ready How to do it... How it works... There’s more... See also Managing the snapshot lifecycle Getting ready How to do it... How it works... There’s more... Configuring Elastic Stack components with Terraform Getting ready How to do it... How it works... There’s more... See also Enabling and configuring cross-cluster search Getting ready How to do it... How it works... There’s more... See also Chapter 13: Elastic Stack Monitoring Technical requirements Setting up Stack Monitoring Getting ready How to do it... How it works... There’s more... See also Building custom visualizations for monitoring data Getting ready How to do it... How it works... There’s more... Monitoring cluster health via an API Getting ready How to do it... How it works... There’s more... See also Enabling audit logging Getting ready How to do it... How it works... There’s more... See also Index Other Books You May Enjoy Unlock the potential of Elastic Stack for search, analytics, security, and observability use cases, and effectively manage substantial data workloads in both on-premise and cloud environments The Elastic Stack (also known as the ELK Stack) is a collection of open-source products - Elasticsearch, Kibana, Elastic Agent, and Logstash that can reliably and securely take data from any source, in any format, and then search, analyze, and visualize it in real-time. This book will give you the knowledge and skills to unlock the full potential of Elastic Stack. In this Cookbook, you'll explore practical recipes, starting with installing and ingesting data using Elastic Agent and Beats, and diving into data transformation and enrichment with various Elastic components. You'll explore the latest advancements in search applications, including semantic search and Generative AI. You'll learn to visualize and explore your data, and create dashboards using Kibana. As you move ahead, you'll advance your skills with machine learning for data science, natural language processing, and vector search. You'll discover Elastic Observability use cases for log, infrastructure, and synthetics monitoring, along with essential strategies for securing the Elastic Stack. Finally, you'll gain expertise in Elastic Stack operations to effectively monitor and manage your system. By the end of the book, you'll acquire the knowledge and skills to build scalable, reliable, and efficient data analytics and search solutions using Elastic Stack. This book is for Elastic Stack users, developers, observability practitioners, and data professionals from beginner to expert level who are looking for practical experience on the Elastic Stack. As a developer, youll find easy-to-follow recipes to use APIs and features to build powerful applications. This book will also help observability practitioners through use cases covering APM, Kubernetes, and cloud monitoring. For data engineers and AI enthusiasts, the book will provide dedicated recipes on vector search and machine learning. No prior knowledge of the Elastic Stack is required. Unlock the full potential of Elastic Stack for search, analytics, security, and observability and manage substantial data workloads in both on-premise and cloud environments Key Features:• Explore the diverse capabilities of the Elastic Stack through a comprehensive set of recipes• Build search applications, analyze your data, and observe cloud-native applications• Harness powerful machine learning and AI features to create data science and search applications Who this book is for:This book is for Elastic Stack users, developers, observability practitioners, and data professionals ranging from beginner to expert level. If you're a developer, you'll benefit from the easy-to-follow recipes for using APIs and features to build powerful applications, and if you're an observability practitioner, this book will help you with use cases covering APM, Kubernetes, and cloud monitoring. For data engineers and AI enthusiasts, the book covers dedicated recipes on vector search and machine learning. No prior knowledge of the Elastic Stack is required.