Elastic Stack 8.x Cookbook: Over 80 recipes to perform ingestion, search, visualization, and monitoring for actionable insights

Unlock the full potential of Elastic Stack for search, analytics, security, and observability and manage substantial data workloads in both on-premise and cloud environments Key Features Explore the diverse capabilities of the Elastic Stack through a comprehensive set of recipes Build search applications, analyze your data, and observe cloud-native applications Harness powerful machine learning and AI features to create data science and search applications Purchase of the print or Kindle book includes a free PDF eBook Book Description Learn how to make the most of the Elastic Stack (ELK Stack) products—including Elasticsearch, Kibana, Elastic Agent, and Logstash—to take data reliably and securely from any source, in any format, and then search, analyze, and visualize it in real-time. This cookbook takes a practical approach to unlocking the full potential of Elastic Stack through detailed recipes step by step. Starting with installing and ingesting data using Elastic Agent and Beats, this book guides you through data transformation and enrichment with various Elastic components and explores the latest advancements in search applications, including semantic search and Generative AI. You'll then visualize and explore your data and create dashboards using Kibana. As you progress, you'll advance your skills with machine learning for data science, get to grips with natural language processing, and discover the power of vector search. The book covers Elastic Observability use cases for log, infrastructure, and synthetics monitoring, along with essential strategies for securing the Elastic Stack. Finally, you'll gain expertise in Elastic Stack operations to effectively monitor and manage your system. What you will learn Discover techniques for collecting data from diverse sources Visualize data and create dashboards using Kibana to extract business insights Explore machine learning, vector search, and AI capabilities of Elastic Stack Handle data transformation and data formatting Build search solutions from the ingested data Leverage data science tools for in-depth data exploration Monitor and manage your system with Elastic Stack Who this book is for This book is for Elastic Stack users, developers, observability practitioners, and data professionals ranging from beginner to expert level. If you’re a developer, you’ll benefit from the easy-to-follow recipes for using APIs and features to build powerful applications, and if you’re an observability practitioner, this book will help you with use cases covering APM, Kubernetes, and cloud monitoring. For data engineers and AI enthusiasts, the book covers dedicated recipes on vector search and machine learning. No prior knowledge of the Elastic Stack is required. Elastic Stack 8.x Cookbook Foreword Contributors About the authors About the reviewers Acknowledgments Preface Who this book is for What this book covers To get the most out of this book Download the example code files Conventions used Sections Getting ready How to do it... How it works... There’s more... See also Get in touch Share Your Thoughts Download a free PDF copy of this book 1 Getting Started – Installing the Elastic Stack Deploying the Elastic Stack on Elastic Cloud How to do it... How it works... There’s more... Installing the Elastic Stack with ECK Technical requirements Getting ready How to do it... How it works... There’s more... See also Installing a self-managed Elastic Stack Getting ready How to do it... How it works... There’s more... Creating and setting up data tiering Getting ready How to do it on your local machine... How it works (on self-managed)... How to do it on Elastic Cloud... How to do it on ECK... There’s more... See also Creating and setting up additional Elasticsearch nodes Getting ready How to do it... How it works... How to do it on Elastic Cloud... How to do it on ECK... There’s more... See also Creating and setting up Fleet Server Getting ready How to do it on a self-managed Elastic Stack... How it works... Setting up on Elastic Cloud See also Setting up snapshot repository Getting ready How to do it... How it works... There’s more... 2 Ingesting General Content Data Introducing the Wikipedia Movie Plots dataset Technical requirements Adding data from the Elasticsearch client Getting ready How to do it... How it works... There’s more... Updating data in Elasticsearch Getting ready How to do it... How it works... There’s more... Deleting data in Elasticsearch Getting ready How to do it... How it works... There’s more... See also Using an analyzer Getting ready How to do it... How it works... There’s more... Defining index mapping Getting ready How to do it... How it works... There’s more... See also Using dynamic templates in document mapping Getting ready How to do it... How it works... There’s more... See also Creating an index template Getting ready How to do it... How it works... There’s more... Indexing multiple documents using Bulk API Getting ready How to do it... How it works... There’s more... See also 3 Building Search Applications Technical requirements Searching with Query DSL Getting ready How to do it... How it works... There’s more... Building advanced search queries with Query DSL Getting ready How to do it... How it works... There’s more... See also Using search templates to pre-render search requests Getting ready How to do it... How it works... There’s more... See also Getting started with Search Applications for your Elasticsearch index Getting ready How to do it... How it works... Building a search experience with the Search Application client Getting ready How to do it... How it works... There’s more... See also Measuring the performance of your Search Applications with Behavioral Analytics Getting ready How to do it... How it works... There’s more... See also 4 Timestamped Data Ingestion Technical requirements Deploying Elastic Agent with Fleet Getting ready How to do it... How it works... There’s more... See also Monitoring Apache HTTP logs and metrics using the Apache integration Getting ready How to do it... How it works... There’s more... See also Deploying standalone Elastic Agent Getting ready How to do it... How it works... There’s more... See also Adding data using Beats Getting ready How to do it... How it works... There’s more... See also Setting up a data stream manually Dataset Getting ready How to do it... How it works... There’s more... See also Setting up a time series data stream manually Getting ready How to do it... How it works... There’s more... See also 5 Transform Data Technical requirements Creating an ingest pipeline Getting ready How to do it... How it works... There’s more... See also Enriching data with a custom ingest pipeline for an existing Elastic Agent integration Getting ready How to do it... How it works... There’s more... Using a processor to enrich your data before ingesting with Elastic Agent Getting ready How to do it... How it works... There’s more... See also Installing self-managed Logstash Getting ready How to do it... How it works... There’s more... See also Creating a Logstash pipeline Getting ready How to do it... How it works... There’s more... See also Setting up pivot data transform Getting ready How to do it... How it works... There’s more... See also Setting up the latest data transform Getting ready How to do it... How it works... There’s more... See also Downsampling your time series data Getting ready How to do it... How it works... There’s more... See also 6 Visualize and Explore Data Technical requirements Exploring your data in Discover Getting ready How to do it... How it works... There’s more... See also Exploring your data with ES|QL Getting ready How to do it... How it works... There’s more... See also Creating visualizations with Kibana Lens Getting ready How to do it... How it works... There’s more... See also Creating visualizations from runtime fields Getting ready How to do it... How it works... There’s more... See also Creating Kibana maps Getting ready How to do it... How it works... There’s more... See also Creating and using Kibana dashboards Getting ready How to do it... How it works... There’s more... See also Creating Canvas workpads Getting ready How to do it... How it works... There’s more... See also 7 Alerting and Anomaly Detection Technical requirements Creating alerts in Kibana Getting ready How to do it... How it works... There’s more... See also Monitoring alert rules Getting ready How to do it... How it works... There’s more... See also Investigating data with log rate analysis Getting ready How to do it... How it works... There’s more... See also Investigating data with log pattern analysis Getting ready How to do it... How it works... There’s more... Investigating data with change point detection Getting ready How to do it... How it works... There’s more... See also Detecting anomalies in your data with unsupervised machine learning jobs Getting ready How to do it... How it works... There’s more... See also Creating anomaly detection jobs from a Lens visualization Getting ready How to do it... How it works... There’s more... 8 Advanced Data Analysis and Processing Technical requirements Finding deviations in your data with outlier detection Getting ready How to do it... How it works... See also Building a model to perform regression analysis Getting ready How to do it... How it works... There’s more... See also Building a model for classification Getting ready How to do it... How it works... There’s more... See also Using a trained model for inference Getting ready How to do it... How it works... There’s more... See also Deploying third-party NLP models and testing via the UI Getting ready How to do it... How it works... There’s more... See also Running advanced data processing with trained models Getting ready How to do it... How it works... There’s more... See also 9 Vector Search and Generative AI Integration Technical requirements Implementing semantic search with dense vectors Getting ready How to do it... How it works... There’s more... See also Implementing semantic search with sparse vectors Getting ready How to do it... How it works... There’s more... See also Using hybrid search to build advanced search applications Getting ready How to do it... How it works... There’s more... See also Developing question-answering applications with Generative AI Getting ready How to do it... How it works... There’s more... See also Using advanced techniques for RAG applications Getting ready How to do it... How it works... There’s more... See also 10 Elastic Observability Solution Technical requirements Instrumenting your application with Elastic APM Getting ready How to do it... How it works... There’s more... See also Setting up RUM Getting ready How to do it... How it works... There’s more... See also Instrumenting and monitoring with OpenTelemetry Getting ready How to do it... How it works... There’s more... See also Monitoring Kubernetes environments with Elastic Agent Getting ready How to do it... How it works... There’s more... See also Managing synthetics monitoring Getting ready How to do it... How it works... There’s more... See also Gaining comprehensive system visibility with Elastic Universal Profiling Getting ready How to do it... How it works... There’s more... See also Detecting incidents with alerting and machine learning Getting ready How to do it... How it works... There’s more... See also Gaining insights with the AI Assistant Getting ready How to do it... How it works... There’s more... See also 11 Managing Access Control Technical requirements Using built-in roles Getting ready How to do it... How it works... See also Defining custom roles Getting ready How to do it... How it works... There’s more... See also Granting additional privileges Getting ready How to do it... How it works... There’s more... See also Managing and securing access to Kibana spaces Getting ready How to do it... How it works... There’s more... See also Managing access with API keys Getting ready How to do it... How it works... There’s more... See also Configuring single sign-on Getting ready How to do it... How it works... There’s more... See also Mapping users and groups to roles Getting ready How to do it... How it works... There’s more... 12 Elastic Stack Operation Technical requirements Setting up an index lifecycle policy Getting ready How to do it... How it works... There’s more... See also Optimizing time series data streams with downsampling Getting ready How to do it... How it works... There’s more... See also Managing the snapshot lifecycle Getting ready How to do it... How it works... There’s more... Configuring Elastic Stack components with Terraform Getting ready How to do it... How it works... There’s more... See also Enabling and configuring cross-cluster search Getting ready How to do it... How it works... There’s more... See also 13 Elastic Stack Monitoring Technical requirements Setting up Stack Monitoring Getting ready How to do it... How it works... There’s more... See also Building custom visualizations for monitoring data Getting ready How to do it... How it works... There’s more... Monitoring cluster health via an API Getting ready How to do it... How it works... There’s more... See also Enabling audit logging Getting ready How to do it... How it works... There’s more... See also Index Why subscribe? Other Books You May Enjoy Packt is searching for authors like you Share Your Thoughts Download a free PDF copy of this book Optimize your search capabilities in Elastic by operationalizing and fine-tuning vector search and enhance your search relevance while improving overall search performance Key FeaturesInstall, configure, and optimize the ChatGPT-Elasticsearch plugin with a focus on vector dataLearn how to load transformer models, generate vectors, and implement vector search with ElasticDevelop a practical understanding of vector search, including a review of current vector databasesPurchase of the print or Kindle book includes a free PDF eBookBook DescriptionWhile natural language processing (NLP) is largely used in search use cases, this book aims to inspire you to start using vectors to overcome equally important domain challenges like observability and cybersecurity. The chapters focus mainly on integrating vector search with Elastic to enhance not only their search but also observability and cybersecurity capabilities. The book begins by teaching you about NLP and the functionality of Elastic in NLP processes. Next, youll delve into resource requirements and find out how vectors are stored in the dense-vector type along with specific page cache requirements for fast response times. As you advance, youll discover various tuning techniques and strategies to improve machine learning model deployment, including node scaling, configuration tuning, and load testing with Rally and Python. Youll also cover techniques for vector search with images, fine-tuning models for improved performance, and the use of clip models for image similarity search in Elasticsearch. Finally, youll explore retrieval-augmented generation (RAG) and learn to integrate ChatGPT with Elasticsearch to leverage vectorized data, ELSER's capabilities, and RRF's refined search mechanism. By the end of this NLP book, youll have all the necessary skills needed to implement and optimize vector search in your projects with Elastic. What you will learnOptimize performance by harnessing the capabilities of vector searchExplore image vector search and its applicationsDetect and mask personally identifiable informationImplement log prediction for next-generation observabilityUse vector-based bot detection for cybersecurityVisualize the vector space and explore Search.Next with ElasticImplement a RAG-enhanced application using StreamlitWho this book is forIf you're a data professional with experience in Elastic observability, search, or cybersecurity and are looking to expand your knowledge of vector search, this book is for you. This book provides practical knowledge useful for search application owners, product managers, observability platform owners, and security operations center professionals. Experience in Python, using machine learning models, and data management will help you get the most out of this book. Table of ContentsIntroduction to Vectors and EmbeddingsGetting started with Vector Search in ElasticModel Management and Vector Considerations in ElasticHow to talk dataImage SearchPersonal Identifiable Information detection and maskingNext generation of observability with Log PredictionVector based Bot detection for cybersecurity practitionerVisualise Vector spaceSearch.Next with Elastic 'This book delves into the practical applications of vector search in Elastic and embodies a broader philosophy. It underscores the importance of search in the age of Generative Al and Large Language Models. This narrative goes beyond the'how'to address the'why'- highlighting our belief in the transformative power of search and our dedication to pushing boundaries to meet and exceed customer expectations.'Shay Banon Founder & CTO at ElasticKey FeaturesInstall, configure, and optimize the ChatGPT-Elasticsearch plugin with a focus on vector dataLearn how to load transformer models, generate vectors, and implement vector search with ElasticDevelop a practical understanding of vector search, including a review of current vector databasesPurchase of the print or Kindle book includes a free PDF eBookBook DescriptionWhile natural language processing (NLP) is largely used in search use cases, this book aims to inspire you to start using vectors to overcome equally important domain challenges like observability and cybersecurity. The chapters focus mainly on integrating vector search with Elastic to enhance not only their search but also observability and cybersecurity capabilities. The book, which also features a foreword written by the founder of Elastic, begins by teaching you about NLP and the functionality of Elastic in NLP processes. Here you'll delve into resource requirements and find out how vectors are stored in the dense-vector type along with specific page cache requirements for fast response times. As you advance, you'll discover various tuning techniques and strategies to improve machine learning model deployment, including node scaling, configuration tuning, and load testing with Rally and Python. You'll also cover techniques for vector search with images, fine-tuning models for improved performance, and the use of clip models for image similarity search in Elasticsearch. Finally, you'll explore retrieval-augmented generation (RAG) and learn to integrate ChatGPT with Elasticsearch to leverage vectorized data, ELSER's capabilities, and RRF's refined search mechanism. By the end of this NLP book, you'll have all the necessary skills needed to implement and optimize vector search in your projects with Elastic.What you will learnOptimize performance by harnessing the capabilities of vector searchExplore image vector search and its applicationsDetect and mask personally identifiable informationImplement log prediction for next-generation observabilityUse vector-based bot detection for cybersecurityVisualize the vector space and explore Search.Next with ElasticImplement a RAG-enhanced application using StreamlitWho this book is forIf you're a data professional with experience in Elastic observability, search, or cybersecurity and are looking to expand your knowledge of vector search, this book is for you. This book provides practical knowledge useful for search application owners, product managers, observability platform owners, and security operations center professionals. Experience in Python, using machine learning models, and data management will help you get the most out of this book. Analyze and transform data efficiently with DuckDB, a versatile, modern, in-process SQL databaseKey FeaturesUse DuckDB to rapidly load, transform, and query data across a range of sources and formatsGain practical experience using SQL, Python, and R to effectively analyze dataLearn how open source tools and cloud services in the broader data ecosystem complement DuckDB's versatile capabilitiesPurchase of the print or Kindle book includes a free PDF eBookBook DescriptionDuckDB is a fast in-process analytical database. Its ease of use, versatile feature set, and powerful analytical capabilities make DuckDB a valuable addition to the data practitioner's toolkit. Getting Started with DuckDB offers a practical overview of DuckDB's fundamentals and guidance for effectively using its powerful capabilities. Through extensive hands-on examples, you'll learn how to use DuckDB to load, transform, and query a variety of data sources and formats, including CSV, JSON, and Parquet files, semi-structured data, remotely-hosted files, and external databases. You'll also find out how to leverage DuckDB's performance optimizations and friendly SQL enhancements. You'll explore how to use DuckDB's extensions for specialized applications, such as geospatial analysis and text search over document collections. In addition to working through examples in SQL, Python, and R, you'll also dive into using DuckDB for analyzing public datasets and discover the wider ecosystem of open-source tools and cloud services that supercharge DuckDB-powered workflows and applications. Whether you're a seasoned data practitioner or new to working with analytical data, this book will rapidly get you up to speed with DuckDB's versatile and powerful capabilities, enabling you to apply them in your analytical workflows and projects.What you will learnUnderstand the properties and applications of a columnar in-process databaseUse SQL to load, transform, and query a range of data formatsDiscover DuckDB's rich extensions and learn how to apply themUse nested data types to model semi-structured data and extract and model JSON dataIntegrate DuckDB into your Python and R analytical workflowsEffectively leverage DuckDB's convenient SQL enhancementsExplore the wider ecosystem and pathways for building DuckDB-powered data applicationsWho this book is forIf you're interested in expanding your analytical toolkit, this book is for you. It will be particularly valuable for data analysts wanting to rapidly explore and query complex data, data and software engineers looking for a lean and versatile data processing tool, along with data scientists needing a scalable data manipulation library that integrates seamlessly with Python and R. You will get the most from this book if you have some familiarity with SQL and foundational database concepts, as well as exposure to a programming language such as Python or R. Unlock the potential of Elastic Stack for search, analytics, security, and observability use cases, and effectively manage substantial data workloads in both on-premise and cloud environments The Elastic Stack (also known as the ELK Stack) is a collection of open-source products - Elasticsearch, Kibana, Elastic Agent, and Logstash that can reliably and securely take data from any source, in any format, and then search, analyze, and visualize it in real-time. This book will give you the knowledge and skills to unlock the full potential of Elastic Stack. In this Cookbook, you'll explore practical recipes, starting with installing and ingesting data using Elastic Agent and Beats, and diving into data transformation and enrichment with various Elastic components. You'll explore the latest advancements in search applications, including semantic search and Generative AI. You'll learn to visualize and explore your data, and create dashboards using Kibana. As you move ahead, you'll advance your skills with machine learning for data science, natural language processing, and vector search. You'll discover Elastic Observability use cases for log, infrastructure, and synthetics monitoring, along with essential strategies for securing the Elastic Stack. Finally, you'll gain expertise in Elastic Stack operations to effectively monitor and manage your system. By the end of the book, you'll acquire the knowledge and skills to build scalable, reliable, and efficient data analytics and search solutions using Elastic Stack. This book is for Elastic Stack users, developers, observability practitioners, and data professionals from beginner to expert level who are looking for practical experience on the Elastic Stack. As a developer, youll find easy-to-follow recipes to use APIs and features to build powerful applications. This book will also help observability practitioners through use cases covering APM, Kubernetes, and cloud monitoring. For data engineers and AI enthusiasts, the book will provide dedicated recipes on vector search and machine learning. No prior knowledge of the Elastic Stack is required. Run complex queries on large datasets with amazing speed using a flexible and extensible SQL-embedded database. DuckDB is one of the hottest and fastest growing databases, driven by its powerful analytical capabilities, ease of use, versatility, and engaged community. It provides readers with an efficient, in-memory, column-oriented, and standards-compliant database to quickly process analytical query workloads through a standard SQL interface. This book teaches you how to install and deploy DuckDB on different platforms and environments. You'll learn to create tables, load and query data with SQL, and progress to cleaning, reshaping, and manipulating data. You'll discover advanced features and techniques to improve operations and performance in DuckDB. You'll explore how DuckDB can be used for complex and efficient data analysis. As you explore later chapters, you'll learn to perform descriptive statistics and exploratory data analysis, and integrate DuckDB with Python, R, and other data analysis libraries. You'll also explore creating, reading, and modifying JSON data in DuckDB, extending the database with SQL editors and third-party data viewers, and optimizing query performance. Lastly, you'll learn the best practices for using DuckDB effectively, including a roadmap of future enhancements. By the end of this book, you will have the skills to leverage DuckDB and unlock meaningful insights from data, making it more impactful. This book is for data analysts who want to explore complex data, data engineers who want a lean and efficient transformation tool, and data scientists who need the flexibility of a data manipulation library that integrates seamlessly with Python and R. The readers are required to understand foundational data concepts, such as querying database tables, and have exposure to a programming language such as Python or JavaScript. Theyll also need familiarity interacting with command line interfaces and will benefit from having exposure to traditional databases such as PostgreSQL or SQL Server