AI-Powered Search

Apply cutting-edge machine learning techniques—from crowdsourced relevance and knowledge graph learning, to Large Language Models (LLMs)—to enhance the accuracy and relevance of your search results. Delivering effective search is one of the biggest challenges you can face as an engineer. AI-Powered Search is an in-depth guide to building intelligent search systems you can be proud of. It covers the critical tools you need to automate ongoing relevance improvements within your search applications. Inside you’ll learn modern, data-science-driven search techniques like: • Semantic search using dense vector embeddings from foundation models • Retrieval augmented generation (RAG) • Question answering and summarization combining search and LLMs • Fine-tuning transformer-based LLMs • Personalized search based on user signals and vector embeddings • Collecting user behavioral signals and building signals boosting models • Semantic knowledge graphs for domain-specific learning • Semantic query parsing, query-sense disambiguation, and query intent classification • Implementing machine-learned ranking models (Learning to Rank) • Building click models to automate machine-learned ranking • Generative search, hybrid search, multimodal search, and the search frontier AI-Powered Search will help you build the kind of highly intelligent search applications demanded by modern users. Whether you’re enhancing your existing search engine or building from scratch, you’ll learn how to deliver an AI-powered service that can continuously learn from every content update, user interaction, and the hidden semantic relationships in your content. You’ll learn both how to enhance your AI systems with search and how to integrate large language models (LLMs) and other foundation models to massively accelerate the capabilities of your search technology. Foreword by Grant Ingersoll. About the technology Modern search is more than keyword matching. Much, much more. Search that learns from user interactions, interprets intent, and takes advantage of AI tools like large language models (LLMs) can deliver highly targeted and relevant results. This book shows you how to up your search game using state-of-the-art AI algorithms, techniques, and tools. About the book AI-Powered Search teaches you to create a search that understands natural language and improves automatically the more it is used. As you work through dozens of interesting and relevant examples, you’ll learn powerful AI-based techniques like semantic search on embeddings, question answering powered by LLMs, real-time personalization, and Retrieval Augmented Generation (RAG). What's inside • Sparse lexical and embedding-based semantic search • Question answering, RAG, and summarization using LLMs • Personalized search and signals boosting models • Learning to Rank, multimodal, and hybrid search About the reader For software developers and data scientists familiar with the basics of search engine technology. About the author Trey Grainger is the Founder of Searchkernel and former Chief Algorithms Officer and SVP of Engineering at Lucidworks. Doug Turnbull is a Principal Engineer at Reddit and former Staff Relevance Engineer at Spotify. Max Irwin is the Founder of Max.io and former Managing Consultant at OpenSource Connections. AI-Powered Search brief contents contents foreword preface acknowledgments about this book Who should read this book How this book is organized: A road map About the code liveBook discussion forum Other online resources about the authors about the cover illustration Part 1 Modern search relevance 1 Introducing AI-powered search 1.1 What is AI-powered search? 1.2 Understanding user intent 1.2.1 What is a search engine? 1.2.2 What do recommendation engines offer? 1.2.3 The personalization spectrum between search and recommendations 1.2.4 Semantic search and knowledge graphs 1.2.5 Understanding the dimensions of user intent 1.3 How does AI-powered search work? 1.3.1 The core search foundation 1.3.2 Reflected intelligence through feedback loops 1.3.3 Signals boosting, collaborative filtering, and learning to rank 1.3.4 Content and domain intelligence 1.3.5 Generative AI and retrieval augmented generation 1.3.6 Curated vs. black-box AI 1.3.7 Architecture for an AI-powered search engine Summary 2 Working with natural language 2.1 The myth of unstructured data 2.1.1 Types of unstructured data 2.1.2 Data types in traditional structured databases 2.1.3 Joins, fuzzy joins, and entity resolution in unstructured data 2.2 The structure of natural language 2.3 Distributional semantics and embeddings 2.4 Modeling domain-specific knowledge 2.5 Challenges in natural language understanding for search 2.5.1 The challenge of ambiguity (polysemy) 2.5.2 The challenge of understanding context 2.5.3 The challenge of personalization 2.5.4 Challenges interpreting queries vs. documents 2.5.5 Challenges interpreting query intent 2.6 Content + signals: The fuel powering AI-powered search Summary 3 Ranking and content-based relevance 3.1 Scoring query and document vectors with cosine similarity 3.1.1 Mapping text to vectors 3.1.2 Calculating similarity between dense vector representations 3.1.3 Calculating similarity between sparse vector representations 3.1.4 Term frequency: Measuring how well documents match a term 3.1.5 Inverse document frequency: Measuring the importance of a term in the query 3.1.6 TF-IDF: A balanced weighting metric for text-based relevance 3.2 Controlling the relevance calculation 3.2.1 BM25: The industry standard default text-similarity algorithm 3.2.2 Functions, functions, everywhere! 3.2.3 Choosing multiplicative vs. additive boosting for relevance functions 3.2.4 Differentiating matching (filtering) vs. ranking (scoring) of documents 3.2.5 Logical matching: Weighting the relationships between terms in a query 3.2.6 Separating concerns: Filtering vs. scoring 3.3 Implementing user and domain-specific relevance ranking Summary 4 Crowdsourced relevance 4.1 Working with user signals 4.1.1 Content vs. signals vs. models 4.1.2 Setting up our product and signals datasets (RetroTech) 4.1.3 Exploring the signals data 4.1.4 Modeling users, sessions, and requests 4.2 Introducing reflected intelligence 4.2.1 What is reflected intelligence? 4.2.2 Popularized relevance through signals boosting 4.2.3 Personalized relevance through collaborative filtering 4.2.4 Generalized relevance through learning to rank 4.2.5 Other reflected intelligence models 4.2.6 Crowdsourcing from content Summary Part 2 Learning domain-specific intent 5 Knowledge graph learning 5.1 Working with knowledge graphs 5.2 Using our search engine as a knowledge graph 5.3 Automatically extracting knowledge graphs from content 5.3.1 Extracting arbitrary relationships from text 5.3.2 Extracting hyponyms and hypernyms from text 5.4 Learning intent by traversing semantic knowledge graphs 5.4.1 What is a semantic knowledge graph? 5.4.2 Indexing the datasets 5.4.3 Structure of an SKG 5.4.4 Calculating edge weights to measure the relatedness of nodes 5.4.5 Using SKGs for query expansion 5.4.6 Using SKGs for content-based recommendations 5.4.7 Using SKGs to model arbitrary relationships 5.5 Using knowledge graphs for semantic search Summary 6 Using context to learn domain-specific language 6.1 Classifying query intent 6.2 Query-sense disambiguation 6.3 Learning related phrases from query signals 6.3.1 Mining query logs for related queries 6.3.2 Finding related queries through product interactions 6.4 Phrase detection from user signals 6.4.1 Treating queries as entities 6.4.2 Extracting entities from more complex queries 6.5 Misspellings and alternative representations 6.5.1 Learning spelling corrections from documents 6.5.2 Learning spelling corrections from user signals 6.6 Pulling it all together Summary 7 Interpreting query intent through semantic search 7.1 The mechanics of query interpretation 7.2 Indexing and searching on a local reviews dataset 7.3 An end-to-end semantic search example 7.4 Query interpretation pipelines 7.4.1 Parsing a query for semantic search 7.4.2 Enriching a query for semantic search 7.4.3 Sparse lexical and expansion models 7.4.4 Transforming a query for semantic search 7.4.5 Searching with a semantically enhanced query Summary Part 3 Reflected intelligence 8 Signals-boosting models 8.1 Basic signals boosting 8.2 Normalizing signals 8.3 Fighting signal spam 8.3.1 Using signal spam to manipulate search results 8.3.2 Combating signal spam through user-based filtering 8.4 Combining multiple signal types 8.5 Time decays and short-lived signals 8.5.1 Handling time-insensitive signals 8.5.2 Handling time-sensitive signals 8.6 Index-time vs. query-time boosting: Balancing scale vs. flexibility 8.6.1 Tradeoffs when using query-time boosting 8.6.2 Implementing index-time signals boosting 8.6.3 Tradeoffs when implementing index-time boosting Summary 9 Personalized search 9.1 Personalized search vs. recommendations 9.1.1 Personalized queries 9.1.2 User-guided recommendations 9.2 Recommendation algorithm approaches 9.2.1 Content-based recommenders 9.2.2 Behavior-based recommenders 9.2.3 Multimodal recommenders 9.3 Implementing collaborative filtering 9.3.1 Learning latent user and item features through matrix factorization 9.3.2 Implementing collaborative filtering with Alternating Least Squares 9.3.3 Personalizing search results with recommendation boosting 9.4 Personalizing search using content-based embeddings 9.4.1 Generating content-based latent features 9.4.2 Implementing categorical guardrails for personalization 9.4.3 Integrating embedding-based personalization into search results 9.5 Challenges with personalizing search results Summary 10 Learning to rank for generalizable search relevance 10.1 What is LTR? 10.1.1 Moving beyond manual relevance tuning 10.1.2 Implementing LTR in the real world 10.2 Step 1: A judgment list, starting with the training data 10.3 Step 2: Feature logging and engineering 10.3.1 Storing features in a modern search engine 10.3.2 Logging features from our search engine corpus 10.4 Step 3: Transforming LTR to a traditional machine learning problem 10.4.1 SVMrank: Transforming ranking to binary classification 10.4.2 Transforming our LTR training task to binary classification 10.5 Step 4: Training (and testing!) the model 10.5.1 Turning a separating hyperplane’s vector into a scoring function 10.5.2 Taking the model for a test drive 10.5.3 Validating the model 10.6 Steps 5 and 6: Upload a model and search 10.6.1 Deploying and using the LTR model 10.6.2 A note on LTR performance 10.7 Rinse and repeat Summary 11 Automating learning to rank with click models 11.1 (Re)creating judgment lists from signals 11.1.1 Generating implicit, probabilistic judgments from signals 11.1.2 Training an LTR model using probabilistic judgments 11.1.3 Click-Through Rate: Your first click model 11.1.4 Common biases in judgments 11.2 Overcoming position bias 11.2.1 Defining position bias 11.2.2 Position bias in RetroTech data 11.2.3 Simplified dynamic Bayesian network: A click model that overcomes position bias 11.3 Handling confidence bias: Not upending your model due to a few lucky clicks 11.3.1 The low-confidence problem in click data 11.3.2 Using a beta prior to model confidence probabilistically 11.4 Exploring your training data in an LTR system Summary 12 Overcoming ranking bias through active learning 12.1 Our automated LTR engine in a few lines of code 12.1.1 Turning clicks into training data (chapter 11 in one line of code) 12.1.2 Model training and evaluation in a few function calls 12.2 A/B testing a new model 12.2.1 Taking a better model out for a test drive 12.2.2 Defining an A/B test in the context of automated LTR 12.2.3 Graduating the better model into an A/B test 12.2.4 When “good” models go bad: What we can learn from a failed A/B test 12.3 Overcoming presentation bias: Knowing when to explore vs. exploit 12.3.1 Presentation bias in the RetroTech training data 12.3.2 Beyond the ad hoc: Thoughtfully exploring with a Gaussian process 12.3.3 Examining the outcome of our explorations 12.4 Exploit, explore, gather, rinse, repeat: A robust automated LTR loop Summary Part 4 The search frontier 13 Semantic search with dense vectors 13.1 Representation of meaning through embeddings 13.2 Search using dense vectors 13.2.1 A brief refresher on sparse vectors 13.2.2 A conceptual dense vector search engine 13.3 Getting text embeddings by using a Transformer encoder 13.3.1 What is a Transformer? 13.3.2 Openly available pretrained Transformer models 13.4 Applying Transformers to search 13.4.1 Using the Stack Exchange outdoors dataset 13.4.2 Fine-tuning and the Semantic Text Similarity Benchmark 13.4.3 Introducing the SBERT Transformer library 13.5 Natural language autocomplete 13.5.1 Getting noun and verb phrases for our nearest-neighbor vocabulary 13.5.2 Getting embeddings 13.5.3 ANN search 13.5.4 ANN index implementation 13.6 Semantic search with LLM embeddings 13.6.1 Getting titles and their embeddings 13.6.2 Creating and searching the nearest-neighbor index 13.7 Quantization and representation learning for more efficient vector search 13.7.1 Scalar quantization 13.7.2 Binary quantization 13.7.3 Product quantization 13.7.4 Matryoshka Representation Learning 13.7.5 Combining multiple vector search optimization approaches 13.8 Cross-encoders vs. bi-encoders Summary 14 Question answering with a fine-tuned large language model 14.1 Question-answering overview 14.1.1 How a question-answering model works 14.1.2 The retriever-reader pattern 14.2 Constructing a question-answering training dataset 14.2.1 Gathering and cleaning a question-answering dataset 14.2.2 Creating the silver set: Automatically labeling data from a pretrained model 14.2.3 Human-in-the-loop training: Manually correcting the silver set to produce a golden set 14.2.4 Formatting the golden set for training, testing, and validation 14.3 Fine-tuning the question-answering model 14.3.1 Tokenizing and shaping our labeled data 14.3.2 Configuring the model trainer 14.3.3 Performing training and evaluating loss 14.3.4 Holdout validation and confirmation 14.4 Building the reader with the new fine-tuned model 14.5 Incorporating the retriever: Using the question-answering model with the search engine 14.5.1 Step 1: Querying the retriever 14.5.2 Step 2: Inferring answers from the reader model 14.5.3 Step 3: Reranking the answers 14.5.4 Step 4: Returning results by combining the retriever, reader, and reranker Summary 15 Foundation models and emerging search paradigms 15.1 Understanding foundation models 15.1.1 What qualifies as a foundation model? 15.1.2 Training vs. fine-tuning vs. prompting 15.2 Generative search 15.2.1 Retrieval augmented generation 15.2.2 Results summarization using foundation models 15.2.3 Data generation using foundation models 15.2.4 Evaluating generative output 15.2.5 Constructing your own metric 15.2.6 Algorithmic prompt optimization 15.3 Multimodal search 15.3.1 Common modes for multimodal search 15.3.2 Implementing multimodal search 15.4 Other emerging AI-powered search paradigms 15.4.1 Conversational and contextual search 15.4.2 Agent-based search 15.5 Hybrid search 15.5.1 Reciprocal rank fusion 15.5.2 Other hybrid search algorithms 15.6 Convergence of contextual technologies 15.7 All the above, please! Summary appendix A Running the code examples A.1 Overall structure of code examples A.2 Pulling the source code A.3 Building and running the code A.4 Working with Jupyter A.5 Working with Docker appendix B Supported search engines and vector databases B.1 Supported engines B.2 Swapping out the engine B.3 The engine and collection abstractions B.4 Adding support for additional engines index A B C D E F G H I J K L M N O P Q R S T U V W AI-Powered Search - back Great search is all about delivering the right results. Todays search engines are expected to be smart, understanding the nuances of natural language queries, as well as each users preferences and context. AI-Powered Search teaches you the latest machine learning techniques to create search engines that continuously learn from your users and your content, to drive more domain-aware and intelligent search. Written by Trey Grainger, the Chief Algorithms Officer at Lucidworks, this authoritative book empowers you to create and deploy search engines that take advantage of user interactions and the hidden semantic relationships in your content to constantly get smarter and automatically deliver better, more relevant search experiences.