50 Short Science Fiction Tales

Entity resolution is a key analytic technique that enables you to identify multiple data records that refer to the same real-world entity. With this hands-on guide, product managers, data analysts, and data scientists will learn how to add value to data by cleansing, analyzing, and resolving datasets using open source Python libraries and cloud APIs. Author Michael Shearer shows you how to scale up your data matching processes and improve the accuracy of your reconciliations. You'll be able to remove duplicate entries within a single source and join disparate data sources together when common keys aren't available. Using real-world data examples, this book helps you gain practical understanding to accelerate the delivery of real business value. With entity resolution, you'll build rich and comprehensive data assets that reveal relationships for marketing and risk management purposes, key to harnessing the full potential of ML and AI. This book covers: Challenges in deduplicating and joining datasets Extracting, cleansing, and preparing datasets for matching Text matching algorithms to identify equivalent entities Techniques for deduplicating and joining datasets at scale Matching datasets containing persons and organizations Evaluating data matches Optimizing and tuning data matching algorithms Entity resolution using cloud APIs Matching using privacy-enhancing technologies Copyright Table of Contents Preface Who Should Read This Book Why I Wrote This Book Navigating This Book Conventions Used in This Book Using Code Examples O’Reilly Online Learning How to Contact Us Acknowledgments Chapter 1. Introduction to Entity Resolution What Is Entity Resolution? Why Is Entity Resolution Needed? Main Challenges of Entity Resolution Lack of Unique Names Inconsistent Naming Conventions Data Capture Inconsistencies Worked Example Deliberate Obfuscation Match Permutations Blind Matching? The Entity Resolution Process Data Standardization Record Blocking Attribute Comparison Match Classification Clustering Canonicalization Worked Example Measuring Performance Getting Started Chapter 2. Data Standardization Sample Problem Environment Setup Acquiring Data Wikipedia Data TheyWorkForYou Data Cleansing Data Wikipedia TheyWorkForYou Attribute Comparison Constituency Measuring Performance Sample Calculation Summary Chapter 3. Text Matching Edit Distance Matching Levenshtein Distance Jaro Similarity Jaro-Winkler Similarity Phonetic Matching Metaphone Match Rating Approach Comparing the Techniques Sample Problem Full Similarity Comparison Measuring Performance Summary Chapter 4. Probabilistic Matching Sample Problem Single Attribute Match Probability First Name Match Probability Last Name Match Probability Multiple Attribute Match Probability Probabilistic Models Bayes’ Theorem m Value u Value Lambda ( λ) Value Bayes Factor Fellegi-Sunter Model Match Weight Expectation-Maximization Algorithm Iteration 1 Iteration 2 Iteration 3 Introducing Splink Configuring Splink Splink Performance Summary Chapter 5. Record Blocking Sample Problem Data Acquisition Wikipedia Data UK Companies House Data Data Standardization Wikipedia Data UK Companies House Data Record Blocking and Attribute Comparison Record Blocking with Splink Attribute Comparison Match Classification Measuring Performance Summary Chapter 6. Company Matching Sample Problem Data Acquisition Data Standardization Companies House Data Maritime and Coastguard Agency Data Record Blocking and Attribute Comparison Match Classification Measuring Performance Matching New Entities Summary Chapter 7. Clustering Simple Exact Match Clustering Approximate Match Clustering Sample Problem Data Acquisition Data Standardization Record Blocking and Attribute Comparison Data Analysis Expectation-Maximization Blocking Rules Match Classification and Clustering Cluster Visualization Cluster Analysis Summary Chapter 8. Scaling Up on Google Cloud Google Cloud Setup Setting Up Project Storage Creating a Dataproc Cluster Configuring a Dataproc Cluster Entity Resolution on Spark Measuring Performance Tidy Up! Summary Chapter 9. Cloud Entity Resolution Services Introduction to BigQuery Enterprise Knowledge Graph API Schema Mapping Reconciliation Job Result Processing Entity Reconciliation Python Client Measuring Performance Summary Chapter 10. Privacy-Preserving Record Linkage An Introduction to Private Set Intersection How PSI Works PSI Protocol Based on ECDH Bloom Filters Golomb-Coded Sets Example: Using the PSI Process Environment Setup Server Code Client Code Full MCA and Companies House Sample Example Summary Chapter 11. Further Considerations Data Considerations Unstructured Data Data Quality Temporal Equivalence Attribute Comparison Set Matching Geocoding Location Matching Aggregating Comparisons Post Processing Graphical Representation Real-Time Considerations Performance Evaluation Pairwise Approach Cluster-Based Approach Future of Entity Resolution Index About the Author Colophon