Steamy Stories Volume 1

Learn the skills necessary to design, build, and deploy applications powered by machine learning (ML). Through the course of this hands-on book, you’ll build an example ML-driven application from initial idea to deployed product. Data scientists, software engineers, and product managers—including experienced practitioners and novices alike—will learn the tools, best practices, and challenges involved in building a real-world ML application step by step. Author Emmanuel Ameisen, an experienced data scientist who led an AI education program, demonstrates practical ML concepts using code snippets, illustrations, screenshots, and interviews with industry leaders. Part I teaches you how to plan an ML application and measure success. Part II explains how to build a working ML model. Part III demonstrates ways to improve the model until it fulfills your original vision. Part IV covers deployment and monitoring strategies. This book will help you: • Define your product goal and set up a machine learning problem • Build your first end-to-end pipeline quickly and acquire an initial dataset • Train and evaluate your ML models and address performance bottlenecks • Deploy and monitor your models in a production environment Copyright 4 Table of Contents 5 Preface 11 The Goal of Using Machine Learning Powered Applications 11 Use ML to Build Practical Applications 11 Additional Resources 12 Practical ML 13 What This Book Covers 13 Prerequisites 14 Our Case Study: ML–Assisted Writing 14 The ML Process 15 Conventions Used in This Book 16 Using Code Examples 17 O’Reilly Online Learning 17 How to Contact Us 18 Acknowledgments 18 Part I. Find the Correct ML Approach 21 Chapter 1. From Product Goal to ML Framing 23 Estimate What Is Possible 24 Models 25 Data 33 Framing the ML Editor 35 Trying to Do It All with ML: An End-to-End Framework 36 The Simplest Approach: Being the Algorithm 37 Middle Ground: Learning from Our Experience 38 Monica Rogati: How to Choose and Prioritize ML Projects 40 Conclusion 42 Chapter 2. Create a Plan 43 Measuring Success 43 Business Performance 44 Model Performance 45 Freshness and Distribution Shift 48 Speed 50 Estimate Scope and Challenges 51 Leverage Domain Expertise 51 Stand on the Shoulders of Giants 52 ML Editor Planning 56 Initial Plan for an Editor 56 Always Start with a Simple Model 56 To Make Regular Progress: Start Simple 57 Start with a Simple Pipeline 57 Pipeline for the ML Editor 59 Conclusion 60 Part II. Build a Working Pipeline 63 Chapter 3. Build Your First End-to-End Pipeline 65 The Simplest Scaffolding 65 Prototype of an ML Editor 67 Parse and Clean Data 67 Tokenizing Text 68 Generating Features 68 Test Your Workflow 70 User Experience 70 Modeling Results 71 ML Editor Prototype Evaluation 72 Model 73 User Experience 73 Conclusion 74 Chapter 4. Acquire an Initial Dataset 75 Iterate on Datasets 75 Do Data Science 76 Explore Your First Dataset 77 Be Efficient, Start Small 77 Insights Versus Products 78 A Data Quality Rubric 78 Label to Find Data Trends 84 Summary Statistics 85 Explore and Label Efficiently 87 Be the Algorithm 102 Data Trends 104 Let Data Inform Features and Models 105 Build Features Out of Patterns 105 ML Editor Features 108 Robert Munro: How Do You Find, Label, and Leverage Data? 109 Conclusion 110 Part III. Iterate on Models 113 Chapter 5. Train and Evaluate Your Model 115 The Simplest Appropriate Model 115 Simple Models 116 From Patterns to Models 118 Split Your Dataset 119 ML Editor Data Split 125 Judge Performance 126 Evaluate Your Model: Look Beyond Accuracy 129 Contrast Data and Predictions 129 Confusion Matrix 130 ROC Curve 131 Calibration Curve 134 Dimensionality Reduction for Errors 136 The Top-k Method 136 Other Models 141 Evaluate Feature Importance 141 Directly from a Classifier 142 Black-Box Explainers 143 Conclusion 145 Chapter 6. Debug Your ML Problems 147 Software Best Practices 147 ML-Specific Best Practices 148 Debug Wiring: Visualizing and Testing 150 Start with One Example 150 Test Your ML Code 156 Debug Training: Make Your Model Learn 160 Task Difficulty 162 Optimization Problems 164 Debug Generalization: Make Your Model Useful 166 Data Leakage 167 Overfitting 167 Consider the Task at Hand 170 Conclusion 171 Chapter 7. Using Classifiers for Writing Recommendations 173 Extracting Recommendations from Models 174 What Can We Achieve Without a Model? 174 Extracting Global Feature Importance 175 Using a Model’s Score 176 Extracting Local Feature Importance 177 Comparing Models 179 Version 1: The Report Card 180 Version 2: More Powerful, More Unclear 180 Version 3: Understandable Recommendations 182 Generating Editing Recommendations 183 Conclusion 187 Part IV. Deploy and Monitor 189 Chapter 8. Considerations When Deploying Models 191 Data Concerns 192 Data Ownership 192 Data Bias 193 Systemic Bias 194 Modeling Concerns 195 Feedback Loops 195 Inclusive Model Performance 197 Considering Context 197 Adversaries 198 Abuse Concerns and Dual-Use 199 Chris Harland: Shipping Experiments 200 Conclusion 202 Chapter 9. Choose Your Deployment Option 203 Server-Side Deployment 203 Streaming Application or API 204 Batch Predictions 206 Client-Side Deployment 208 On Device 209 Browser Side 211 Federated Learning: A Hybrid Approach 211 Conclusion 213 Chapter 10. Build Safeguards for Models 215 Engineer Around Failures 215 Input and Output Checks 216 Model Failure Fallbacks 220 Engineer for Performance 224 Scale to Multiple Users 224 Model and Data Life Cycle Management 227 Data Processing and DAGs 230 Ask for Feedback 231 Chris Moody: Empowering Data Scientists to Deploy Models 234 Conclusion 236 Chapter 11. Monitor and Update Models 237 Monitoring Saves Lives 237 Monitoring to Inform Refresh Rate 237 Monitor to Detect Abuse 238 Choose What to Monitor 239 Performance Metrics 239 Business Metrics 242 CI/CD for ML 243 A/B Testing and Experimentation 244 Other Approaches 247 Conclusion 248 Index 251 About the Author 259 Colophon 259 Learn the skills necessary to design, build, and deploy applications powered by machine learning. Through the course of this hands-on book, you'll build an example ML-driven application from initial idea to deployed product. Data scientists, software engineers, and product managers with little or no ML experience will learn the tools, best practices, and challenges involved in building a real-world ML application step-by-step. Author Emmanuel Ameisen, who worked as a data scientist at Zipcar and led Insight Data Science's AI program, demonstrates key ML concepts with code snippets, illustrations, and screenshots from the book's example application. The first part of this guide shows you how to plan and measure success for an ML application. Part II shows you how to build a working ML model, and Part III explains how to improve the model until it fulfills your original vision. Part IV covers deployment and monitoring strategies. This book will help you: Determine your product goal and set up a machine learning problem Build your first end-to-end pipeline quickly and acquire an initial dataset Train and evaluate your ML model and address performance bottlenecks Deploy and monitor models in a production environment