Transactional Machine Learning with Data Streams and AutoML : Build Frictionless and Elastic Machine Learning Solutions with Apache Kafka in the Cloud Using Python

Understand how to apply auto machine learning to data streams and create transactional machine learning (TML) solutions that are frictionless (require minimal to no human intervention) and elastic (machine learning solutions that can scale up or down by controlling the number of data streams, algorithms, and users of the insights). This book will strengthen your knowledge of the inner workings of TML solutions using data streams with auto machine learning integrated with Apache Kafka. Transactional Machine Learning with Data Streams and AutoML introduces the industry challenges with applying machine learning to data streams. You will learn the framework that will help you in choosing business problems that are best suited for TML. You will also see how to measure the business value of TML solutions. You will then learn the technical components of TML solutions, including the reference and technical architecture of a TML solution. This book also presents a TML solution template that will make it easy for you to quickly start building your own TML solutions. Specifically, you are given access to a TML Python library and integration technologies for download. You will also learn how TML will evolve in the future, and the growing need by organizations for deeper insights from data streams. By the end of the book, you will have a solid understanding of TML. You will know how to build TML solutions with all the necessary details, and all the resources at your fingertips. What You Will Learn Discover transactional machine learning Measure the business value of TML Choose TML use cases Design technical architecture of TML solutions with Apache Kafka Work with the technologies used to build TML solutions Build transactional machine learning solutions with hands-on code together with Apache Kafka in the cloud Who This Book Is For Data scientists, machine learning engineers and architects, and AI and machine learning business leaders. Table of Contents 5 About the Author 9 About the Technical Reviewer 10 Acknowledgments 11 Introduction 12 Chapter 1: Introduction: Big Data, Auto Machine Learning, and Data Streams 13 Structured Data 15 Semi-structured Data 16 Unstructured Data 16 A Quick Take on Big Data 17 Data Quality 19 Data Streams 24 Stream Mining 25 Auto Machine Learning (AutoML) 29 Machine Learning Model Building Process 32 Concluding Remarks 33 Chapter 2: Transactional Machine Learning 37 Examining TML 37 Features of TML 41 Data Fluidity 42 Joining Data Streams 47 Data Stream Standardization 48 Data Stream Integration with AutoML 49 Low Code 50 Data Stream Storage Platform (DSSP) 54 MAADS-VIPER 55 Algorithm and Insights Management System (AiMS) Dashboard 58 AutoML Technology 63 Unsupervised Learning: Detecting Anomalies 66 Frictionless Machine Learning 69 Concluding Remarks 70 Chapter 3: Overcoming Challenges to ML Adoption 72 Overview of Challenges 73 Understanding the Root Causes of Challenges in Adopting Advanced Technologies 73 Data Decentralization 74 Lack of Corporate Strategy 76 Advanced Technology Costs 78 Choosing ML Use Cases 79 ML Change Acceptance 79 Technological Barriers 80 Skill Gap to Adopting ML 80 Strategy Gap in Adopting ML 80 Communication Gap in Adopting ML 81 Approaches to Addressing the Challenges 81 Discussion and Path Forward 85 Chapter 4: The Business Value of Transactional Machine Learning 88 Conventional Machine Learning (CML) 90 The TML Opportunity 94 Core Areas of Value from TML 100 TML Value Areas (Levers) 101 Measuring Value from TML Solutions 105 Choosing the Right TML Use Cases 109 Benefits and Costs 112 Risks and Pitfalls 114 Concluding Remarks 118 Chapter 5: The Technical Components and Architecture for Transactional Machine Learning Solutions 120 Overview of a TML Solution 120 Reference Architecture of a TML Solution 122 Description of Technical Components 126 Technical Architecture of a TML Solution 132 Unsupervised Learning 136 Communication Process Between Components 137 Data Flows 139 Example Architecture 142 TML Cost Management 144 Concluding Remarks 145 Chapter 6: Transactional Machine Learning Solution Template with Streaming Visualization 148 Overview of TML Solution Template 149 Template Component Details 150 Kafka Cloud via Confluent Cloud 151 VIPER Environment File 157 VIPER, VIPERviz, and HPDE Setup 161 Kafka Topics and Data Streams 162 TML Example Code 184 Walmart Foot Traffic Prediction and Optimization with TML 186 Unsupervised Learning for Anomaly Detection 198 Anomaly Detection on Banking Transactions with TML 208 Concluding Remarks 212 Chapter 7: Visualize Your TML Model Insights: Optimization, Predictions, and Anomalies 218 Streaming Anomaly Detection Visualization 223 Streaming Prediction Visualization 225 Streaming Optimization Visualization 227 AiMS Dashboard 228 Generic Topics’ Visualization 230 Visualization with WebSockets 231 Concluding Remarks 232 Chapter 8: Evolution and Opportunities for Transactional Machine Learning in Almost Every Industry 234 Areas of Further Exploration 235 Faster and More Complex Decision-Making by Machines 235 Broader Adoption of AutoML Techniques and Processes to Data Streams 237 Stacking and Chaining Different TML Solutions 249 Concluding Remarks 251 Chapter 9: TML Project Planning Approach and Closing Thoughts 253 TML Technology Stack 255 TML Project Planning Approach 259 TML Value Creation 264 Closing Thoughts 266 Definitions 268 References 271 Index 277