Data fabric and data mesh approaches with AI : a guide to AI-based data cataloging, governance, integration, orchestration, and consumption

Understand modern data fabric and data mesh concepts using AI-based self-service data discovery and delivery capabilities, a range of intelligent data integration styles, and automated unified data governance—all designed to deliver "data as a product" within hybrid cloud landscapes. This book teaches you how to successfully deploy state-of-the-art data mesh solutions and gain a comprehensive overview on how a data fabric architecture uses artificial intelligence (AI) and machine learning (ML) for automated metadata management and self-service data discovery and consumption. You will learn how data fabric and data mesh relate to other concepts such as data DataOps, MLOps, AIDevOps, and more. Many examples are included to demonstrate how to modernize the consumption of data to enable a shopping-for-data (data as a product) experience. By the end of this book, you will understand the data fabric concept and architecture as it relates to themes such as automated unified data governance and compliance, enterprise information architecture, AI and hybrid cloud landscapes, and intelligent cataloging and metadata management. What You Will Learn Discover best practices and methods to successfully implement a data fabric architecture and data mesh solution Understand key data fabric capabilities, e.g., self-service data discovery, intelligent data integration techniques, intelligent cataloging and metadata management, and trustworthy AI Recognize the importance of data fabric to accelerate digital transformation and democratize data access Dive into important data fabric topics, addressing current data fabric challenges Conceive data fabric and data mesh concepts holistically within an enterprise context Become acquainted with the business benefits of data fabric and data mesh Who This Book Is For Anyone who is interested in deploying modern data fabric architectures and data mesh solutions within an enterprise, including IT and business leaders, data governance and data office professionals, data stewards and engineers, data scientists, and information and data architects. Readers should have a basic understanding of enterprise information architecture. Table of Contents About the Authors About the Technical Reviewer Acknowledgments Introduction Foreword Chapter 1: Evolution of Data Architecture Introduction Data Architectures: Values and Challenges Enterprise Data Warehouse (EDW) Big Data, Data Lake, and Data Lakehouse Key Takeaways References Chapter 2: Terminology: Data Fabric and Data Mesh Introduction Data Fabric Concept Data Fabric Framework AI-Infused Data Fabric Data Mesh Concept Relationship: Data Fabric and Data Mesh Data Product Key Takeaways References Chapter 3: Data Fabric and Data Mesh Use Case Scenarios Introduction Automated and Consistent Governance Include IBM zSystems Data in AI Governance Unified View of Data Across a Hybrid Cloud Provide a Comprehensive View of Customers, Vendors, and Other Parties Unlock the Trustworthy AI Concept Key Takeaways References Chapter 4: Data Fabric and Data Mesh Business Benefits Introduction Business Requirements and Pain Points for Data Management and Consumption Benefits of a Data Fabric and Data Mesh for Technical Teams Managing Data Benefits of a Data Fabric and Data Mesh for Business Teams Consuming Data Key Takeaways References Chapter 5: Key Data Fabric and Data Mesh Capabilities Introduction Knowledge Catalog Active Metadata Data Curation Semantic Knowledge Graphs Self-Service Capabilities Trustworthy AI Introduction Model Fairness Drift Detection Model Explainability Model Quality Metrics Intelligent Information Integration Key Takeaways References Chapter 6: Relevant ML and DL Concepts Introduction to AI, ML, and DL ML and DL Industry Use Cases Data Exploration and Preparation Model Selection, Training, and Evaluation Model Deployment Natural Language Processing (NLP) Key Takeaways References Chapter 7: AI and ML for a Data Fabric and Data Mesh Introduction General Overview Cataloging AI-Infused Understanding of Assets Asset Discovery Asset Profiling Automatic Asset Quality Assessment Asset Access AI/ML for Entity Matching AI/ML to Activate the Digital Exhaust AI/ML for Semantic Enrichment Key Takeaways References Chapter 8: AI for Entity Resolution Introduction Introducing Entity Matching Traditional Entity Resolution Approaches Use of AI to Resolve Entity Challenges The Benefits and Cost of an AI-Based Solution Considerations for MDM Solutions Key Takeaways References Chapter 9: Data Fabric and Data Mesh for the AI Lifecycle Introduction to the AI Lifecycle Key Aspects: DataOps, ModelOps, MLOps Case Study 1: Consolidating Fragmented Data in a Hybrid Cloud Environment Case Study 2: Operationalizing AI Accelerate MLOps with AutoAI Deployment Patterns for AI Engineering Key Takeaways References Chapter 10: Data Fabric Architecture Patterns Introduction Data Fabric and Data Mesh Evolution Data Consumption Patterns Data Fabric for a Data Mesh Solution Data Mesh Self-Service Capabilities Data Mesh Architecture Overview Diagram Intelligent Information Integration Styles Key Takeaways References Chapter 11: Data Fabric Within an Enterprise Architecture Introduction What Is Enterprise Architecture? What Is Application Architecture? Data Fabric as a Data Architecture Sample of a Data Fabric Within an Enterprise Architecture Key Takeaways References Chapter 12: Data Fabric and Data Mesh in a Hybrid Cloud Landscape Introduction What Is Hybrid Cloud? Key Challenges for Data Architecture Data Fabric and Data Mesh in Hybrid Cloud Data Fabric Architecture in Hybrid Cloud Data Mesh Solution in Hybrid Cloud Benefits of Data Fabric and Data Mesh for Hybrid Cloud Key Takeaways References Chapter 13: Intelligent Cataloging and Metadata Management Introduction to Metadata Management Key Aspects of Intelligent Cataloging Build an Intelligent Catalog by Automating Data Discovery and Enrichment Find Data Assets with Semantic Search and Recommendation Provide Data Insight and Provenance as Data Flows Across the Enterprise Key Takeaways References Chapter 14: Automated Data Fabric and Data Mesh Aspects Introduction Intelligent Automation of Metadata Automated Analysis and Profiling of Data Automated Tagging, Annotation, and Labeling Automated Data Quality Assessment Key Takeaways References Chapter 15: Data Governance in the Context of Data Fabric and Data Mesh Introduction Importance of Data and AI Governance Key Aspects of Data and AI Governance Establishing a Data Governance Foundation with a Data Fabric Architecture Establishing Automated Regulation with a Data Fabric Architecture Automatic Enforcement of Data Regulations in Data Fabric Automate Quality Analysis with Data Fabric Key Takeaways References Chapter 16: Sample Vendor Offerings Introduction IBM Cloud Pak for Data Amazon Web Services Microsoft Azure Denodo Informatica Key Takeaways References Chapter 17: Data Fabric and Data Mesh Research Areas Introduction AI-Based Augmented Insight AI-Infused Automated AI Governance Hyper-automated Data and AI Fabric Key Takeaways References Chapter 18: In Summary and Onward Data Fabric and Data Mesh Summarized Where to Go from Here Key Takeaways Part IV: Current Offerings and Future Aspects Capture.PNG