Modern Data Architecture on Azure: Design Data-centric Solutions on Microsoft Azure

This book is an exhaustive guide to designing and implementing data solutions on Azure. It covers the process of managing data from end to end, starting from data collection, all the way through transformation, distribution, and consumption. A Definitive Guide to Modern Data Architecture on Azure begins with an introduction to the fundaments of data management, followed by demonstration how to build relational and non-relational data solutions on Azure. Here, you will learn data processing for complex analysis and how to work with CSV and JSON files. Moving forward, you will learn the foundational concepts of big data architecture, along with data management patterns and technology options offered by Azure. From there, you’ll be walked through the data architecture process, including data consortium on Azure, enterprise data governance, and much more. The book culminates with a deep dive into data architecture frameworks with data modeling. After reading this book, you will have a thorough understanding of data design and analytics using Azure, allowing you to collect and analyze massive amounts of data to optimize business performance, forecast future results, and more. What Will You Learn• Understand the fundamentals of data architecture including data management, data handling ethics, data governance, and metadata management• Analyze and understand business needs to choose the right Azure services and make informed business decisions• Understand Azure Cloud Data Design patterns for relational and non-relational data, batch-real-time processing, and ETL/ELT pipeline • Modernize data architecture using Azure to leverage data and AI to enable digital transformation by securing and optimizing overall data lifecycle management Who Is This Book For:Data solution architects, data engineers, and IT consultants who want to gain a better understanding of modern data architecture design and implementation on Azure. Table of Contents 5 About the Author 10 About the Technical Reviewer 11 Acknowledgments 12 Introduction 13 Chapter 1: Introduction: Fundamentals of Data Management 14 Introduction to DAMA and DMBOK 15 Essential Data Concepts 16 Types of Data 16 Qualitative Data 17 Nominal Data 17 Ordinal Data 18 Quantitative Data 18 Discrete Data 19 Continuous Data 19 Data Management Principles 19 The Data Lifecycle 20 Consistency Models 21 Data Ingestion Patterns 21 Data Platform Paradigm 22 Data Management Principles and Challenges 25 Preparing a Data Strategy 25 Defining Roles and Responsibilities 26 Data Lifecycle Management 27 Data Quality Measurements 28 Metadata 28 Maximizing Data Value for Data-Driven Decisions 30 Dealing with Substantial Volumes of Data 30 Siloed and Varied Data Sources 31 Maintaining the Quality of the Data 31 Data Integration 31 Data Governance and Security 32 Data Automation 32 Data Management Frameworks 33 The Strategic Alignment Model 33 The Amsterdam Information Model 35 The DAMA DMBOK Framework 36 The DAMA Wheel 42 Data Governance 43 Data Architecture 44 Data Modeling and Design 45 Data Storage and Operations 46 Data Security 46 Data Integration and Interoperability 47 Document and Content Management 47 Reference and Master Data 47 Data Warehousing and Business Intelligence 48 Metadata 48 Data Quality 49 Understanding the Environmental Factors Hexagon 50 Understanding the Knowledge Area Context Diagram 51 Conclusion 52 Chapter 2: Build Relational and Non-Relational Data Solutions on Azure 53 Data Integration Using ETL 54 Data Extraction 55 Data Transformation 56 Data Loading 56 Designing ELT Pipelines Using the Azure Synapse Server 57 Online Analytical Processing for Complex Analyses 59 Semantic Data Modeling 63 Challenges of Using OLAP Solutions 65 Managing Transaction Data Using OLTP 66 Managing Non-Relational Data 72 Key-Value Pair Databases 73 Column Family Databases 74 Document Databases 74 Graph Databases 75 Handling Time-Series and Free-Form Search Data 77 Working with CSV and JSON Files for Data Solutions 84 Conclusion 87 Chapter 3: Building a Big Data Architecture 88 Core Components of a Big Data Architecture 89 Data Ingestion and Processing 90 Data Analysis 92 Data Visualization 93 Data Governance 94 Using Batch Processing 94 Azure Synapse Analytics 97 Azure Data Lake Analytics 97 Azure Databricks 98 Azure Data Explorer 99 Real-Time Processing 100 Real-Time Data Ingestion 103 The Lambda Architecture 106 The Kappa Architecture 110 Internet of Things (IoT) 116 Data Mesh Principles and the Logical Architecture 118 Conclusion 123 Chapter 4: Data Management Patterns and Technology Choices with Azure 124 Data Patterns and Trends in Depth 125 CQRS Pattern 125 Event Sourcing 128 Materialized Views 128 Index Table Pattern 129 Analytical Store for Big Data Analytics 131 Azure Synapse Analytics 131 Azure Databricks 133 Data Ingestion Process 134 Data Storage 135 Data Transformation and Model Training 135 Analytics 135 Azure Data Explorer 136 Building Enterprise Data Lakes and Data Lakehouses 137 Enterprise Data Lakes 138 Enterprise Data Lakehouses 142 Data Pipeline Orchestration 144 Real-Time Stream Processing in Azure 149 Conclusion 152 Chapter 5: Data Architecture Process 153 Guide to Data Modeling 153 Conceptual Data Model 155 Logical Data Model 156 Physical Data Model 157 Focus on Business Objectives and its Requirements 157 Data Lake for Ad Hoc Queries 160 Enterprise Data Governance: Data Scrambling, Obfuscation, and DataOps 165 Data Masking Techniques 168 Data Scrambling 170 Data Encryption 171 Data Ageing 171 Data Substitution 171 Data Shuffling 171 Pseudonymization 172 Master Data Management and Storage Optimization 173 Master Data Management 174 Data Encryption Patterns 180 Conclusion 184 Chapter 6: Data Architecture Framework Explained 185 Fundamentals of Data Modeling 185 The Network Data Model 187 The Hierarchical Data Model 188 The Relational Data Model 189 The Object-Oriented Data Model 190 The Dimensional Data Model 191 The Graph Data Model 192 The Entity Relationship Data Model 193 The Open Group Architecture Framework 194 Preliminary Phase 197 Defining the Architecture Vision 197 Business Architecture 198 Information System Architecture 198 Technology Architecture 198 Opportunities and Solutions 199 Migration Planning 199 Governance Implementation 200 Architecture Change Management 200 DAMA DMBOK 200 The Zachman Framework 205 Conclusion 208 Index 209 This book is an exhaustive guide to designing and implementing data solutions on Azure. It covers the process of managing data from end to end, starting from data collection all the way through transformation, distribution, and consumption. Modern Data Architecture on Azure begins with an introduction to the fundaments of data management, followed by a demonstration of how to build relational and non-relational data solutions on Azure. Here, you will learn data processing for complex analysis and how to work with CSV and JSON files. Moving forward, you will learn the foundational concepts of big data architecture, along with data management patterns and technology options offered by Azure. From there, you’ll be walked through the data architecture process, including data consortium on Azure, enterprise data governance, and much more. The book culminates with a deep dive into data architecture frameworks with data modeling. After reading this book, you will have a thorough understanding of data design and analytics using Azure, allowing you to collect and analyze massive amounts of data to optimize business performance, forecast future results, and more. What Will You Learn Understand the fundamentals of data architecture including data management, data handling ethics, data governance, and metadata management Analyze and understand business needs to choose the right Azure services and make informed business decisions Understand Azure Cloud Data design patterns for relational and non-relational data, batch real-time processing, and ETL/ELT pipelines Modernize data architecture using Azure to leverage data and AI to enable digital transformation by securing and optimizing overall data lifecycle management Who Is This Book For: Data solution architects, data engineers, and IT consultants who want to gain a better understanding of modern data architecture design and implementation on Azure.