Data Lake Analytics on Microsoft Azure : A Practitioner's Guide to Big Data Engineering

Get a 360-degree view of how the journey of data analytics solutions has evolved from monolithic data stores and enterprise data warehouses to data lakes and modern data warehouses. You will This book includes comprehensive coverage of how: * To architect data lake analytics solutions by choosing suitable technologies available on Microsoft Azure * The advent of microservices applications covering ecommerce or modern solutions built on IoT and how real-time streaming data has completely disrupted this ecosystem * These data analytics solutions have been transformed from solely understanding the trends from historical data to building predictions by infusing machine learning technologies into the solutions Data platform professionals who have been working on relational data stores, non-relational data stores, and big data technologies will find the content in this book useful. The book also can help you start your journey into the data engineer world as it provides an overview of advanced data analytics and touches on data science concepts and various artificial intelligence and machine learning technologies available on Microsoft Azure. **What Will You Learn** You will understand the:* Concepts of data lake analytics, the modern data warehouse, and advanced data analytics * Architecture patterns of the modern data warehouse and advanced data analytics solutions * Phases—such as Data Ingestion, Store, Prep and Train, and Model and Serve—of data analytics solutions and technology choices available on Azure under each phase * In-depth coverage of real-time and batch mode data analytics solutions architecture * Various managed services available on Azure such as Synapse analytics, event hubs, Stream analytics, CosmosDB, and managed Hadoop services such as Databricks and HDInsight **Who This Book Is For**Data platform professionals, database architects, engineers, and solution architects Table of Contents 5 About the Authors 8 About the Technical Reviewer 9 Foreword 10 Acknowledgments 12 Introduction 14 Chapter 1: Data Lake Analytics Concepts 15 What’s Data Analytics? 15 Relational and Nonrelational Data Stores 16 Relational Data Stores 17 Nonrelational Data Stores 17 Evolution of Data Analytics Systems 19 Enterprise Data Warehouse 20 Big Data Systems 21 Massively Parallel Processing 22 Data Lake Analytics Rationale 23 Conclusion 24 Chapter 2: Building Blocks of Data Analytics 25 Building Blocks of Data Analytics Solutions 25 Data Sources 26 Data Ingestion 27 Data Storage 32 Data Preparation and Training 34 Model and Serve 37 Data consumption and data visualization 38 Conclusion 39 Chapter 3: Data Analytics on Public Cloud 40 Reference Architectures on Public Cloud 40 Traditional on-premises 41 Infrastructure as a service (IaaS) 41 Platform as a service (PaaS) 42 Data Analytics on Microsoft Azure 47 Conclusion 54 Chapter 4: Data Ingestion 55 Data Ingestion 55 Real-time mode 56 Apache Kafka on HDInsight Cluster 59 Exercise: Create an Apache Kafka Cluster in Azure HDInsight for Data Ingestion and Analysis 66 Event Hub 69 Why Event Hubs? 71 Exercise: Ingesting Real-Time Twitter Data Using Event Hub 73 Batch Data Ingestion 83 Azure Data Factory 84 Exercise: Incrementally Load Data from On-premises SQL Server to Azure Blob Storage 87 Conclusion 97 Chapter 5: Data Storage 98 Data Store 98 Data Storage Options 100 Blob Storage 100 Azure Data Lake Storage (ADLS) 106 Exercise: Put streaming Data Coming from Event Hubs Directly to Azure Data Lake Storage 108 Summary 109 Chapter 6: Data Preparation and Training Part I 110 Data Preparation and Training 111 Data Preparation 113 Process Real-Time Data Streams 113 Apache Spark 114 Continuous Application 119 Apache Spark in Azure Databricks 120 Exercise: Sentiment Analysis of Streaming Data Using Azure Databricks 122 Stream Analytics 130 Exercise: Sentiment Analysis on Streaming Data Using Stream Analytics 132 Process Batch Mode Data 140 Enterprise Data Warehouse 140 Data Lake Analytics 141 Modern Data Warehouse 142 Exercise: ELT Using Azure Data Factory and Azure Databricks 143 Summary 153 Chapter 7: Data Preparation and Training Part II 154 Advanced Data Analytics 154 Basics of Data Science 155 Machine Learning and Deep Learning Overview 162 Summary 191 Chapter 8: Model and Serve 192 Model and Serve 192 Real-Time Data Refresh 195 Stream Analytics and Power BI: A Real-Time Analytics Dashboard for Streaming Data 195 Modify the Stream Analytics Job to Add Power BI Output 195 Create the Dashboard in Power BI 198 Azure Synapse Analytics 209 Azure Analysis Services 212 Power BI 213 Azure Data Explorer 215 Summary 220 Chapter 9: Summary 221 Index 224 Get a 360-degree view of how the journey of data analytics solutions has evolved from monolithic data stores and enterprise data warehouses to data lakes and modern data warehouses. You will This book includes comprehensive coverage of how: To architect data lake analytics solutions by choosing suitable technologies available on Microsoft Azure The advent of microservices applications covering ecommerce or modern solutions built on IoT and how real-time streaming data has completely disrupted this ecosystem These data analytics solutions have been transformed from solely understanding the trends from historical data to building predictions by infusing machine learning technologies into the solutions Data platform professionals who have been working on relational data stores, non-relational data stores, and big data technologies will find the content in this book useful. The book also can help you start your journey into the data engineer world as it provides an overview of advanced data analytics and touches on data science concepts and various artificial intelligence and machine learning technologies available on Microsoft Azure. What Will You Learn You will understand the: Concepts of data lake analytics, the modern data warehouse, and advanced data analytics Architecture patterns of the modern data warehouse and advanced data analytics solutions Phases—such as Data Ingestion, Store, Prep and Train, and Model and Serve—of data analytics solutions and technology choices available on Azure under each phase In-depth coverage of real-time and batch mode data analytics solutions architecture Various managed services available on Azure such as Synapse analytics, event hubs, Stream analytics, CosmosDB, and managed Hadoop services such as Databricks and HDInsight Who This Book Is For Data platform professionals, database architects, engineers, and solution architects Get a 360-degree view of how the journey of data analytics solutions has evolved from monolithic data stores and enterprise data warehouses to data lakes and modern data warehouses. You will learn from the authors' experience working with large-scale enterprise customer engagements. This book includes comprehensive coverage of how: To architect data lake analytics solutions by choosing suitable technologies available on Microsoft Azure The advent of microservices applications covering ecommerce or modern solutions built on IoT and how real-time streaming data has completely disrupted this ecosystem These data analytics solutions have been transformed from solely understanding the trends from historical data to building predictions by infusing machine learning technologies into the solutions Data platform professionals who have been working on relational data stores, non-relational data stores, and big data technologies will find the content in this book useful. The book also can help you start your journey into the data engineer world as it provides an overview of advanced data analytics and touches on data science concepts and various artificial intelligence and machine learning technologies available on Microsoft Azure. You will understand the: Concepts of data lake analytics, the modern data warehouse, and advanced data analytics Architecture patterns of the modern data warehouse and advanced data analytics solutions Phases--such as Data Ingestion, Store, Prep and Train, and Model and Serve--of data analytics solutions and technology choices available on Azure under each phase In-depth coverage of real-time and batch mode data analytics solutions architecture Various managed services available on Azure such as Synapse analytics, event hubs, Stream analytics, CosmosDB, and managed Hadoop services such as Databricks and HDInsight Front Matter ....Pages i-xvii Data Lake Analytics Concepts (Harsh Chawla, Pankaj Khattar)....Pages 1-10 Building Blocks of Data Analytics (Harsh Chawla, Pankaj Khattar)....Pages 11-25 Data Analytics on Public Cloud (Harsh Chawla, Pankaj Khattar)....Pages 27-41 Data Ingestion (Harsh Chawla, Pankaj Khattar)....Pages 43-85 Data Storage (Harsh Chawla, Pankaj Khattar)....Pages 87-98 Data Preparation and Training Part I (Harsh Chawla, Pankaj Khattar)....Pages 99-142 Data Preparation and Training Part II (Harsh Chawla, Pankaj Khattar)....Pages 143-180 Model and Serve (Harsh Chawla, Pankaj Khattar)....Pages 181-209 Summary (Harsh Chawla, Pankaj Khattar)....Pages 211-213 Back Matter ....Pages 215-222