PRO SERVERLESS DATA HANDLING WITH MICROSOFT AZURE : architecting etl and data driven applications... in the cloud

Design and build architectures on the Microsoft Azure platform specifically for data-driven and ETL applications. Modern cloud architectures rely on serverless components more than ever, and this book helps you identify those components of data-driven or ETL applications that can be tackled using the technologies available on the Azure platform. The book shows you which Azure components are best suited to form a strong foundation for data-driven applications in the Microsoft Azure Cloud. If you are a solution architect or a decision maker, the conceptual aspects of this book will help you gain a deeper understanding of the underlying technology and its capabilities. You will understand how to develop using Azure Functions, Azure Data Factory, Logic Apps, and to employ serverless databases in your application to achieve the best scalability and design. If you are a developer, you will benefit from the hands-on approach used throughout this book. Many practical examples andarchitectures applied in real-world projects will be valuable to you on your path to serverless success. What You Will Learn Know what services are available in Microsoft Azure that can deal with large amounts of data Design modern data applications based on serverless technology in the cloud Transform and present data without the use of infrastructure Employ proven design patterns for rapid implementation of serverless data applications Choose the correct set of development tools for the services you are using Understand the term "serverless" and how it can be a benefit Identify scenarios in which serverless is not the best option available Who This Book Is For Architects and decision makers who want to understand how modern architectures are designed and how to modernize their applications. The book is aimed at the developer who needs a steppingstone to quickly implement a serverless data application. And the book is for any IT professional who seeks a head start to serverless computing for data-heavy applications on the Azure platform. Table of Contents 5 About the Authors 11 About the Technical Reviewer 12 Acknowledgments 13 Introduction 14 Part I: The Basics 16 Chapter 1: Azure Basics 17 The Different Cloud Service Models 17 Infrastructure as a Service (IaaS) 18 Platform as a Service (PaaS) 18 Software as a Service (SaaS) 19 Cloud Model Responsibilities 19 The Structure of Microsoft Azure 20 Azure Geographies 20 Azure Regions 21 Azure Availability Zones 22 Azure Account 22 Azure Subscription 22 Azure Resource Groups 22 Azure Resource Manager 23 Creating and Naming the Resources 24 Creating Resources 25 Naming Resources 25 Overview of Data Services 26 Data Categories 26 Azure Data Services 27 Summary 30 Chapter 2: Serverless Computing 31 Cloud Software Delivery 31 Serverless Delivery 35 The Cost of Perfection 40 Handling Data 42 Chapter 3: Data-Driven Applications 45 ETL the Classic Way 45 Transformation: What Does That Mean? 46 Different Data Models for Different Applications 48 OLTP: The Relational Model 48 Table 48 Key 50 Relationship 52 OLAP: Star and Snowflake Schemas 55 Modern Data Warehouses and Data Applications 58 Part II: Hands-On 61 Chapter 4: Azure Functions 62 The Flavors of Azure Functions 62 Triggers and Bindings 63 Creating Your First Azure Function 67 Creating the Azure Resources 67 Creating the Function 70 A Look at the Code 74 Testing the Function 76 Deploying Your Function 80 Handling State 83 The Basics 84 The Code 85 Running It in the Cloud 89 Chapter 5: Logic Apps 92 Principles of Code-Free Implementation 93 Creating a Logic App 94 The Logic Apps UI 97 Chapter 6: Azure Data Factory 106 The Building Blocks of ADF 107 Working with Azure Data Factory 108 Creating an ADF Using Azure CLI 108 Preparing Resources 113 Creating a Pipeline 117 Parametrizing Your Pipeline 123 Creating a Data Flow 128 Best Practices 130 Using Git 130 Using Azure Key Vault 132 Chapter 7: Database and Storage Options 133 Relational and Non-Relational Data Explained 133 Storage Accounts 135 Storage Account Basics 135 Creating a Storage Account 138 Using Azure Table Storage 140 Azure Queue Storage 143 Cosmos DB 147 Use Cases for Cosmos DB Accounts 151 Azure SQL DB Serverless 155 Creating a Serverless SQL Database 158 When to Choose What? 162 Chapter 8: IoT Hub, Event Hub, and Streaming Data 164 IoT Hub 165 Event Hub 170 Service Bus 173 Stream Analytics 175 Chapter 9: Power BI 180 Power BI Service and Power BI Desktop 181 Building Data Visualizations with Power BI Reports 188 Visualizing Data Streams 195 Sharing Content 197 Licensing of Power BI 201 Part III: Design Practices 203 Chapter 10: Achieving Resiliency 204 What Is Resiliency? 204 How Is Resiliency Ensured? 208 Different Areas to Be Resilient 209 Patterns That Support Resiliency 212 Choosing the Right Services for Resiliency 216 Achieving Resiliency 219 Chapter 11: Queues, Messages, and Commands 221 Messages 221 Events 222 Commands 224 Scenarios for Events and Commands 225 Implementing the Scenario 228 Chapter 12: Processing Streams of Data 239 Streaming Data—What Is It About? 239 Stream Processing: Lambda Architecture 242 Implementing a Lambda Architecture in Azure 243 There’s More... 247 Chapter 13: Monitoring Serverless Applications 249 Monitoring and Alerting 249 Serverless and Monitoring 252 Implementing Monitoring 253 Implementing Alerting 256 Part IV: Putting It All Together 258 Chapter 14: Tools and Helpers 259 Visual Studio Code 259 Azure Data Studio 260 Docker / Docker Desktop 261 Azure CLI 261 PowerShell 262 Bicep / ARM Templates 262 Azure Storage Explorer 263 Azure DevOps 263 dbatools 264 Azure Quickstart Templates 264 Git 264 Git Kraken 265 Chocolatey 265 Azure Data Community 266 Useful Visual Studio Code Plugins 266 Chapter 15: Data-Loading Patterns 269 Data-Loading Patterns for Flat Files 271 Data-Loading Patterns for REST APIs 275 Data-Loading Patterns for Databases 278 Data-Loading Patterns for Data Streams 281 Chapter 16: Data Storage Patterns 284 Relational Databases 285 Storage Accounts 288 Non-Relational Databases 293 Chapter 17: Architecture for a Modern Data-Driven Application 301 REST API, Tracking & Transaction Data 303 Communicating with the Shops 307 Data Warehousing and Analytics 309 Index 312 Design and build architectures on the Microsoft Azure platform specifically for data-driven and ETL applications. Modern cloud architectures rely on serverless components more than ever, and this book helps you identify those components of data-driven or ETL applications that can be tackled using the technologies available on the Azure platform. The book shows you which Azure components are best suited to form a strong foundation for data-driven applications in the Microsoft Azure Cloud. If you are a solution architect or a decision maker, the conceptual aspects of this book will help you gain a deeper understanding of the underlying technology and its capabilities. You will understand how to develop using Azure Functions, Azure Data Factory, Logic Apps, and to employ serverless databases in your application to achieve the best scalability and design. If you are a developer, you will benefit from the hands-on approach used throughout this book. Many practical examples and architectures applied in real-world projects will be valuable to you on your path to serverless success. What You Will Learn• Know what services are available in Microsoft Azure that can deal with large amounts of data• Design modern data applications based on serverless technology in the cloud• Transform and present data without the use of infrastructure• Employ proven design patterns for rapid implementation of serverless data applications• Choose the correct set of development tools for the services you are using• Understand the term “serverless” and how it can be a benefit• Identify scenarios in which serverless is not the best option available