Beginning Apache Spark Using Azure Databricks : Unleashing Large Cluster Analytics in the Cloud

Analyze vast amounts of data in record time using Apache Spark with Databricks in the Cloud. Learn the fundamentals, and more, of running analytics on large clusters in Azure and AWS, using Apache Spark with Databricks on top. Discover how to squeeze the most value out of your data at a mere fraction of what classical analytics solutions cost, while at the same time getting the results you need, incrementally faster. This book explains how the confluence of these pivotal technologies gives you enormous power, and cheaply, when it comes to huge datasets. You will begin by learning how cloud infrastructure makes it possible to scale your code to large amounts of processing units, without having to pay for the machinery in advance. From there you will learn how Apache Spark, an open source framework, can enable all those CPUs for data analytics use. Finally, you will see how services such as Databricks provide the power of Apache Spark, without you having to know anything about configuring hardware or software. By removing the need for expensive experts and hardware, your resources can instead be allocated to actually finding business value in the data. This book guides you through some advanced topics such as analytics in the cloud, data lakes, data ingestion, architecture, machine learning, and tools, including Apache Spark, Apache Hadoop, Apache Hive, Python, and SQL. Valuable exercises help reinforce what you have learned. **What You Will Learn** * Discover the value of big data analytics that leverage the power of the cloud * Get started with Databricks using SQL and Python in either Microsoft Azure or AWS * Understand the underlying technology, and how the cloud and Apache Spark fit into the bigger picture * See how these tools are used in the real world * Run basic analytics, including machine learning, on billions of rows at a fraction of a cost or free **Who This Book Is For** Data engineers, data scientists, and cloud architects who want or need to run advanced analytics in the cloud. It is assumed that the reader has data experience, but perhaps minimal exposure to Apache Spark and Azure Databricks. The book is also recommended for people who want to get started in the analytics field, as it provides a strong foundation. Table of Contents 5 About the Author 12 About the Technical Reviewer 13 Introduction 14 Chapter 1: Introduction to Large-Scale聽Data Analytics 15 Analytics, the聽hype 15 Analytics, the聽reality 16 Large-scale analytics for聽fun and聽profit 17 Data: Fueling analytics 19 Free as聽in聽speech. And聽beer! 21 Into the聽clouds 22 Databricks: Analytics for聽the聽lazy ones 23 How to聽analyze data 24 Large-scale examples from聽the聽real world 26 Telematics at Volvo Trucks 26 Fraud detection at聽Visa 26 Customer analytics at聽Target 27 Targeted ads at聽Cambridge Analytica 27 Summary 28 Chapter 2: Spark and聽Databricks 29 Apache Spark, the聽short overview 29 Databricks: Managed Spark 31 The far side of聽the聽Databricks moon 32 Spark architecture 33 Apache Spark processing 34 Working with聽data 35 Data processing 36 Storing data 37 Cool components on聽top of聽Core 38 Summary 38 Chapter 3: Getting Started with聽Databricks 40 Cloud-only 40 Community edition: No money? No problem 41 Mostly good enough 41 Getting started with聽the聽community edition 42 Commercial editions: The聽ones you聽want 43 Databricks on聽Amazon Web Services 45 Azure Databricks 49 Summary 51 Chapter 4: Workspaces, Clusters, and聽Notebooks 52 Getting around in聽the聽UI 52 Clusters: Powering up聽the聽engines 55 Data: Getting access to聽the聽fuel 58 Notebooks: Where the聽work happens 59 Summary 62 Chapter 5: Getting Data into Databricks 63 Databricks File System 63 Navigating the聽file system 64 The FileStore, a聽portal to聽your data 67 Schemas, databases, and聽tables 67 Hive Metastore 68 The many types of聽source files 69 Going binary 70 Alternative transportation 72 Importing from聽your computer 72 Getting data from聽the聽Web 74 Working with聽the聽shell 74 Basic importing with聽Python 76 Getting data with聽SQL 78 Mounting a聽file system 79 Mounting example Amazon S3 79 Mounting example Microsoft Blog Storage 81 Getting rid of聽the聽mounts 82 How to聽get data out of聽Databricks 83 Summary 84 Chapter 6: Querying Data Using SQL 86 The Databricks flavor 86 Getting started 87 Picking up聽data 88 Filtering data 90 Joins and聽merges 93 Ordering data 95 Functions 96 Windowing functions 97 A view worth keeping 100 Hierarchical data 101 Creating data 102 Manipulating data 105 Delta Lake SQL 106 UPDATE, DELETE, and聽MERGE 107 Keeping Delta Lake in聽order 109 Transaction logs 110 Selecting metadata 110 Gathering statistics 112 Summary 113 Chapter 7: The Power of聽Python 114 Python: The聽language of聽choice 114 A turbo-charged intro to聽Python 115 Finding the聽data 118 DataFrames: Where active data lives 119 Getting some data 121 Selecting data from聽DataFrames 125 Chaining combo commands 127 Working with聽multiple DataFrames 136 Slamming data together 143 Summary 147 Chapter 8: ETL and聽Advanced Data聽Wrangling 149 ETL: A聽recap 149 An overview of聽the聽Spark UI 150 Cleaning and聽transforming data 152 Finding nulls 153 Getting rid of聽nulls 154 Filling nulls with聽values 156 Removing duplicates 158 Identifying and聽clearing out extreme values 160 Taking care of聽columns 163 Pivoting 164 Explode 166 When being lazy is good 166 Caching data 168 Data compression 170 A short note about functions 173 Lambda functions 174 Storing and聽shuffling data 175 Save modes 175 Managed vs. unmanaged tables 177 Handling partitions 179 Summary 184 Chapter 9: Connecting to and from Databricks 186 Connecting to聽and聽from聽Databricks 186 Getting ODBC and聽JDBC up聽and聽running 187 Creating a聽token 188 Preparing the聽cluster 189 Let鈥檚 create a聽test table 189 Setting up聽ODBC on聽Windows 190 Setting up聽ODBC on聽OS X 191 Connecting tools to聽Databricks 194 Microsoft Excel on聽Windows 194 Microsoft Power BI Desktop on聽Windows 195 Tableau on聽OS X 196 PyCharm (and more) via Databricks Connect 197 Using RStudio Server 200 Accessing external systems 202 A quick recap of聽libraries 203 Connecting to聽external systems 204 Azure SQL 204 Oracle 205 MongoDB 207 Summary 208 Chapter 10: Running in聽Production 209 General advice 209 Assume the聽worst 210 Write rerunnable code 210 Document in聽the聽code 210 Write clear, simple code 211 Print relevant stuff 212 Jobs 212 Scheduling 214 Running notebooks from聽notebooks 214 Widgets 216 Running jobs with聽parameters 218 The command line interface 220 Setting up聽the聽CLI 220 Running CLI commands 221 Creating and聽running jobs 222 Accessing the聽Databricks File System 223 Picking up聽notebooks 224 Keeping secrets 225 Secrets with聽privileges 227 Revisiting cost 228 Users, groups, and聽security options 228 Users and聽groups 229 Using SCIM provisioning 230 Access Control 230 Workspace Access Control 230 Cluster, Pool, and聽Jobs Access Control 231 Table Access Control 231 Personal Access Tokens 233 The rest 233 Summary 233 Chapter 11: Bits and聽Pieces 235 MLlib 236 Frequent Pattern Growth 236 Creating some data 237 Preparing the聽data 238 Running the聽algorithm 239 Parsing the聽results 240 MLflow 241 Running the聽code 241 Checking the聽results 244 Updating tables 244 Create the聽original table 245 Connect from聽Databricks 246 Pulling the聽delta 247 Verifying the聽formats 248 Update the聽table 249 A short note about Pandas 250 Koalas, Pandas for聽Spark 250 Playing around with聽Koalas 251 The future of聽Koalas 253 The art of聽presenting data 254 Preparing data 255 Using Matplotlib 256 Building and聽showing the聽dashboard 257 Adding a聽widget 257 Adding a聽graph 258 Schedule run 259 REST API and聽Databricks 259 What you聽can do 259 What you聽can鈥檛 do 260 Getting ready for聽APIs 260 Example: Get cluster data 261 Example: Set up聽and聽execute a聽job 263 Example: Get the聽notebooks 266 All the聽APIs and聽what they do 267 Delta streaming 268 Running a聽stream 269 Checking and聽stopping the聽streams 272 Running it faster 273 Using checkpoints 274 Index 276 Table of Contents About the Author About the Technical Reviewer Introduction Chapter 1: Introduction to Large-Scale Data Analytics Analytics, the hype Analytics, the reality Large-scale analytics for fun and profit Data: Fueling analytics Free as in speech. And beer! Into the clouds Databricks: Analytics for the lazy ones How to analyze data Large-scale examples from the real world Telematics at Volvo Trucks Fraud detection at Visa Customer analytics at Target Targeted ads at Cambridge Analytica Summary Chapter 2: Spark and Databricks Apache Spark, the short overview Databricks: Managed Spark The far side of the Databricks moon Spark architecture Apache Spark processing Working with data Data processing Storing data Cool components on top of Core Summary Chapter 3: Getting Started with Databricks Cloud-only Community edition: No money? No problem Mostly good enough Getting started with the community edition Commercial editions: The ones you want Databricks on Amazon Web Services Azure Databricks Summary Chapter 4: Workspaces, Clusters, and Notebooks Getting around in the UI Clusters: Powering up the engines Data: Getting access to the fuel Notebooks: Where the work happens Summary Chapter 5: Getting Data into Databricks Databricks File System Navigating the file system The FileStore, a portal to your data Schemas, databases, and tables Hive Metastore The many types of source files Going binary Alternative transportation Importing from your computer Getting data from the Web Working with the shell Basic importing with Python Getting data with SQL Mounting a file system Mounting example Amazon S3 Mounting example Microsoft Blog Storage Getting rid of the mounts How to get data out of Databricks Summary Chapter 6: Querying Data Using SQL The Databricks flavor Getting started Picking up data Filtering data Joins and merges Ordering data Functions Windowing functions A view worth keeping Hierarchical data Creating data Manipulating data Delta Lake SQL UPDATE, DELETE, and MERGE Keeping Delta Lake in order Transaction logs Selecting metadata Gathering statistics Summary Chapter 7: The Power of Python Python: The language of choice A turbo-charged intro to Python Finding the data DataFrames: Where active data lives Getting some data Selecting data from DataFrames Chaining combo commands Working with multiple DataFrames Slamming data together Summary Chapter 8: ETL and Advanced Data Wrangling ETL: A recap An overview of the Spark UI Cleaning and transforming data Finding nulls Getting rid of nulls Filling nulls with values Removing duplicates Identifying and clearing out extreme values Taking care of columns Pivoting Explode When being lazy is good Caching data Data compression A short note about functions Lambda functions Storing and shuffling data Save modes Managed vs. unmanaged tables Handling partitions Summary Chapter 9: Connecting to and from Databricks Connecting to and from Databricks Getting ODBC and JDBC up and running Creating a token Preparing the cluster Let’s create a test table Setting up ODBC on Windows Setting up ODBC on OS X Connecting tools to Databricks Microsoft Excel on Windows Microsoft Power BI Desktop on Windows Tableau on OS X PyCharm (and more) via Databricks Connect Using RStudio Server Accessing external systems A quick recap of libraries Connecting to external systems Azure SQL Oracle MongoDB Summary Chapter 10: Running in Production General advice Assume the worst Write rerunnable code Document in the code Write clear, simple code Print relevant stuff Jobs Scheduling Running notebooks from notebooks Widgets Running jobs with parameters The command line interface Setting up the CLI Running CLI commands Creating and running jobs Accessing the Databricks File System Picking up notebooks Keeping secrets Secrets with privileges Revisiting cost Users, groups, and security options Users and groups Using SCIM provisioning Access Control Workspace Access Control Cluster, Pool, and Jobs Access Control Table Access Control Personal Access Tokens The rest Summary Chapter 11: Bits and Pieces MLlib Frequent Pattern Growth Creating some data Preparing the data Running the algorithm Parsing the results MLflow Running the code Checking the results Updating tables Create the original table Connect from Databricks Pulling the delta Verifying the formats Update the table A short note about Pandas Koalas, Pandas for Spark Playing around with Koalas The future of Koalas The art of presenting data Preparing data Using Matplotlib Building and showing the dashboard Adding a widget Adding a graph Schedule run REST API and Databricks What you can do What you can’t do Getting ready for APIs Example: Get cluster data Example: Set up and execute a job Example: Get the notebooks All the APIs and what they do Delta streaming Running a stream Checking and stopping the streams Running it faster Using checkpoints Index Analyze vast amounts of data in record time using Apache Spark with Databricks in the Cloud. Learn the fundamentals, and more, of running analytics on large clusters in Azure and AWS, using Apache Spark with Databricks on top. Discover how to squeeze the most value out of your data at a mere fraction of what classical analytics solutions cost, while at the same time getting the results you need, incrementally faster. This book explains how the confluence of these pivotal technologies gives you enormous power, and cheaply, when it comes to huge datasets. You will begin by learning how cloud infrastructure makes it possible to scale your code to large amounts of processing units, without having to pay for the machinery in advance. From there you will learn how Apache Spark, an open source framework, can enable all those CPUs for data analytics use. Finally, you will see how services such as Databricks provide the power of Apache Spark, without you having to know anything about configuring hardware or software. By removing the need for expensive experts and hardware, your resources can instead be allocated to actually finding business value in the data. This book guides you through some advanced topics such as analytics in the cloud, data lakes, data ingestion, architecture, machine learning, and tools, including Apache Spark, Apache Hadoop, Apache Hive, Python, and SQL. Valuable exercises help reinforce what you have learned. What You Will Learn Discover the value of big data analytics that leverage the power of the cloud Get started with Databricks using SQL and Python in either Microsoft Azure or AWS Understand the underlying technology, and how the cloud and Spark fit into the bigger picture See how these tools are used in the real world Run basic analytics, including machine learning, on billions of rows at a fraction of a cost or free This book is for data engineers, data scientists, and cloud architects who want or need to run advanced analytics in the cloud. It is assumed that the reader has data experience, but perhaps minimal exposure to Apache Spark and Azure Databricks. The book is also recommended for people who want to get started in the analytics field, as it provides a strong foundation. Robert Ilijason is a 20-year veteran in the business intelligence (BI) segment. He has worked as a contractor for some of Europe's biggest companies and has conducted large-scale analytics projects within the areas of retail, telecom, banking, government, and more. Robert has seen his share of analytic trends come and go over the years, but unlike most of them, he strongly believes that Apache Spark in the cloud, especially with Azure Databricks, is a game changer