Learn PySpark : Build Python-based Machine Learning and Deep Learning Models

Leverage machine and deep learning models to build applications on real-time data using PySpark. This book is perfect for those who want to learn to use this language to perform exploratory data analysis and solve an array of business challenges.You'll start by reviewing PySpark fundamentals, such as Spark’s core architecture, and see how to use PySpark for big data processing like data ingestion, cleaning, and transformations techniques. This is followed by building workflows for analyzing streaming data using PySpark and a comparison of various streaming platforms. You'll then see how to schedule different spark jobs using Airflow with PySpark and book examine tuning machine and deep learning models for real-time predictions. This book concludes with a discussion on graph frames and performing network analysis using graph algorithms in PySpark. All the code presented in the book will be available in Python scripts on Github.**What You'll Learn** * Develop pipelines for streaming data processing using PySpark * Build Machine Learning & Deep Learning models using PySpark latest offerings * Use graph analytics using PySpark * Create Sequence Embeddings from Text data **Who This Book is For** Data Scientists, machine learning and deep learning engineers who want to learn and use PySpark for real time analysis on streaming data. Table of Contents 5 About the Author 10 About the Technical Reviewer 11 Acknowledgments 12 Introduction 13 Chapter 1: Introduction to Spark 15 History 15 Data Collection 16 Data Storage 17 Data Processing 17 Spark Architecture 18 Storage 19 Resource Management 19 Engine and Ecosystem 22 Spark SQL 22 MLlib 22 Structured Streaming 22 Graph X 23 Programming Language APIs 23 Setting Up Your Environment 24 Local Setup 24 Dockers 25 Cloud Environments 25 Databricks 25 Conclusion 30 Chapter 2: Data Processing 31 Creating a SparkSession Object 32 Creating Dataframes 32 Null Values 33 Subset of a Dataframe 37 Select 38 Filter 39 Where 40 Aggregations 40 Collect 49 User-Defined Functions (UDFs) 51 Pandas UDF 54 Joins 55 Pivoting 57 Window Functions or Windowed Aggregates 58 Conclusion 62 Chapter 3: Spark Structured Streaming 63 Batch vs. Stream 63 Batch Data 64 Stream Processing 64 Spark Streaming 65 Structured Streaming 67 Data Input 70 Data Processing 71 Final Output 71 Building a Structured App 71 Operations 73 Joins 77 Structured Streaming Alternatives 79 Conclusion 79 Chapter 4: Airflow 80 Workflows 80 Graph Overview 82 Undirected Graphs 82 Directed Graphs 83 DAG Overview 84 Operators 86 Installing Airflow 87 Airflow Using Docker 87 Airflow Setup (Mac) 89 Creating Your First DAG 89 Step 1: Importing the Required Libraries 91 Step 2: Defining the Default Arguments 91 Step 3: Creating a DAG 92 Step 4: Declaring Tasks 92 Step 5: Mentioning Dependencies 93 Conclusion 97 Chapter 5: MLlib: Machine Learning Library 98 Calculating Correlations 99 Chi-Square Test 102 Transformations 107 Binarizer 107 Principal Component Analysis 109 Normalizer 111 Standard Scaling 113 Min-Max Scaling 114 MaxAbsScaler 116 Binning 117 Building a Classification Model 120 Step 1: Load the Dataset 120 Step 2: Explore the Dataframe 121 Step 3: Data Transformation 123 Step 4: Splitting into Train and Test Data 125 Step 5: Model Training 125 Step 6: Hyperparameter Tuning 126 Step 7: Best Model 128 Conclusion 128 Chapter 6: Supervised Machine Learning 129 Supervised Machine Learning Primer 129 Binary Classification 132 Multi-class Classification 133 Building a Linear Regression Model 133 Reviewing the Data Information 135 Step 1: Create the Spark Session Object 135 Step 2: Read the Dataset 135 Step 3: Feature Engineering 137 Step 4: Split the Dataset 138 Step 5: Build and Train Linear Regression Model 139 Step 6: Evaluate Linear Regression Model on Test Data 140 Generalized Linear Model Regression 140 Step 1: Build and Train Generalized Linear Regression Model 140 Step 2: Evaluate the Model Performance on Test Data 141 Decision Tree Regression 143 Step 1: Build and Train Decision Tree Regressor Model 144 Step 2: Evaluate the Model Performance on Test Data 144 Random Forest Regressors 145 Step 1: Build and Train Random Forest Regressor Model 146 Step 2: Evaluate the Model Performance on Test Data 147 Gradient-Boosted Tree Regressor 148 Step 1: Build and Train a GBT Regressor Model 148 Step 2: Evaluate the Model Performance on Test Data 149 Building Multiple Models for Binary Classification Tasks 150 Logistic Regression 150 Step 1: Read the Dataset 151 Step 2: Feature Engineering for Model 152 Step 3: Split the Data into Train and Test Datasets 154 Step 4: Build and Train the Logistic Regression Model 154 Step 5: Evaluate Performance on Training Data 155 Step 6: Evaluate Performance on Test Data 158 Decision Tree Classifier 160 Step 1: Build and Train Decision Tree Classifier Model 160 Step 2: Evaluate Performance on Test Data 161 Support Vector Machines Classifiers 162 Step 1: Build and Train SVM Model 163 Step 2: Evaluate Performance on Test Data 163 Naive Bayes Classifier 164 Step 1: Build and Train SVM Model 165 Step 2: Evaluate Performance on Test Data 165 Gradient Boosted Tree Classifier 166 Step 1: Build and Train the GBT Model 166 Step 2: Evaluate Performance on Test Data 167 Random Forest Classifier 168 Step 1: Build and Train the Random Forest Model 168 Step 2: Evaluate Performance on Test Data 169 Hyperparameter Tuning and Cross-Validation 170 Conclusion 171 Chapter 7: Unsupervised Machine Learning 172 Unsupervised Machine Learning Primer 172 Reviewing the Dataset 176 Importing SparkSession and Creating an Object 176 Reshaping a Dataframe for Clustering 180 Building Clusters with K-Means 184 Conclusion 192 Chapter 8: Deep Learning Using PySpark 193 Deep Learning Fundamentals 193 Human Brain Neuron vs. Artificial Neuron 195 Activation Functions 198 Sigmoid 198 Hyperbolic Tangent 198 Rectified Linear Unit 199 Neuron Computation 200 Training Process: Neural Network 202 Building a Multilayer Perceptron Model 208 Conclusion 213 Index 214 Leverage machine and deep learning models to build applications on real-time data using PySpark. This book is perfect for those who want to learn to use this language to perform exploratory data analysis and solve an array of business challenges. You'll start by reviewing PySpark fundamentals, such as Spark’s core architecture, and see how to use PySpark for big data processing like data ingestion, cleaning, and transformations techniques. This is followed by building workflows for analyzing streaming data using PySpark and a comparison of various streaming platforms. You'll then see how to schedule different spark jobs using Airflow with PySpark and book examine tuning machine and deep learning models for real-time predictions. This book concludes with a discussion on graph frames and performing network analysis using graph algorithms in PySpark. All the code presented in the book will be available in Python scripts on Github. What You'll Learn Develop pipelines for streaming data processing using PySpark Build Machine Learning & Deep Learning models using PySpark latest offerings Use graph analytics using PySpark Create Sequence Embeddings from Text data Who This Book is For Data Scientists, machine learning and deep learning engineers who want to learn and use PySpark for real time analysis on streaming data. Front Matter ....Pages i-xviii Introduction to Spark (Pramod Singh)....Pages 1-16 Data Processing (Pramod Singh)....Pages 17-48 Spark Structured Streaming (Pramod Singh)....Pages 49-65 Airflow (Pramod Singh)....Pages 67-84 MLlib: Machine Learning Library (Pramod Singh)....Pages 85-115 Supervised Machine Learning (Pramod Singh)....Pages 117-159 Unsupervised Machine Learning (Pramod Singh)....Pages 161-181 Deep Learning Using PySpark (Pramod Singh)....Pages 183-203 Back Matter ....Pages 205-210