Applied Data Science Using PySpark: Learn the End-to-End Predictive Model-Building Cycle (Second Edition)

This comprehensive guide, featuring hand-picked examples of daily use cases, will walk you through the end-to-end predictive model-building cycle using the latest techniques and industry tricks. In Chapters 1, 2, and 3, we will begin by setting up the environment and covering the basics of PySpark, focusing on data manipulation. Chapter 4 delves into the art of variable selection, demonstrating various techniques available in PySpark. In Chapters 5, 6, and 7, we explore machine learning algorithms, their implementations, and fine-tuning techniques. Chapters 8 and 9 will guide you through machine learning pipelines and various methods to operationalize and serve models using Docker/API. Chapter 10 will demonstrate how to unlock the power of predictive models to create a meaningful impact on your business. Chapter 11 introduces some of the most widely used and powerful modeling frameworks to unlock real value from data. In this new edition, you will learn predictive modeling frameworks that can quantify customer lifetime values and estimate the return on your predictive modeling investments. This edition also includes methods to measure engagement and identify actionable populations for effective churn treatments. Additionally, a dedicated chapter on experimentation design has been added, covering steps to efficiently design, conduct, test, and measure the results of your models. All code examples have been updated to reflect the latest stable version of Spark. You will: Gain an overview of end-to-end predictive model building Understand multiple variable selection techniques and their implementations Learn how to operationalize models Perform data science experiments and learn useful tips Table of Contents About the Authors About the Technical Reviewer Acknowledgments Introduction Chapter 1: Setting Up the PySpark Environment Local Installation Using Anaconda Install Anaconda Conda Environment Creation Download and Unpack Apache Spark Install Java 8 or Later Mac and Linux Users Windows Users Run PySpark Jupyter Notebook Extension Docker-Based Installation Why Do We Need to Use Docker? What Is Docker? Create a Simple Docker Image Download PySpark Docker Step-by-Step Approach to Understanding the PySpark Docker Commands Databricks Community Edition Create a Databricks Account Create a New Cluster Create Notebooks Import Data Files into the Databricks Environment GitHub Codespaces Basic Operations Upload Data Access Data Calculate Pi Summary Chapter 2: PySpark Basics PySpark Background PySpark Resilient Distributed Datasets (RDDs) and DataFrames Data Manipulations Reading Data from a File Reading Data from a Hive Table Reading Metadata Counting Records Subset Columns and View a Glimpse of the Data Missing Values One-Way Frequencies Sorting and Filtering One-Way Frequencies Casting Variables Descriptive Statistics Unique/Distinct Values and Counts Filtering Creating New Columns Deleting and Renaming Columns Pandas API on Spark Summary Chapter 3: Utility Functions and Visualizations Additional Data Manipulations String Functions Registering DataFrames Window Functions Other Useful Functions Collect List Sampling Caching and Persisting Saving Data Pandas Support Joins Dropping Duplicates Data Visualizations Introduction to Machine Learning Summary Chapter 4: Variable Selection Exploratory Data Analysis Cardinality Missing Values Missing at Random (MAR) Missing Completely at Random (MCAR) Missing Not at Random (MNAR) Code 1: Cardinality Check Code 2: Missing Values Check Step 1: Identify Variable Types Step 2: Apply StringIndexer to Character Columns Step 3: Apply StringIndexer to Target Column Step 4: Assemble Features Built-in Variable Selection Process: Without Target Principal Component Analysis (PCA) Mechanics Singular Value Decomposition (SVD) Built-in Variable Selection Process: With Target ChiSq Selector Model-based Feature Selection Custom-built Variable Selection Process Information Value Using Weight of Evidence Monotonic Binning Using Spearman Correlation How Do You Calculate the Spearman Correlation by Hand? How Is Spearman Correlation Used to Create Monotonic Bins for Continuous Variables? Custom Transformers Main Concepts in Pipelines Voting-based Selection Summary Chapter 5: Supervised Learning Algorithms Machine Learning Algorithm Basics Regression Classification Loss Functions Optimizers Gradient Descent Choosing Learning Rate Local Minima Versus Global Minima Model-training Time Stochastic/Mini-batch Gradient Descent Momentum AdaGrad (Adaptive Gradient) Optimizer Root Mean Square Propagation (RMSprop) Optimizer Adaptive Moment (Adam) Optimizer Quick Recap Activation Functions Linear Activation Function Sigmoid Activation Function Hyperbolic Tangent (TanH) Function Rectified Linear Unit (ReLu) Function Leaky ReLu or Parametric ReLu Function Swish Activation Function Softmax Function Batch Normalization Dropout Supervised Machine Learning Algorithms Linear Regression Interpreting the Linear Regression Model Multicollinearity Logistic Regression Maximum Likelihood Estimation or Maximum Log-likelihood (MLE) How Do You Rotate the Line to Find the Best Fit? Binary Versus Multinomial Classification PySpark Code Interpreting the Model Results Binary Variable Interpretation Continuous Variable Interpretation Decision Trees Interpretation of the Decision Tree Entropy Information Gain Gini Impurity Variance Determine the Cut-off for Numerical Variables Pruning the Tree/Stopping Criterion PySpark Code for Decision Tree Classification PySpark Code for Decision Tree Regression Feature Importance Using Decision Trees Random Forests Hyperparameter Tuning Feature Importance Using Random Forest PySpark Code for Random Forest Classification Regression Why Random Forest? Gradient Boosting Boosting Learning Process PySpark Code for Gradient Boosting Classification Regression Why Gradient Boosting? Support Vector Machine (SVM) Classification Error Margin Error Total Error PySpark Code Neural Networks Things to Know About ANN Architecture How Does a Neural Network Fit the Data? Step 1: Feed-Forward Step 2: Backpropagation Hyperparameters Tuning in Neural Networks PySpark Code One-vs-Rest Classifier PySpark Code Naïve Bayes Classifier PySpark Code Regularization Parameters Lasso or L1 Penalty Ridge or L2 Penalty Summary Chapter 6: Model Evaluation Model Complexity Underfitting Best Fitting Overfitting Bias and Variance Model Validation Training/Test Split Simple Random Sampling Stratified Sampling Option 1 Option 2 Option 3 Sampling Bias Holdout/Out-of-Time Population Stability Index k-fold Cross-Validation Leave-One-Out Cross-Validation Leave-One-Group-Out Cross-Validation Time-series Model Validation Leakage Target Leakage Data Leakage Issues with Leakage Model Assessment Continuous Target Code to Replicate the Output Using PySpark Model Equation Error Mean Squared Error (MSE) Root Mean Squared Error (RMSE) Mean Absolute Error (MAE) R-squared (R2) Using Model Fit Using Model Variance Explained Variance (Var) Adjusted R-Squared (Adj. R2) Binary Target Code to Replicate the Output Using PySpark Model Equation y-hat Prediction to Probability to Final Prediction Confusion Matrix Accuracy Misclassification Rate Precision Recall F1-score Receiver Operating Characteristics (ROC)/Area Under the Curve (AUC) What Happens to the ROC Curve When the Classifier Is Bad? Precision Recall Curve What Happens to the PR Curve When the Classifier Is Bad? Kolmogorov Smirnov (KS) Statistic and Deciles Deciles KS Statistics Actual vs Predicted, Gains Chart, Lift Chart Actual Versus Predicted Gains Chart Lift Chart Multiclass, Multilabel Evaluation Metrics Summary Chapter 7: Unsupervised Learning and Recommendation Algorithms Segmentation Distance Measures Types of Clustering Bisecting k-Means K-means Latent Dirichlet Allocation (LDA) LDA Implementation Collaborative Filtering User-based Collaborative Filtering Item-based Collaborative Filtering Matrix Factorization Optimization Using Alternating Least Squares (ALS) Summary Chapter 8: Machine Learning Flow and Automated Pipelines MLflow MLflow Code Setup and Installation MLflow User Interface Demonstration Automated Machine Learning Pipelines Pipeline Requirements and Framework Data Manipulations Feature Selection Model Building Metrics Calculation Validation and Plot Generation Model Selection Score Code Creation Collating Results Framework Pipeline Outputs Summary Chapter 9: Deploying Machine Learning Models Starter Code Save Model Objects and Create Score Code Model Objects Score Code Model Deployment Using HDFS Object and Pickle Files Model Deployment Using Docker The requirements.txt File Dockerfile Changes Made in the helper.py and run.py Files Create Docker and Execute Score Code Real-Time Scoring API Postman API Test Real-Time Using Postman API Build UI The streamlitapi Directory real_time_scoring Directory Executing the docker-compose.yml File Real-time Scoring API Summary Chapter 10: Experimentation and Optimization Strategies Importance of Experimentation The Experimentation Process Hypothesis Testing Experimentation Case Study Why Use PySpark for Experimentation? Optimization Strategies with PySpark Random Data Generation Sampling Simple Random Sampling (SRS) Stratified Sampling Outline Placeholder Switching between Python and PySpark Curious Character of Nulls Common Function Conflicts Join Conditions User Defined Functions (UDFs) Handle the Skewness Using Cache Persist/Unpersist Shuffle Partitions Use Optimal Formats Data Serialization Accomplishments Chapter 11: Modeling Frameworks Customer Lifetime Value Understanding Customer Lifetime Value (CLV) Importance of CLV Types of CLV CLV in Different Business Models Challenges and Considerations in Implementing CLV Best Practices for Leveraging CLV Types of CLV Models Simple CLV Model Historical CLV Model Predictive CLV Models Survival Analysis Models Traditional CLV Model Accelerated Failure Time (AFT) Model Example 11.1 Example 11.2 Return on Marketing Investment (ROMI) The Basic Formula for ROMI Step-by-Step Approach to Calculating ROMI Using Data Science Models Step 1: Data Collection and Preparation Step 2: Customer Segmentation Step 3: Attribution Modeling Step 4: Predictive Modeling for Incremental Revenue Step 5: Customer Lifetime Value (CLV) Analysis Step 6: Marketing Cost Calculation Step 7: ROMI Calculation Advanced Techniques and Considerations Advanced Attribution Models Machine Learning for Predictive Modeling Real-Time Data Processing Experimentation and A/B Testing ROMI Conclusion Product Engagement Scores Importance of Product Engagement Scores Example 1: Streaming Platform Calculating the Engagement Score Step 1: Normalize Metrics Step 2: Assign Weights Step 3: Calculate Composite Score Example 2: Software Application Calculating the Engagement Score Step 1: Normalize Metrics Step 2: Assign Weights Step 3: Calculate Composite Score Careful Considerations Using Engagement Scores for Decision Making Importance of Product Engagement Scores Uplift Modeling Importance of Uplift Modeling The Theory of Uplift Modeling Basic Steps in Uplift Modeling Using the Uplift Quadrant Matrix Practical Application Example: Uplift Modeling for a Retail Business The Step-by-Step Uplift Modeling Process Step 1: Data Collection and Preparation Step 2: Feature Engineering Step 3: Model Building Step 4: Uplift Calculation Step 5: Evaluation Detailed Example Step 1: Data Collection and Preparation Step 2: Feature Engineering Step 3: Model Building Step 4: Uplift Calculation Step 5: Evaluation Example: Email Marketing Campaign Benefits of Uplift Modeling Summary Conclusion