Distributed Machine Learning with PySpark: Migrating Effortlessly from Pandas and Scikit-Learn

Migrate from Pandas and Scikit-learn to PySpark to handle vast amounts of data and achieve faster data processing time. This book will show you how to make this transition by adapting your skills and leveraging the similarities in syntax, functionality, and interoperability between these tools. Distributed Machine Learning with PySpark offers a roadmap to data scientists considering transitioning from small data libraries (pandas/scikit-learn) to big data processing and Machine Learning with PySpark. You will learn to translate Python code from Pandas/Scikit-learn to PySpark to preprocess large volumes of data and build, train, test, and evaluate popular Machine Learning algorithms such as linear and logistic regression, decision trees, random forests, support vector machines, Naïve Bayes, and neural networks. After completing this book, you will understand the foundational concepts of data preparation and machine learning and will have the skills necessary to apply these methods using PySpark, the industry standard for building scalable ML data pipelines. What You Will Learn: Master the fundamentals of supervised learning, unsupervised learning, NLP, and recommender systems Understand the differences between PySpark, scikit-learn, and pandas Perform linear regression, logistic regression, and decision tree regression with pandas, scikit-learn, and PySpark Distinguish between the pipelines of PySpark and scikit-learn Who This Book Is For: Data scientists, data engineers, and machine learning practitioners who have some familiarity with Python, but who are new to distributed machine learning and the PySpark framework. Table of Contents About the Author About the Technical Reviewer Acknowledgments Introduction Chapter 1: An Easy Transition PySpark and Pandas Integration Similarity in Syntax Loading Data Selecting Columns Aggregating Data Filtering Data Joining Data Saving Data Modeling Steps Pipelines Summary Chapter 2: Selecting Algorithms The Dataset Selecting Algorithms with Cross-Validation Scikit-Learn PySpark Bringing It All Together Scikit-Learn PySpark Summary Chapter 3: Multiple Linear Regression with Pandas, Scikit-Learn, and PySpark The Dataset Multiple Linear Regression Multiple Linear Regression with Scikit-Learn Multiple Linear Regression with PySpark Summary Chapter 4: Decision Tree Regression with Pandas, Scikit-Learn, and PySpark The Dataset Decision Tree Regression Decision Tree Regression with Scikit-Learn The Modeling Steps Decision Tree Regression with PySpark The Modeling Steps Bringing It All Together Scikit-Learn PySpark Summary Chapter 5: Random Forest Regression with Pandas, Scikit-Learn, and PySpark The Dataset Random Forest Regression Random Forest with Scikit-Learn Random Forest with PySpark Bringing It All Together Scikit-Learn PySpark Summary Chapter 6: Gradient-Boosted Tree Regression with Pandas, Scikit-Learn, and PySpark The Dataset Gradient-Boosted Tree (GBT) Regression GBT with Scikit-Learn GBT with PySpark Bringing It All Together Scikit-Learn PySpark Summary Chapter 7: Logistic Regression with Pandas, Scikit-Learn, and PySpark The Dataset Logistic Regression Logistic Regression with Scikit-Learn Logistic Regression with PySpark Putting It All Together Scikit-Learn PySpark Summary Chapter 8: Decision Tree Classification with Pandas, Scikit-Learn, and PySpark The Dataset Decision Tree Classification Scikit-Learn and PySpark Similarities Decision Tree Classification with Scikit-Learn Decision Tree Classification with PySpark Bringing It All Together Scikit-Learn PySpark Summary Chapter 9: Random Forest Classification with Scikit- Learn and PySpark Random Forest Classification Scikit-Learn and PySpark Similarities for Random Forests Random Forests with Scikit-Learn Random Forests with PySpark Bringing It All Together Scikit-Learn PySpark Summary Chapter 10: Support Vector Machine Classification with Pandas, Scikit-Learn, and PySpark The Dataset Support Vector Machine Classification Linear SVM with Scikit-Learn Linear SVM with PySpark Bringing It All Together Scikit-Learn PySpark Summary Chapter 11: Naive Bayes Classification with Pandas, Scikit-Learn, and PySpark The Dataset Naive Bayes Classification Naive Bayes with Scikit-Learn Naive Bayes with PySpark Bringing It All Together Scikit-Learn PySpark Summary Chapter 12: Neural Network Classification with Pandas, Scikit-Learn, and PySpark The Dataset MLP Classification MLP Classification with Scikit-Learn MLP Classification with PySpark Bringing It All Together Scikit-Learn PySpark Summary Chapter 13: Recommender Systems with Pandas, Surprise, and PySpark The Dataset Building a Recommender System Recommender System with Surprise Recommender System with PySpark Bringing It All Together Surprise PySpark Summary Chapter 14: Natural Language Processing with Pandas, Scikit-Learn, and PySpark The Dataset Cleaning, Tokenization, and Vectorization Naive Bayes Classification Naive Bayes with Scikit-Learn Naive Bayes with PySpark Bringing It All Together Scikit-Learn PySpark Summary Chapter 15: k-Means Clustering with Pandas, Scikit-Learn, and PySpark The Dataset Machine Learning with k-Means k-Means Clustering with Scikit-Learn k-Means Clustering with PySpark Bringing It All Together Scikit-Learn PySpark Summary Chapter 16: Hyperparameter Tuning with Scikit-Learn and PySpark Examples of Hyperparameters Tuning the Parameters of a Random Forest Hyperparameter Tuning in Scikit-Learn Hyperparameter Tuning in PySpark Bringing It All Together Scikit-Learn PySpark Summary Chapter 17: Pipelines with Scikit- Learn and PySpark The Significance of Pipelines Pipelines with Scikit-Learn Pipelines with PySpark Bringing It All Together Scikit-Learn PySpark Summary Chapter 18: Deploying Models in Production with Scikit- Learn and PySpark Steps in Model Deployment Deploying a Multilayer Perceptron (MLP) Deployment with Scikit-Learn PySpark Bringing It All Together Scikit-Learn PySpark Summary Index