Next-Generation Machine Learning with Spark : Covers XGBoost, LightGBM, Spark NLP, Distributed Deep Learning with Keras, and More

Access real-world documentation and examples for the Spark platform for building large-scale, enterprise-grade machine learning applications. The past decade has seen an astonishing series of advances in machine learning. These breakthroughs are disrupting our everyday life and making an impact across every industry. Next-Generation Machine Learning with Spark provides a gentle introduction to Spark and Spark MLlib and advances to more powerful, third-party machine learning algorithms and libraries beyond what is available in the standard Spark MLlib library. By the end of this book, you will be able to apply your knowledge to real-world use cases through dozens of practical examples and insightful explanations. What You Will Learn Be introduced to machine learning, Spark, and Spark MLlib 2.4.x Achieve lightning-fast gradient boosting on Spark with the XGBoost4J-Spark and LightGBM libraries Detect anomalies with the Isolation Forest algorithm for Spark Use the Spark NLP and Stanford CoreNLP libraries that support multiple languages Optimize your ML workload with the Alluxio in-memory data accelerator for Spark Use GraphX and GraphFrames for Graph Analysis Perform image recognition using convolutional neural networks Utilize the Keras framework and distributed deep learning libraries with Spark Who This Book Is For Data scientists and machine learning engineers who want to take their knowledge to the next level and use Spark and more powerful, next-generation algorithms and libraries beyond what is available in the standard Spark MLlib library; also serves as a primer for aspiring data scientists and engineers who need an introduction to machine learning, Spark, and Spark MLlib. Table of Contents About the Author About the Technical Reviewer Acknowledgments Introduction Chapter 1: Introduction to Machine Learning AI and Machine Learning Use Cases Retail Transportation Financial Services Healthcare and Biotechnology Manufacturing Government Machine Learning and Data Observations Features Class Labels Models Machine Learning Methods Supervised Learning Classification Binary Classification Multiclass Classification Multilabel Classification Classification and Regression Algorithms Support Vector Machine Logistic Regression Naïve Bayes Multilayer Perceptron Decision Trees Random Forest Gradient-Boosted Trees XGBoost LightGBM Regression Linear Regression Survival Regression Unsupervised Learning Clustering K-Means Topic Modeling Latent Dirichlet Allocation Anomaly Detection Isolation Forest One-Class Support Vector Machines Dimensionality Reduction Principal Component Analysis Recommendations Semi-supervised Learning Reinforcement Learning Deep Learning Neural Networks Convolutional Neural Networks Feature Engineering Feature Selection Filter Methods Wrapper Methods Embedded Methods Feature Importance Feature Extraction Feature Construction Model Evaluation Accuracy Precision Recall F1 Measure Area Under the Receiver Operating Characteristic (AUROC) Overfitting and Underfitting Model Selection Summary References Chapter 2: Introduction to Spark and Spark MLlib Overview Cluster Managers Architecture Executing Spark Applications Cluster Mode Client Mode Introduction to the spark-shell SparkSession Resilient Distributed Dataset (RDD) Creating an RDD parallelize textFile Transformations Map FlatMap Filter Distinct ReduceByKey Keys Values Inner Join RightOuterJoin and LeftOuterJoin Union Subtract Coalesce Repartition Actions Collect Count Take Foreach Lazy Evaluation Caching Accumulator Broadcast Variable Spark SQL, Dataset, and DataFrames API Spark Data Sources CSV XML JSON Relational and MPP Databases Parquet HBase Amazon S3 Solr Microsoft Excel Secure FTP Introduction to Spark MLlib Spark MLlib Algorithms ML Pipelines Pipeline Transformer Estimator ParamGridBuilder CrossValidator Evaluator Feature Extraction, Transformation, and Selection StringIndexer Tokenizer VectorAssembler StandardScaler StopWordsRemover n-gram OneHotEncoderEstimator SQLTransformer Term Frequency–Inverse Document Frequency (TF–IDF) Principal Component Analysis (PCA) ChiSqSelector Correlation Evaluation Metrics Area Under the Receiver Operating Characteristic (AUROC) F1 Measure Root Mean Squared Error (RMSE) Model Persistence Spark MLlib Example Graph Processing Beyond Spark MLlib: Third-Party Machine Learning Integrations Optimizing Spark and Spark MLlib with Alluxio Architecture Why Use Alluxio? Significantly Improves Big Data Processing Performance and Scalability Multiple Frameworks and Applications Can Share Data at Memory Speed Provides High Availability and Persistence in Case of Application Termination or Failure Optimizes Overall Memory Usage and Minimizes Garbage Collection Reduces Hardware Requirements Apache Spark and Alluxio Summary References Chapter 3: Supervised Learning Classification Binary Classification Multiclass Classification Multilabel Classification Spark MLlib Classification Algorithms Logistic Regression Support Vector Machine Naïve Bayes Multilayer Perceptron Decision Trees Random Forest Gradient-Boosted Trees Random Forests vs. Gradient-Boosted Trees Third-Party Classification and Regression Algorithms Multiclass Classification with Logistic Regression Example Churn Prediction with Random Forest Parameters Example Feature Importance eXtreme Gradient Boosting with XGBoost4J-Spark Parameters Example LightGBM: Fast Gradient Boosting from Microsoft Parameters Example Sentiment Analysis with Naïve Bayes Example Regression Simple Linear Regression Example Multiple Regression with XGBoost4J-Spark Example Multiple Regression with LightGBM Summary References Chapter 4: Unsupervised Learning Clustering with K-Means Example Topic Modeling with Latent Dirichlet Allocation (LDA) Stanford CoreNLP for Spark Spark NLP from John Snow Labs Pre-trained Pipelines Pre-trained Pipelines with Spark DataFrames Pre-trained Pipelines with Spark MLlib Pipelines Creating Your Own Spark MLlib Pipeline Spark NLP LightPipeline Spark NLP OCR Module Example Anomaly Detection with Isolation Forest Parameters Example Dimensionality Reduction with Principal Component Analysis Example Summary References Chapter 5: Recommendations Types of Recommendation Engines Collaborative Filtering with Alternating Least Squares Parameters Example Market Basket Analysis with FP-Growth Example Content-Based Filtering Summary References Chapter 6: Graph Analysis Introduction to Graphs Undirected Graph Directed Graph Directed Multigraph Property Graph Graph Analysis Use Cases Fraud Detection and Anti-Money Laundering (AML) Data Governance and Regulatory Compliance Risk Management Transportation Social Networking Network Infrastructure Management Introduction to GraphX Graph VertexRDD Edge EdgeRDD EdgeTriplet EdgeContext GraphX Example Graph Algorithms PageRank Dynamic PageRank Static PageRank TriangleCount ConnectedComponents GraphFrames Summary References Chapter 7: Deep Learning Neural Networks A Brief History of Neural Networks Convolutional Neural Networks Convolutional Neural Network Architecture Feature Detection Layers Convolutional Layer Rectified Linear Unit (ReLU) Activation Function Pooling Layer Classification Layers Flatten Layer Fully Connected (Dense) Layer Dropout Layer Softmax and Sigmoid Functions Deep Learning Frameworks TensorFlow Theano PyTorch DeepLearning4J CNTK Keras Deep Learning with Keras Multiclass Classification Using the Iris Dataset Handwritten Digit Recognition with MNIST Distributed Deep Learning with Spark Model Parallelism vs. Data Parallelism Distributed Deep Learning Frameworks for Spark Deep Learning Pipelines BigDL CaffeOnSpark TensorFlowOnSpark TensorFrames Elephas Distributed Keras Elephas: Distributed Deep Learning with Keras and Spark Handwritten Digit Recognition with MNIST Using Elephas with Keras and Spark Distributed Keras (Dist-Keras) Handwritten Digit Recognition with MNIST Using Dist-Keras with Keras and Spark Dogs and Cats Image Classification Summary References Index