Machine Learning with R

Annotation This book helps readers understand the mathematics of machine learning, and apply them in different situations. It is divided into two basic parts, the first of which introduces readers to the theory of linear algebra, probability, and data distributions and it's applications to machine learning. It also includes a detailed introduction to the concepts and constraints of machine learning and what is involved in designing a learning algorithm. This part helps readers understand the mathematical and statistical aspects of machine learning. In turn, the second part discusses the algorithms used in supervised and unsupervised learning. It works out each learning algorithm mathematically and encodes it in R to produce customized learning applications. In the process, it touches upon the specifics of each algorithm and the science behind its formulation. The book includes a wealth of worked-out examples along with R codes. It explains the code for each algorithm, and readers can modify the code to suit their own needs. The book will be of interest to all researchers who intend to use R for machine learning, and those who are interested in the practical aspects of implementing learning algorithms for data analysis. Further, it will be particularly useful and informative for anyone who has struggled to relate the concepts of mathematics and statistics to machine learning Preface 6 The Data-Driven Universe 6 Causality—The Cornerstone of Accountability 7 The Growth of Machines 7 What is Machine Learning? 7 Intended Audience 8 Acknowledgements 10 Contents 11 About the Author 16 1 Linear Algebra, Numerical Optimization, and Its Applications in Machine Learning 17 1.1 Scalars, Vectors, and Linear Functions 17 1.1.1 Scalars 17 1.1.2 Vectors 17 1.2 Linear Functions 20 1.3 Matrices 20 1.3.1 Transpose of a Matrix 20 1.3.2 Identity Matrix 20 1.3.3 Inverse of a Matrix 21 1.3.4 Representing Linear Equations in Matrix Form 21 1.4 Matrix Transformations 22 1.5 Norms 23 1.5.1 ell2 Optimization 24 1.5.2 ell1 Optimization 25 1.6 Rewriting the Regression Model in Matrix Notation 25 1.7 Cost of a n-Dimensional Function 26 1.8 Computing the Gradient of the Cost 27 1.8.1 Closed-Form Solution 27 1.8.2 Gradient Descent 28 1.9 An Example of Gradient Descent Optimization 29 1.10 Eigendecomposition 30 1.11 Singular Value Decomposition (SVD) 34 1.12 Principal Component Analysis (PCA) 37 1.12.1 PCA and SVD 38 1.13 Computational Errors 43 1.13.1 Rounding---Overflow and Underflow 44 1.13.2 Conditioning 44 1.14 Numerical Optimization 45 2 Probability and Distributions 47 2.1 Sources of Uncertainty 47 2.2 Random Experiment 48 2.3 Probability 48 2.3.1 Marginal Probability 49 2.3.2 Conditional Probability 50 2.3.3 The Chain Rule 50 2.4 Bayes' Rule 51 2.5 Probability Distribution 53 2.5.1 Discrete Probability Distribution 53 2.5.2 Continuous Probability Distribution 53 2.5.3 Cumulative Probability Distribution 53 2.5.4 Joint Probability Distribution 54 2.6 Measures of Central Tendency 54 2.7 Dispersion 55 2.8 Covariance and Correlation 55 2.9 Shape of a Distribution 57 2.10 Chebyshev's Inequality 57 2.11 Common Probability Distributions 58 2.11.1 Discrete Distributions 58 2.11.2 Continuous Distributions 59 2.11.3 Summary of Probability Distributions 61 2.12 Tests for Fit 62 2.12.1 Chi-Square Distribution 63 2.12.2 Chi-Square Test 64 2.13 Ratio Distributions 66 2.13.1 Student's t-Distribution 67 2.13.2 F-Distribution 70 3 Introduction to Machine Learning 73 3.1 Scientific Enquiry 74 3.1.1 Empirical Science 74 3.1.2 Theoretical Science 75 3.1.3 Computational Science 75 3.1.4 e-Science 75 3.2 Machine Learning 75 3.2.1 A Learning Task 76 3.2.2 The Performance Measure 76 3.2.3 The Experience 77 3.3 Train and Test Data 77 3.3.1 Training Error, Generalization (True) Error, and Test Error 77 3.4 Irreducible Error, Bias, and Variance 80 3.5 Bias--Variance Trade-off 82 3.6 Deriving the Expected Prediction Error 83 3.7 Underfitting and Overfitting 84 3.8 Regularization 85 3.9 Hyperparameters 87 3.10 Cross-Validation 88 3.11 Maximum Likelihood Estimation 88 3.12 Gradient Descent 91 3.13 Building a Machine Learning Algorithm 92 3.13.1 Challenges in Learning Algorithms 93 3.13.2 Curse of Dimensionality and Feature Engineering 93 3.14 Conclusion 94 4 Regression 95 4.1 Linear Regression 95 4.1.1 Hypothesis Function 95 4.1.2 Cost Function 96 4.2 Linear Regression as Ordinary Least Squares 97 4.3 Linear Regression as Maximum Likelihood 99 4.4 Gradient Descent 100 4.4.1 Gradient of RSS 100 4.4.2 Closed Form Solution 100 4.4.3 Step-by-Step Batch Gradient Descent 100 4.4.4 Writing the Batch Gradient Descent Application 101 4.4.5 Writing the Stochastic Gradient Descent Application 105 4.5 Linear Regression Assumptions 106 4.6 Summary of Regression Outputs 109 4.7 Ridge Regression 111 4.7.1 Computing the Gradient of Ridge Regression 113 4.7.2 Writing the Ridge Regression Gradient Descent Application 115 4.8 Assessing Performance 119 4.8.1 Sources of Error Revisited 120 4.8.2 Bias--Variance Trade-Off in Ridge Regression 122 4.9 Lasso Regression 123 4.9.1 Coordinate Descent for Least Squares Regression 124 4.9.2 Coordinate Descent for Lasso 125 4.9.3 Writing the Lasso Coordinate Descent Application 126 4.9.4 Implementing Coordinate Descent 128 4.9.5 Bias Variance Trade-Off in Lasso Regression 129 5 Classification 130 5.1 Linear Classifiers 130 5.1.1 Linear Classifier Model 131 5.1.2 Interpreting the Score 132 5.2 Logistic Regression 132 5.2.1 Likelihood Function 135 5.2.2 Model Selection with Log-Likelihood 135 5.2.3 Gradient Ascent to Find the Best Linear Classifier 136 5.2.4 Deriving the Log-Likelihood Function 137 5.2.5 Deriving the Gradient of Log-Likelihood 139 5.2.6 Gradient Ascent for Logistic Regression 140 5.2.7 Writing the Logistic Regression Application 140 5.2.8 A Comparison Using the BFGS Optimization Method 144 5.2.9 Regularization 146 5.2.10 \ell_2 Regularized Logistic Regression 146 5.2.11 \ell_2 Regularized Logistic Regression with Gradient Ascent 148 5.2.12 Writing the Ridge Logistic Regression with Gradient Ascent Application 148 5.2.13 Writing the Lasso Regularized Logistic Regression With Gradient Ascent Application 153 5.3 Decision Trees 158 5.3.1 Decision Tree Algorithm 160 5.3.2 Overfitting in Decision Trees 160 5.3.3 Control of Tree Parameters 161 5.3.4 Writing the Decision Tree Application 162 5.3.5 Unbalanced Data 167 5.4 Assessing Performance 168 5.4.1 Assessing Performance--Logistic Regression 170 5.5 Boosting 173 5.5.1 AdaBoost Learning Ensemble 175 5.5.2 AdaBoost: Learning from Weighted Data 175 5.5.3 AdaBoost: Updating the Weights 176 5.5.4 AdaBoost Algorithm 177 5.5.5 Writing the Weighted Decision Tree Algorithm 177 5.5.6 Writing the AdaBoost Application 183 5.5.7 Performance of our AdaBoost Algorithm 187 5.6 Other Variants 190 5.6.1 Bagging 190 5.6.2 Gradient Boosting 191 5.6.3 XGBoost 191 6 Clustering 194 6.1 The Clustering Algorithm 195 6.2 Clustering Algorithm as Coordinate Descent optimization 195 6.3 An Introduction to Text mining 196 6.3.1 Text Mining Application---Reading Multiple Text Files from Multiple Directories 196 6.3.2 Text Mining Application---Creating a Weighted tf-idf Document-Term Matrix 197 6.3.3 Text Mining Application---Exploratory Analysis 198 6.4 Writing the Clustering Application 198 6.4.1 Smart Initialization of k-means 208 6.4.2 Writing the k-means++ Application 208 6.4.3 Finding the Optimal Number of Centroids 214 6.5 Topic Modeling 216 6.5.1 Clustering and Topic Modeling 216 6.5.2 Latent Dirichlet Allocation for Topic Modeling 217 Appendix References and Further Reading 223 Front Matter ....Pages i-xix Linear Algebra, Numerical Optimization, and Its Applications in Machine Learning (Abhijit Ghatak)....Pages 1-30 Probability and Distributions (Abhijit Ghatak)....Pages 31-56 Introduction to Machine Learning (Abhijit Ghatak)....Pages 57-78 Regression (Abhijit Ghatak)....Pages 79-113 Classification (Abhijit Ghatak)....Pages 115-178 Clustering (Abhijit Ghatak)....Pages 179-207 Back Matter ....Pages 209-210