Data Science with R A Step By Step Guide With Visual Illustrations and Examples

A Step By Step Guide with Visual Illustrations and Examples The Data Science field is expected to continue growing rapidly over the next several years and Data Scientist is consistently rated as a top career.Data Science with R gives you the necessery theoretical background to start your Data Science journey and shows you how to apply the R programming language through practical examples in order to extract valuable knowledge from data. Professor Andrew Oleksy guides you through all important concepts of data science including the R programming language, Data Mining, Clustering, Classification and Prediction, Hadoop framework and more. Table of Contents 3 Chapter 1: Introduction to Data Mining 10 Summary 10 Prerequisite Knowledge 10 Introduction to Data Mining 11 1.1 Data Science 11 1.2 Knowledge Discovery in Databases (KDD) 14 1.2.1 Data Collection 14 1.2.2 Preprocessing 15 1.2.3 Transformation 15 1.2.4 Data Mining 15 1.2.5 Interpretation and Evaluation 15 1.3 Model Types 16 1.4 Examples and Counterexamples 17 1.5 Classification of Data Mining methods 18 1.5.1 Classification 18 1.5.2 Regression 18 1.5.3 Clustering 19 1.5.4 Extraction and Association Analysis 20 1.5.5 Visualization 21 1.5.6 Anomaly Detection 21 1.6 Applications 22 1.6.1 Medicine 22 1.6.2 Finance 23 1.6.3 Telecommunications 24 1.7 Challenges 26 1.8 The R Programming Language 27 1.9 Basic Concepts, Definitions and Notations 29 1.10 Tool Installation 30 Chapter 2: Introduction to R 33 Summary 33 Prerequisite Knowledge 33 Introduction to R 34 2.1 Data Types 35 2.1.1 Definition and Object Classes 35 2.1.2 Vectors and Lists 36 2.1.3 Matrix 38 2.1.4. Factors and Nominal Data 39 2.1.5 Missing Values 40 2.1.6 Data Frames 40 2.2 Basic Tasks 42 2.2.1 Reading Data from File 42 2.2.2 Sequence creation 42 2.2.3 Reference to Subsets 43 2.2.4 Vectorization 46 2.3 Control Structures 47 2.3.1 Conditional Statement: if-else 47 2.3.1 Loops: for, repeat and while 47 2.3.3 Next and break statements 49 2.4 Functions 50 2.5 Scoping Rules 52 2.6 Iterated Functions 53 2.6.1 lapply 53 2.6.2 sapply 53 2.6.3 Split 54 2.6.4 tapply 55 2.7 Help from the console and Package Installation 57 Chapter 3: Types, Quality and Data Preprocessing 58 Summary 58 Prerequisite Knowledge 58 Types, Quality and Data Preprocessing 59 3.1 Categories and Types of Variables 60 3.2 Preprocessing processes 61 3.2.1 Data cleansing 61 3.2.1.1 Missing Values 61 3.2.1.2 Data with Noise 61 Example – Data smoothing using binning methods 62 3.2.1.3 Inconsistent data 63 3.2.2 Data Unification 63 3.2.3 Data Transformation and Discretization 64 3.2.3.1 Data Transformation 64 Example – Data Regularization 65 3.2.3.2 Data Discretization 66 Example – Entropy-based discretization 67 3.2.4 Data Reduction 70 3.2.4.1 Dimension Reduction 70 3.2.4.2 Data Compression 71 3.3 dplyr and tidyr packages 74 3.3.1 dplyr 74 3.3.2 tidyr 78 Chapter 4: Summary Statistics and Visualization 83 Summary 83 Prerequisite Knowledge 83 Summary Statistics and Visualization 83 4.1 Measures of Position 84 4.1.1 Mean Value 84 4.1.2 Median 84 4.2 Measures of dispersion 86 4.2.1 Minimum value, Maximum value, Range 86 4.2.2 Percentile values 87 4.2.3 Interquartile Range 88 4.2.4 Variance 88 4.2.5 Standard Deviation 89 4.2.6 Coefficient of Variation 90 4.3 Visualization of Qualitative Data 91 4.3.1 Frequency Table 91 4.3.2 Bar Charts 92 4.3.3 Pie Chart 92 4.3.4 Contingency Matrix 93 4.3.4 Stacked Bar Charts and Grouped Bar Charts 94 4.4 Visualization of Quantitative Data 98 4.4.1 Frequency Table 98 4.4.2. Histograms 98 4.4.3 Frequency Polygon 102 4.4.4 Boxplot 103 Chapter 5: Classification and Prediction 106 Summary 106 Prerequisite Knowledge 106 5.1 Classification 107 5.1.2 Decision Trees 107 5.1.2.1 Description 107 5.1.2.2 Decision Tree creation – ID3 Algorithm 108 5.1.2.3 Decision Tree creation – Gini Index 114 5.2 Prediction 118 5.2.1Difference between Classification and Prediction 118 5.2.2 Linear Regression 118 5.2.2.1 Description, Definitions and Notations 118 5.2.2.2 Cost Function 119 5.2.2.3 Gradient Descent Algorithm 119 5.2.2.4 Gradient Descent in Linear Regression 121 5.2.2.5 Learning Parameter 122 5.3 Overfitting and regularization 125 5.3.1 Overfitting 125 5.3.2 Model Regularization 125 5.3.3 Linear Regression with Normalization 126 Chapter 6: Clustering 128 Summary 128 Prerequisite Knowledge 128 CLUSTERING 129 6.1 Unsupervised Learning 129 6.2 Concept of Cluster 130 6.3 k-means algorithm 131 6.3.1 Algorithm Description 131 6.3.2 Random Centroids Initialization 131 6.3.3 Choosing the number of Clusters 132 6.3.4 Applying k-means in R 133 6.4 Hierarchical Clustering Algorithms 136 6.4.1 Distance Measurements Between Clusters 136 6.4.2 Agglomerative Algorithms 139 6.4.3 Divisive Algorithms 139 6.4.4 Applying Hierarchical Clustering in R 139 6.5 DBSCAN Algorithm 142 6.5.1 Basic Concepts 142 6.5.2 Algorithm Description 143 6.5.3 Algorithm Complexity 144 6.5.4 Advantages 144 6.5.5 Disadvantages 145 Chapter 7: Mining of Frequent Itemsets and Association Rules 147 Summary 147 Prerequisite Knowledge 147 Mining of Frequent Itemsets and Association Rules 147 7.1 Introduction 148 7.2 Theoretical Background 150 7.3 Apriori Algorithm 152 7.4 Frequent Itemsets Types 155 7.5 Positive and Negative Border of Frequent Itemsets 156 7.6 Association Rules Mining 157 7.7 Alternative Methods for Large Itemsets generation 159 7.7.1 Sampling Algorithm 159 7.7.2 Partitioning Algorithm 160 7.8 FP-Growth Algorithm 161 7.9 Arules package 165 Chapter 8: Computational Methods for Big Data Analysis (Hadoop and MapReduce) 169 Summary 169 Prerequisite Knowledge 169 8.1 Introduction 170 8.2 Advantages of Hadoop’s Distributed File System 172 8.3 Hadoop Users 174 8.4 Hadoop Architecture 175 8.4.1 Hadoop Distributed File System (HDFS) 175 8.4.2 HDFS Architecture 175 8.4.3 HDFS – Low Performance Areas 176 8.4.3.1 Low Data Access Time 176 8.4.3.2 Multiple Small Files 176 8.5.3.3 Multiple Data Recording Nodes, Arbitrary File Modifications 176 8.4.4 Basic HDFS Concepts 177 8.4.4.1 Blocks 177 8.4.4.2 Namenodes and Datanodes 178 8.4.4.3 HDFS Federation 179 8.4.4.4 HDFS High Availability 180 8.4.5 Data Flow – Data Reading 182 8.4.6 Network Topology in Hadoop 184 8.4.7 File Writing 185 8.4.8 Copies Placement 188 8.4.9 Consistency Model 189 8.5 The Hadoop Cluster Architecture 191 8.6 Hadoop Java API 192 8.7 Lists Loops & Generic Classes and Methods 199 8.7.1 Generic Classes and Methods 199 8.7.2 The Class Object 200 Table of Contents......Page 3 Prerequisite Knowledge......Page 10 1.1 Data Science......Page 11 1.2.1 Data Collection......Page 14 1.2.5 Interpretation and Evaluation......Page 15 1.3 Model Types......Page 16 1.4 Examples and Counterexamples......Page 17 1.5.2 Regression......Page 18 1.5.3 Clustering......Page 19 1.5.4 Extraction and Association Analysis......Page 20 1.5.6 Anomaly Detection......Page 21 1.6.1 Medicine......Page 22 1.6.2 Finance......Page 23 1.6.3 Telecommunications......Page 24 1.7 Challenges......Page 26 1.8 The R Programming Language......Page 27 1.9 Basic Concepts, Definitions and Notations......Page 29 1.10 Tool Installation......Page 30 Prerequisite Knowledge......Page 33 Introduction to R......Page 34 2.1.1 Definition and Object Classes......Page 35 2.1.2 Vectors and Lists......Page 36 2.1.3 Matrix......Page 38 2.1.4. Factors and Nominal Data......Page 39 2.1.6 Data Frames......Page 40 2.2.2 Sequence creation......Page 42 2.2.3 Reference to Subsets......Page 43 2.2.4 Vectorization......Page 46 2.3.1 Loops: for, repeat and while......Page 47 2.3.3 Next and break statements......Page 49 2.4 Functions......Page 50 2.5 Scoping Rules......Page 52 2.6.2 sapply......Page 53 2.6.3 Split......Page 54 2.6.4 tapply......Page 55 2.7 Help from the console and Package Installation......Page 57 Prerequisite Knowledge......Page 58 Types, Quality and Data Preprocessing......Page 59 3.1 Categories and Types of Variables......Page 60 3.2.1.2 Data with Noise......Page 61 Example – Data smoothing using binning methods......Page 62 3.2.2 Data Unification......Page 63 3.2.3.1 Data Transformation......Page 64 Example – Data Regularization......Page 65 3.2.3.2 Data Discretization......Page 66 Example – Entropy-based discretization......Page 67 3.2.4.1 Dimension Reduction......Page 70 3.2.4.2 Data Compression......Page 71 3.3.1 dplyr......Page 74 3.3.2 tidyr......Page 78 Summary Statistics and Visualization......Page 83 4.1.2 Median......Page 84 4.2.1 Minimum value, Maximum value, Range......Page 86 4.2.2 Percentile values......Page 87 4.2.4 Variance......Page 88 4.2.5 Standard Deviation......Page 89 4.2.6 Coefficient of Variation......Page 90 4.3.1 Frequency Table......Page 91 4.3.3 Pie Chart......Page 92 4.3.4 Contingency Matrix......Page 93 4.3.4 Stacked Bar Charts and Grouped Bar Charts......Page 94 4.4.2. Histograms......Page 98 4.4.3 Frequency Polygon......Page 102 4.4.4 Boxplot......Page 103 Prerequisite Knowledge......Page 106 5.1.2.1 Description......Page 107 5.1.2.2 Decision Tree creation – ID3 Algorithm......Page 108 5.1.2.3 Decision Tree creation – Gini Index......Page 114 5.2.2.1 Description, Definitions and Notations......Page 118 5.2.2.3 Gradient Descent Algorithm......Page 119 5.2.2.4 Gradient Descent in Linear Regression......Page 121 5.2.2.5 Learning Parameter......Page 122 5.3.2 Model Regularization......Page 125 5.3.3 Linear Regression with Normalization......Page 126 Prerequisite Knowledge......Page 128 6.1 Unsupervised Learning......Page 129 6.2 Concept of Cluster......Page 130 6.3.2 Random Centroids Initialization......Page 131 6.3.3 Choosing the number of Clusters......Page 132 6.3.4 Applying k-means in R......Page 133 6.4.1 Distance Measurements Between Clusters......Page 136 6.4.4 Applying Hierarchical Clustering in R......Page 139 6.5.1 Basic Concepts......Page 142 6.5.2 Algorithm Description......Page 143 6.5.4 Advantages......Page 144 6.5.5 Disadvantages......Page 145 Mining of Frequent Itemsets and Association Rules......Page 147 7.1 Introduction......Page 148 7.2 Theoretical Background......Page 150 7.3 Apriori Algorithm......Page 152 7.4 Frequent Itemsets Types......Page 155 7.5 Positive and Negative Border of Frequent Itemsets......Page 156 7.6 Association Rules Mining......Page 157 7.7.1 Sampling Algorithm......Page 159 7.7.2 Partitioning Algorithm......Page 160 7.8 FP-Growth Algorithm......Page 161 7.9 Arules package......Page 165 Prerequisite Knowledge......Page 169 8.1 Introduction......Page 170 8.2 Advantages of Hadoop’s Distributed File System......Page 172 8.3 Hadoop Users......Page 174 8.4.2 HDFS Architecture......Page 175 8.5.3.3 Multiple Data Recording Nodes, Arbitrary File Modifications......Page 176 8.4.4.1 Blocks......Page 177 8.4.4.2 Namenodes and Datanodes......Page 178 8.4.4.3 HDFS Federation......Page 179 8.4.4.4 HDFS High Availability......Page 180 8.4.5 Data Flow – Data Reading......Page 182 8.4.6 Network Topology in Hadoop......Page 184 8.4.7 File Writing......Page 185 8.4.8 Copies Placement......Page 188 8.4.9 Consistency Model......Page 189 8.5 The Hadoop Cluster Architecture......Page 191 8.6 Hadoop Java API......Page 192 8.7.1 Generic Classes and Methods......Page 199 8.7.2 The Class Object......Page 200