راهنمای بقا در علم داده

This book has been a work in progress since 1995, just as Data Science continues to develop and expand into our lives. Every section (eventually) will begin with a date that indicates the currency of the section—when it was last reviewed and/or updated. Since beginning the survival guide books in 1995 they have grown in all kinds of directions. My original aim was to capture useful notes for the varied and many common tasks we find ourselves doing as data scientists (or data miners back then). I structured the book as one page nuggets of information. That is, each section within a chapter targeted a single printed page, and focused on a single task. This was the origin of my OnePageR Desktop Survival Guides. It seems to have worked well over the years, from my personal use and your feedback. This material has also lead to the publication of two books. Readers are invited to send corrections, comments, suggestions, and updates to me at Graham.Williams@togaware.com. Your feedback is most welcome and will be acknowledged within the book. A pdf version of this book is available for a small financial donation which goes towards supporting the development and availability of the book. Please visit Togaware for the details. The html version contains the same material and remains freely available from Togaware https://survivor.togaware.com/datascience/ Data Science Desktop Survival Guide Data Science Desktop Survival Guide Preface About this Book Organisation of the Book Acknowledgements Waiver and Copyright 1 Data Science 1.1 Data Wrangling 1.2 Data Scientist 1.3 Data Science Through Doing 2 Introducing R 2.1 Tooling For R Programming 2.2 Introducing RStudio 2.3 Editing Code 2.4 Executing Code 2.5 RStudio Review 2.6 Packages and Libraries 2.7 A Glimpse of the Dataset 2.8 Attaching a Package 2.9 Simplifying Commands 2.10 Working with the Library 2.11 Getting Help 3 R Constructs 3.1 A Data Frame as a Dataset 3.2 Assignment 3.3 Case Statement 3.4 Commands 3.5 Functions 3.6 Operators 3.7 Pipe Operator 3.8 Pipeline 3.9 Pipeline Construction 3.10 Pipeline Identity Operator 3.11 Pipeline Syntactic Sugar 3.12 Pipes: Assignment Pipe 3.13 Pipes: Exposition Pipe 3.14 Pipes: Tee Pipe 3.15 Pipes: Tee Pipe Intermediate Results 3.16 Pipes: Tee Pipe Load CSV Files 3.17 Pipes and Plots 3.18 Variables 4 R Tasks 4.1 Create Directory 4.2 File Exists 4.3 Format Date 5 Dates 5.1 Dates Setup 5.2 Extract Year, Month and Day 5.3 Format Dates 6 Strings 6.1 Strings Setup 6.2 Case Conversion 6.3 Concatenate Strings 6.4 Concatenate NULL 6.5 Glue Strings Together 6.6 Glue Pipelines 6.7 Last Character of a String 6.8 Length of String 6.9 Random Strings 6.10 Regexp Pattern Matching 6.11 Regexp Quantifiers 6.12 Regexp Character Classes 6.13 Sub Strings in Base R 6.14 Sub Strings in the Tidyverse 6.15 Substitute Strings 6.16 Trim and Pad 6.17 Wrapping and Words 7 R Read, Write, and Create 7.1 Data Creation Setup 7.2 Clipboard Data 7.3 CSV Data 7.4 Excel Data Read 7.5 Excel Data Write 7.6 MATLAB Data 7.7 Random Dataset 7.8 Read Strings from a File 7.9 TSV Data 8 Data Template 8.1 Dataset Setup 8.2 Data Glimpse 8.3 Normalise Variable Names 8.4 Variables and Model Target 8.5 Identify Numeric Variables 8.6 Normalise Factor Levels 8.7 Modelling Roles 8.8 Variable Types 8.9 Formula to Describe the Goal 8.10 Missing Values 8.11 Random Seed 8.12 Train, Tune, and Test Datasets 8.13 Review the Dataset 8.14 Data Template 9 Data Exploration 9.1 Exploration Setup 9.2 Counting Groups 9.3 Random Sample 9.4 Constant Variables 9.5 Correlated Numeric Variables 9.6 Selecting Columns 9.7 Selecting Rows 9.8 Shuffle Rows 10 Data Wrangling 10.1 Wrangling Setup 10.2 Wrangling Data Review 10.3 Add Columns 10.4 Add Columns Using Variables 10.5 Add Counts 10.6 Combine Rows 10.7 Counting Groups 10.8 Dollar to Numeric Conversion 10.9 Extract Column as Vector 10.10 Filter Rows Having Missing Values 10.11 Missing Value Imputation 10.12 Modify Columns 10.13 Normalise Variables 10.14 Pivot Pairwise Binary Table 10.15 Rename Variables 10.16 Replacing Missing Values 10.17 Subset of Rows Within Groups 10.18 Data Source 10.19 Data Ingestion 10.20 The Shape of the Dataset 10.21 A Glimpse of the Dataset 10.22 Introducing Template Variables 10.23 Locating Datasets in Memory 10.24 Changing Datasets in Memory 10.25 Reviewing Variable Names 10.26 Effect on Data Storage 10.27 Special Case Variable Name Transformations 10.28 Data Review 10.29 Dataset Head and Tail 10.30 Random Observations 10.31 Characters 10.32 Factors 10.33 Location 10.34 Evaporation and Sunshine 10.35 Wind Directions 10.36 Ordered Factor 10.37 Rain 10.38 Numeric 10.39 Logical 10.40 Variable Roles 10.41 Risk Variable 10.42 ID Variables 10.43 Ignore IDs and Outputs 10.44 Ignore Missing 10.45 Ignore Excessive Level Variables 10.46 Dealing with Correlations 10.47 Removing Ignored Variables 10.48 Feature Selection 10.49 Missing Targets 10.50 Missing Values 10.51 Omitting Observations 10.52 Normalise Factors 10.53 Target as a Factor 10.54 Identify Variable Types 10.55 Identify Numeric and Categoric Variables 10.56 Save the Dataset 10.57 A Template for Data Preparation 11 Data Visualisation 11.1 Visualisation Setup 11.2 Visualisation Data 11.3 Visualisation Data Review 11.4 Bar Chart Basic 11.5 Bar Chart Colour No Legend 11.6 Bar Chart Dodge 11.7 Bar Chart Dodge Labelled Colour Brewer 11.8 Bar Chart Faceted Background 11.9 Bar Chart Flipped Colour Mean no Legend 11.10 Bar Chart Flipped Colour Mean Confidence Intervals 11.11 Bar Chart Flipped Sorted Axes 11.12 Bar Chart Flipped Text Annotations 11.13 Bar Chart Flipped Text Annotations Commas 11.14 Bar Chart Labels 11.15 Bar Chart Narrow Bars Economist Theme 11.16 Bar Chart Ordered X Axis 11.17 Bar Chart Stacked 11.18 Bar Chart Supplied Values 11.19 Bar Chart Texts 11.20 Bar Chart Wide Bars 11.21 Bar Chart Wide and Borders 11.22 Bar Chart Showcase Solar 11.23 Box Plot Distributions 11.24 Colour Names 11.25 Colour Ranges 11.26 Faceted Location Scatter Plot 11.27 Faceted Location Thin Lines 11.28 Faceted Wind Direction 11.29 Labels 11.30 Labels with Comma 11.31 Labels with Dollars 11.32 Labels Removed 11.33 Labels Rotated 11.34 Line Chart Basic 11.35 Line Chart Density Distribution 11.36 Line Chart Skewed Distributions 11.37 Line Chart Log X Axis 11.38 Line Chart Log Breaks 11.39 Line Chart Log Ticks 11.40 Line Chart Log Custom Labels 11.41 Pie Chart 11.42 Plotting Regions 11.43 Rose Chart 11.44 Rose Chart Discussion 11.45 Save Plot to File 11.46 Scatter Plot 11.47 Scatter Plot Colour 11.48 Scatter Plot Colour Alternative 11.49 Scatter Plot Smooth Gam 11.50 Scatter Plot Smooth Loess 11.51 Transparent Plots 11.52 Violin Plot 11.53 Violin Plot Embedded Box Plot 11.54 Violin Plot Faceted Location 11.55 XFig Support 12 Statistics 12.1 Analysis of Variance ANOVA 13 Spatial Data and Maps 13.1 Geocodes 13.2 Understanding Spatial Data 13.3 Plotting Shapefiles 13.4 Google Maps: Geocoding 14 Model Template 14.1 ML Setup 14.2 ML Data and Variables 14.3 ML Modelling Setup 14.4 ML Data Glimpse 14.5 Model Building 14.6 Predict Class 14.7 Predict Probability 14.8 Evaluating the Model 14.9 Accuracy and Error Rate 14.10 Confusion Matrix 14.11 ROC Chart 14.12 Risk Chart 14.13 Biased Estimate from the Training Dataset 14.14 Step 8: Save the Model to File 14.15 Boilerplate 14.16 Command Summary 14.17 Model Template Further Reading 14.18 Model Template Example 15 ML Scenarios 15.1 Machine Learning Setup 15.2 Reinforcement Learning 15.3 Supervised Learning 15.4 Unsupervised Learning 16 ML Activities 16.1 Classification 16.2 Cluster Analysis 16.3 Outlier Detection 16.4 Prediction 16.5 Text Mining 17 ML Applications 17.1 Airport Gate Assignment 17.2 Article Summarisation 17.3 Facial Recognition 17.4 Gene Detection 17.5 Group Recommendations 17.6 Program Comprehension 17.7 Pull Request Generation 17.8 Reservoir Inflow 17.9 Sentiment Analysis 17.10 Topic Modelling 18 ML Algorithms 18.1 Algorithms Setup 18.2 Algorithms Data and Variables 18.3 Algorithms Data Review 18.4 Collaborative Filtering 18.5 Convolutional Neural Network CNN 18.6 Decision Trees 18.7 Deep Convolutional Generative Adversarial Network 18.8 Generative Adversarial Network 18.9 K Means Clustering 18.10 K Nearest Neighbours 18.11 Linear Regression 18.12 Logistic Regression 18.13 Long Short-Term Memory Neural Networks LSTM 18.14 Multi Layer Perceptron 18.15 Neural Networks 18.16 Naive Bayes 18.17 One-Class Support Vector Machine 18.18 Recurrent Neural Networks 18.19 Residual Neural Network 18.20 Support Vector Machine 19 Cluster Analysis 19.1 Clustering Setup 19.2 Biclustering 19.3 References 20 Decision Trees 20.1 Decision Trees Setup 20.2 Decision Trees Modelling Setup 20.3 Rattle Startup 20.4 Rattle Weather Dataset 20.5 Rattle Summary of Dataset 20.6 Rattle Model Tab 20.7 Rattle Build Tree 20.8 Interpret RPart Decision Tree 20.9 Rattle View Decision Tree 20.10 Rattle Error Matrix 20.11 Rattle Risk Chart 20.12 Rattle ROC Curve 20.13 Rattle Hand Plots 20.14 Rattle Score Dataset 20.15 Rattle Log 20.16 Rattle GUI to R 20.17 Build a Decision Tree Model 20.18 Summary of the Model 20.19 Complexity Parameter 20.20 Complexity Parameter Plot 20.21 Complexity Parameter Behaviour 20.22 Complexity Parameter 0 20.23 Complexity Parameter Table 20.24 Variable Importance 20.25 Node Details and Surrogates 20.26 Decision Tree Performance 20.27 Rules from Decision Tree 20.28 Rules Using Rpart Plot 20.29 Plot Decision Trees 20.30 Plot Decision Tree Uniformly 20.31 Plot Decision Tree with Extra Information 20.32 Fancy Rpart Plot 20.33 RPart Plot Default Tree 20.34 RPart Plot Favourite 20.35 Enhanced Plot: With Colour 20.36 Enhanced Plots: Label all Nodes 20.37 Enhanced Plots: Labels Below Nodes 20.38 Enhanced Plots: Split Labels 20.39 Enhanced Plots: Interior Labels 20.40 Enhanced Plots: Number of Observations 20.41 Enhanced Plots: Add Percentage of Observations 20.42 Enhanced Plots: Classification Rate 20.43 Enhanced Plots: Add Percentage of Observations 20.44 Enhanced Plots: Misclassification Rate 20.45 Enhanced Plots: Probability per Class 20.46 Enhanced Plots: Add Percentage Observations 20.47 Enhanced Plots: Only Probability Per Class 20.48 Enhanced Plots: Probability of Second Class 20.49 Enhanced Plots: Add Percentage Observations 20.50 Enhanced Plots: Only Probability of Second Class 20.51 Enhanced Plots: Probability of the Class 20.52 Enhanced Plots: Overall Probability 20.53 Enhanced Plots: Percentage of Observations 20.54 Enhanced Plots: Show the Node Numbers 20.55 Enhanced Plots: Show the Node Indicies 20.56 Enhanced Plots: Line up the Leaves 20.57 Enhanced Plots: Angle Branch Lines 20.58 Enhanced Plots: Do Not Abbreviate Factors 20.59 Enhanced Plots: Add a Shadow to the Nodes 20.60 Enhanced Plots: Draw Branches as Dotted Lines 20.61 Enhanced Plots: Other Options 20.62 Party Tree 20.63 Conditional Decision Tree 20.64 Conditional Decision Tree Performance 20.65 CTree Plot 20.66 Weka Decision Tree 20.67 Weka Decision Tree Performance 20.68 Weka Decision Tree Plot Using Party 20.69 The Original C5.0 Implementation 20.70 C5.0 Summary 20.71 C5.0 Decision Tree Performance 20.72 C5.0 Rules Model 20.73 C5.0 Rules Summary 20.74 C5.0 Rules Performance 20.75 Regression Trees 20.76 Visualise Regression Trees 20.77 Visualise Regression Trees: Uniform 20.78 Visualise Regression Trees: Extra Information 20.79 Fancy Plot of Regression Tree 20.80 Enhanced Plot of Regression Tree: Default 20.81 Enhanced Plot of Regression Tree: Favourite 20.82 Party Regression Tree 20.83 Conditional Regression Tree 20.84 CTree Plot 20.85 Weka Regression Tree 21 Text Mining 21.1 Test Mining Setup 21.2 Corpus as a Data Set 21.3 Corpus Sources and Readers 21.4 Text Documents 21.5 PDF Documents 21.6 Word Documents 21.7 Exploring the Corpus 21.8 Preparing the Corpus 21.9 Simple Transforms 21.10 Conversion to Lower Case 21.11 Remove Numbers 21.12 Remove Punctuation 21.13 Remove English Stop Words 21.14 Remove Own Stop Words 21.15 Strip Whitespace 21.16 Specific Transformations 21.17 Stemming 21.18 Creating a Document Term Matrix 21.19 Exploring the Document Term Matrix 21.20 Distribution of Term Frequencies 21.21 Conversion to Matrix and Save to CSV 21.22 Removing Sparse Terms 21.23 Identifying Frequent Items and Associations 21.24 Correlations Plots 21.25 Correlations Plot—Options 21.26 Plotting Word Frequencies 21.27 Word Clouds 21.28 Reducing Clutter With Max Words 21.29 Reducing Clutter With Min Freq 21.30 Adding Some Colour 21.31 Varying the Scaling 21.32 Rotating Words 21.33 Quantitative Analysis of Text 21.34 Word Length Counts 21.35 Letter Frequency 21.36 Letter and Position Heatmap 21.37 Miscellaneous Functions 21.38 Word Distances 21.39 Review Preparing the Corpus 21.40 Review Analysing the Corpus 21.41 LDA 21.42 Further Reading and Acknowledgements 22 Computer Vision 22.1 Resnet Models 22.2 Document Classification 22.3 Face Recognition 22.4 OCR Optical Character Recognition 23 Natural Language 23.1 Natural Language Processing 23.2 Natural Language Understanding 24 Planning 25 Privacy 25.1 Cloud Privacy 25.2 Computer Vision Privacy 25.3 Differential Privacy 26 Literate Data Science 26.1 KnitR Setup 26.2 Basic LaTeX Template 26.3 RStudio with KnitR 26.4 Template for a Narrative 26.5 RStudio Compile PDF 26.6 Compiled PDF 26.7 SweaveOpts Undefined 26.8 Including R Commands 26.9 KnitR Basic Example 26.10 Inline R Code 26.11 Formatting Tables Using Kable 26.12 Formatting Options 26.13 Improvements Using BookTabs 26.14 Formatting Tables Using XTable 26.15 Formatting Numbers with XTable 26.16 Adding a Caption and Reference Label 26.17 Sophisticated Captions 26.18 Including Figures 26.19 Sample Figure 26.20 Adjusting Aspect 26.21 Choosing Dimensions 26.22 Setting Output Width 26.23 Add a Caption and Label 26.24 Animation: Basic Example 26.25 Adding a Flowchart 26.26 Adding Bibliographies 26.27 Referencing Chunks in LaTeX 26.28 Truncating Long Lines 26.29 Truncating Too Many Lines 26.30 Selective Lines of Code 26.31 Knitr Options 26.32 Knitr Resources 27 Coding with Style 27.1 Style Matters 27.2 Naming Files 27.3 Multiple File Scripts 27.4 Naming Objects 27.5 Naming Functions 27.6 Comments 27.7 Layout 27.8 If-Else Issue 27.9 Indentation 27.10 Alignment 27.11 Sub-Block Alignment 27.12 Function Guidelines 27.13 Function Definition Layout 27.14 Function Call Layout 27.15 Functions from Packages 27.16 Assignment 27.17 Miscellaneous 27.18 Good Practise 27.19 Kuhn Checklist 27.20 Style Resources References