R for Everyone: Advanced Analytics and Graphics (2nd Edition) (Addison-Wesley Data & Analytics Series)

Using the open source R language, you can build powerful statistical models to answer many of your most challenging questions. R has traditionally been difficult for non-statisticians to learn, and most R books assume far too much knowledge to be of help. R for Everyone, Second Edition, is the solution. Drawing on his unsurpassed experience teaching new users, professional data scientist Jared P. Lander has written the perfect tutorial for anyone new to statistical programming and modeling. Organized to make learning easy and intuitive, this guide focuses on the 20 percent of R functionality you'll need to accomplish 80 percent of modern data tasks. Lander's self-contained chapters start with the absolute basics, offering extensive hands-on practice and sample code. You'll download and install R; navigate and use the R environment; master basic program control, data import, manipulation, and visualization; and walk through several essential tests. Then, building on this foundation, you'll construct several complete models, both linear and nonlinear, and use some data mining techniques. After all this you'll make your code reproducible with LaTeX, RMarkdown, and Shiny. By the time you're done, you won't just know how to write R programs, you'll be ready to tackle the statistical problems you care about most. Coverage includes: - Explore R, RStudio, and R packages - Use R for math: variable types, vectors, calling functions, and more - Exploit data structures, including data.frames, matrices, and lists - Read many different types of data - Create attractive, intuitive statistical graphics - Write user-defined functions - Control program flow with if, ifelse, and complex checks - Improve program efficiency with group manipulations - Combine and reshape multiple datasets - Manipulate strings using RTMs facilities and regular expressions - Create normal, binomial, and Poisson probability distributions - Build linear, generalized linear, and nonlinear models - Program basic statistics: mean, standard deviation, and t-tests - Train machine learning models - Assess the quality of models and variable selection - Prevent overfitting and perform variable selection, using the Elastic Net and Bayesian methods - Analyze univariate and multivariate time series data - Group data via K-means and hierarchical clustering - Prepare reports, slideshows, and web pages with knitr - Display interactive data with RMarkdown and htmlwidgets - Implement dashboards with Shiny - Build reusable R packages with devtools and Rcpp Cover 1 Title Page 4 Copyright Page 5 Contents 8 Foreword 16 Preface 18 Acknowledgments 22 About the Author 26 1 Getting R 28 1.1 Downloading R 28 1.2 R Version 29 1.3 32-bit vs. 64-bit 29 1.4 Installing 29 1.5 Microsoft R Open 41 1.6 Conclusion 41 2 The R Environment 42 2.1 Command Line Interface 43 2.2 RStudio 44 2.3 Microsoft Visual Studio 58 2.4 Conclusion 58 3 R Packages 60 3.1 Installing Packages 60 3.2 Loading Packages 63 3.3 Building a Package 64 3.4 Conclusion 64 4 Basics of R 66 4.1 Basic Math 66 4.2 Variables 67 4.3 Data Types 69 4.4 Vectors 74 4.5 Calling Functions 79 4.6 Function Documentation 79 4.7 Missing Data 80 4.8 Pipes 81 4.9 Conclusion 82 5 Advanced Data Structures 84 5.1 data.frames 84 5.2 Lists 91 5.3 Matrices 97 5.4 Arrays 100 5.5 Conclusion 101 6 Reading Data into R 102 6.1 Reading CSVs 102 6.2 Excel Data 106 6.3 Reading from Databases 108 6.4 Data from Other Statistical Tools 111 6.5 R Binary Files 112 6.6 Data Included with R 114 6.7 Extract Data from Web Sites 115 6.8 Reading JSON Data 117 6.9 Conclusion 119 7 Statistical Graphics 120 7.1 Base Graphics 120 7.2 ggplot2 123 7.3 Conclusion 137 8 Writing R functions 138 8.1 Hello, World! 138 8.2 Function Arguments 139 8.3 Return Values 141 8.4 do.call 142 8.5 Conclusion 143 9 Control Statements 144 9.1 if and else 144 9.2 switch 147 9.3 ifelse 148 9.4 Compound Tests 150 9.5 Conclusion 150 10 Loops, the Un-R Way to Iterate 152 10.1 for Loops 152 10.2 while Loops 154 10.3 Controlling Loops 154 10.4 Conclusion 155 11 Group Manipulation 156 11.1 Apply Family 156 11.2 aggregate 159 11.3 plyr 163 11.4 data.table 167 11.5 Conclusion 177 12 Faster Group Manipulation with dplyr 178 12.1 Pipes 178 12.2 tbl 179 12.3 select 180 12.4 filter 188 12.5 slice 194 12.6 mutate 195 12.7 summarize 198 12.8 group_by 199 12.9 arrange 200 12.10 do 201 12.11 dplyr with Databases 203 12.12 Conclusion 205 13 Iterating with purrr 206 13.1 map 206 13.2 map with Specified Types 208 13.3 Iterating over a data.frame 213 13.4 map with Multiple Inputs 214 13.5 Conclusion 215 14 Data Reshaping 216 14.1 cbind and rbind 216 14.2 Joins 217 14.3 reshape2 224 14.4 Conclusion 227 15 Reshaping Data in the Tidyverse 228 15.1 Binding Rows and Columns 228 15.2 Joins with dplyr 229 15.3 Converting Data Formats 234 15.4 Conclusion 237 16 Manipulating Strings 238 16.1 paste 238 16.2 sprintf 239 16.3 Extracting Text 240 16.4 Regular Expressions 244 16.5 Conclusion 251 17 Probability Distributions 252 17.1 Normal Distribution 252 17.2 Binomial Distribution 257 17.3 Poisson Distribution 262 17.4 Other Distributions 265 17.5 Conclusion 267 18 Basic Statistics 268 18.1 Summary Statistics 268 18.2 Correlation and Covariance 271 18.3 T-Tests 279 18.4 ANOVA 287 18.5 Conclusion 290 19 Linear Models 292 19.1 Simple Linear Regression 292 19.2 Multiple Regression 297 19.3 Conclusion 314 20 Generalized Linear Models 316 20.1 Logistic Regression 316 20.2 Poisson Regression 320 20.3 Other Generalized Linear Models 324 20.4 Survival Analysis 324 20.5 Conclusion 329 21 Model Diagnostics 330 21.1 Residuals 330 21.2 Comparing Models 336 21.3 Cross-Validation 340 21.4 Bootstrap 345 21.5 Stepwise Variable Selection 348 21.6 Conclusion 351 22 Regularization and Shrinkage 352 22.1 Elastic Net 352 22.2 Bayesian Shrinkage 369 22.3 Conclusion 373 23 Nonlinear Models 374 23.1 Nonlinear Least Squares 374 23.2 Splines 377 23.3 Generalized Additive Models 380 23.4 Decision Trees 386 23.5 Boosted Trees 388 23.6 Random Forests 391 23.7 Conclusion 393 24 Time Series and Autocorrelation 394 24.1 Autoregressive Moving Average 394 24.2 VAR 401 24.3 GARCH 406 24.4 Conclusion 415 25 Clustering 416 25.1 K-means 416 25.2 PAM 424 25.3 Hierarchical Clustering 430 25.4 Conclusion 434 26 Model Fitting with Caret 436 26.1 Caret Basics 436 26.2 Caret Options 436 26.3 Tuning a Boosted Tree 438 26.4 Conclusion 442 27 Reproducibility and Reports with knitr 444 27.1 Installing a LaTeX Program 444 27.2 LaTeX Primer 445 27.3 Using knitr with LaTeX 447 27.4 Conclusion 453 28 Rich Documents with RMarkdown 454 28.1 Document Compilation 454 28.2 Document Header 454 28.3 Markdown Primer 456 28.4 Markdown Code Chunks 457 28.5 htmlwidgets 459 28.6 RMarkdown Slideshows 471 28.7 Conclusion 473 29 Interactive Dashboards with Shiny 474 29.1 Shiny in RMarkdown 474 29.2 Reactive Expressions in Shiny 479 29.3 Server and UI 481 29.4 Conclusion 490 30 Building R Packages 492 30.1 Folder Structure 492 30.2 Package Files 492 30.3 Package Documentation 499 30.4 Tests 502 30.5 Checking, Building and Installing 504 30.6 Submitting to CRAN 506 30.7 C++ Code 506 30.8 Conclusion 511 A: Real-Life Resources 512 A.1 Meetups 512 A.2 Stack Overflow 513 A.3 Twitter 514 A.4 Conferences 514 A.5 Web Sites 515 A.6 Documents 515 A.7 Books 515 A.8 Conclusion 516 B: Glossary 518 A 518 B 519 C 520 D 521 E 521 F 522 G 522 H 523 I 524 J 524 K 524 L 524 M 525 N 527 O 527 P 528 Q 529 R 529 S 530 T 531 U 531 V 532 W 532 X 532 List of Figures 534 List of Tables 540 General Index 542 A 542 B 542 C 542 D 542 E 543 F 543 G 543 H 543 I 543 J 543 K 543 L 543 M 544 N 544 O 544 P 544 Q 544 R 544 S 545 T 545 U 545 V 545 W 545 X 546 Y 546 Z 546 Index of Functions 548 A 548 B 548 C 548 D 549 E 549 F 549 G 549 H 550 I 550 J 550 K 550 L 550 M 550 N 550 O 550 P 551 Q 551 R 551 S 551 T 552 U 552 V 552 W 552 X 552 Y 552 Index of Packages 554 A 554 B 554 C 554 D 554 F 554 G 554 H 554 J 554 K 554 L 554 M 554 N 554 P 554 Q 554 R 554 S 555 T 555 U 555 V 555 W 555 X 555 Index of People 556 Data Index 558 Statistical Computation for Programmers, Scientists, Quants, Excel Users, and Other Professionals Using the open source R language, you can build powerful statistical models to answer many of your most challenging questions. R has traditionally been difficult for non-statisticians to learn, and most R books assume far too much knowledge to be of help. R for Everyone, Second Edition, is the solution. Drawing on his unsurpassed experience teaching new users, professional data scientist Jared P. Lander has written the perfect tutorial for anyone new to statistical programming and modeling. Organized to make learning easy and intuitive, this guide focuses on the 20 percent of R functionality you'll need to accomplish 80 percent of modern data tasks. Lander's self-contained chapters start with the absolute basics, offering extensive hands-on practice and sample code. You'll download and install R; navigate and use the R environment; master basic program control, data import, manipulation, and visualization; and walk through several essential tests. Then, building on this foundation, you'll construct several complete models, both linear and nonlinear, and use some data mining techniques. After all this you'll make your code reproducible with LaTeX, RMarkdown, and Shiny. By the time you're done, you won't just know how to write R programs, you'll be ready to tackle the statistical problems you care about most. Coverage includes Explore R, RStudio, and R packages Use R for math: variable types, vectors, calling functions, and more Exploit data structures, including data.frames, matrices, and lists Read many different types of data Create attractive, intuitive statistical graphics Write user-defined functions Control program flow with if, ifelse, and complex checks Improve program efficiency with group manipulations Combine and reshape multiple datasets Manipulate strings using R's facilities and regular expressions Create normal, binomial, and Poisson probability distributions Build linear, generalized linear, and nonlinear models Program basic statistics: mean, standard deviation, and t-tests Train machine learning models Assess the quality of models and variable selection Prevent overfitting and perform variable selection, using the Elastic Net and Bayesian methods Analyze univariate and multivariate time series data Group data via K-means and hierarchical clustering Prepare reports, slideshows, and web pages with knitr Display interactive data with RMarkdown and htmlwidgets Implement dashboards with Shiny Build reusable R packages with devtools and Rcpp Register your product at informit.com/register for convenient access to downloads, updates, and corrections as they become available. Using the open source R language, you can build powerful statistical models to answer many of your most challenging questions. R has traditionally been difficult for non-statisticians to learn, and most R books assume far too much knowledge to be of help. R for Everyone is the solution. Drawing on his unsurpassed experience teaching new users, professional data scientist Jared P. Lander has written the perfect tutorial for anyone new to statistical programming and modeling. Organized to make learning easy and intuitive, this guide focuses on the 20 percent of R functionality youll need to accomplish 80 percent of modern data tasks. Landers self-contained chapters start with the absolute basics, offering extensive hands-on practice and sample code. Youll download and install R; navigate and use the R environment; master basic program control, data import, and manipulation; and walk through several essential tests. Then, building on this foundation, youll construct several complete models, both linear and nonlinear, and use some data mining techniques. By the time youre done, you wont just know how to write R programs, youll be ready to tackle the statistical problems you care about most. Coverage