Computational Statistics (Statistics and Computing)

Computational inference has taken its place alongside asymptotic inference and exact techniques in the standard collection of statistical methods. Computational inference is based on an approach to statistical methods that uses modern computational power to simulate distributional properties of estimators and test statistics. This book describes computationally-intensive statistical methods in a unified presentation, emphasizing techniques, such as the PDF decomposition, that arise in a wide range of methods. The book assumes an intermediate background in mathematics, computing, and applied and theoretical statistics. The first part of the book, consisting of a single long chapter, reviews this background material while introducing computationally-intensive exploratory data analysis and computational inference. The six chapters in the second part of the book are on statistical computing. This part describes arithmetic in digital computers and how the nature of digital computations affects algorithms used in statistical methods. Building on the first chapters on numerical computations and algorithm design, the following chapters cover the main areas of statistical numerical analysis, that is, approximation of functions, numerical quadrature, numerical linear algebra, solution of nonlinear equations, optimization, and random number generation. The third and fourth parts of the book cover methods of computational statistics, including Monte Carlo methods, randomization and cross validation, the bootstrap, probability density estimation, and statistical learning. The book includes a large number of exercises with some solutions provided in an appendix. James E. Gentle is University Professor of Computational Statistics at George Mason University. He is a Fellow of the American Statistical Association (ASA) and of the American Association for the Advancement of Science. He has held several national offices in the ASA and has served as associate editor of journals of the ASA as well as for other journals in statistics and computing. He is author of __Random Number Generation and Monte Carlo Methods__ and __Matrix Algebra__. Contents......Page 18 Preface......Page 8 Part I: Preliminaries......Page 23 Introduction to Part I......Page 25 Mathematical and Statistical Preliminaries......Page 27 1.1 Discovering Structure in Data......Page 28 1.2 Mathematical Tools for Identifying Structure in Data......Page 32 1.3 Data-Generating Processes; Probability Distributions......Page 51 1.4 Statistical Inference......Page 59 1.5 Probability Statements in Statistical Inference......Page 74 1.6 Modeling and Computational Inference......Page 78 1.7 The Role of the Empirical Cumulative Distribution Function......Page 81 1.8 The Role of Optimization in Inference......Page 87 Notes and Further Reading......Page 96 Exercises......Page 97 Part II: Statistical Computing......Page 103 Introduction to Part II......Page 105 Computer Storage and Arithmetic......Page 107 2.1 The Fixed-Point Number System......Page 108 2.2 The Floating-Point Number System......Page 110 2.3 Errors......Page 119 Notes and Further Reading......Page 123 Exercises......Page 124 Algorithms and Programming......Page 129 3.1 Error in Numerical Computations......Page 131 3.2 Algorithms and Data......Page 135 3.3 Efficiency......Page 138 3.4 Iterations and Convergence......Page 150 3.5 Programming......Page 156 3.6 Computational Feasibility......Page 159 Notes and Further Reading......Page 160 Exercises......Page 164 Approximation of Functions and Numerical Quadrature......Page 169 4.1 Function Approximation and Smoothing......Page 175 4.2 Basis Sets in Function Spaces......Page 182 4.3 Orthogonal Polynomials......Page 189 4.4 Splines......Page 200 4.5 Kernel Methods......Page 204 4.6 Numerical Quadrature......Page 206 4.7 Monte Carlo Methods for Quadrature......Page 214 Notes and Further Reading......Page 219 Exercises......Page 221 Numerical Linear Algebra......Page 225 5.1 General Computational Considerations for Vectors and Matrices......Page 227 5.2 Gaussian Elimination and Elementary Operator Matrices......Page 231 5.3 Matrix Decompositions......Page 237 5.4 Iterative Methods......Page 243 5.5 Updating a Solution to a Consistent System......Page 249 5.6 Overdetermined Systems; Least Squares......Page 250 5.7 Other Computations with Matrices......Page 257 Notes and Further Reading......Page 258 Exercises......Page 259 Solution of Nonlinear Equations and Optimization......Page 263 6.1 Finding Roots of Equations......Page 266 6.2 Unconstrained Descent Methods in Dense Domains......Page 283 6.3 Unconstrained Combinatorial and Stochastic Optimization......Page 297 6.4 Optimization under Constraints......Page 306 6.5 Computations for Least Squares......Page 313 6.6 Computations for Maximum Likelihood......Page 316 Notes and Further Reading......Page 320 Exercises......Page 323 7.1 Randomness of Pseudorandom Numbers......Page 327 7.2 Generation of Nonuniform Random Numbers......Page 329 7.3 Acceptance/Rejection Method Using a Markov Chain......Page 335 7.4 Generation of Multivariate Random Variates......Page 337 7.5 Data-Based Random Number Generation......Page 340 7.6 Software for Random Number Generation......Page 342 Exercises......Page 351 Part III: Methods of Computational Statistics......Page 355 Introduction to Part III......Page 357 Graphical Methods in Computational Statistics......Page 359 8.1 Smoothing and Drawing Lines......Page 363 8.2 Viewing One, Two, or Three Variables......Page 366 8.3 Viewing Multivariate Data......Page 377 Notes and Further Reading......Page 387 Exercises......Page 390 Tools for Identification of Structure in Data......Page 393 9.1 Transformations......Page 395 9.2 Measures of Similarity and Dissimilarity......Page 405 Exercises......Page 419 Estimation of Functions......Page 423 10.1 General Approaches to Function Estimation......Page 425 10.2 Pointwise Properties of Function Estimators......Page 429 10.3 Global Properties of Estimators of Functions......Page 432 Exercises......Page 436 Monte Carlo Methods for Statistical Inference......Page 439 11.1 Monte Carlo Estimation......Page 440 11.2 Simulation of Data from a Hypothesized Model: Monte Carlo Tests......Page 444 11.4 Random Sampling from Data......Page 446 11.5 Reducing Variance in Monte Carlo Methods......Page 447 11.6 Software for Monte Carlo......Page 451 Notes and Further Reading......Page 452 Exercises......Page 453 Data Randomization, Partitioning, and Augmentation......Page 457 12.1 Randomization Methods......Page 458 12.2 Cross Validation for Smoothing and Fitting......Page 462 12.3 Jackknife Methods......Page 464 Notes and Further Reading......Page 470 Exercises......Page 471 Bootstrap Methods......Page 475 13.1 Bootstrap Bias Corrections......Page 476 13.2 Bootstrap Estimation of Variance......Page 478 13.3 Bootstrap Confidence Intervals......Page 479 13.4 Bootstrapping Data with Dependencies......Page 483 13.5 Variance Reduction in Monte Carlo Bootstrap......Page 484 Notes and Further Reading......Page 486 Exercises......Page 487 Part IV: Exploring Data Density and Relationships......Page 491 Introduction to Part IV......Page 493 Estimation of Probability Density Functions Using Parametric Models......Page 497 14.1 Fitting a Parametric Probability Distribution......Page 498 14.2 General Families of Probability Distributions......Page 499 14.3 Mixtures of Parametric Families......Page 502 14.4 Statistical Properties of Density Estimators Based on Parametric Families......Page 504 Notes and Further Reading......Page 505 Exercises......Page 506 15.1 The Likelihood Function......Page 509 15.2 Histogram Estimators......Page 512 15.3 Kernel Estimators......Page 521 15.4 Choice of Window Widths......Page 526 15.5 Orthogonal Series Estimators......Page 527 15.6 Other Methods of Density Estimation......Page 528 Notes and Further Reading......Page 531 Exercises......Page 532 Statistical Learning and Data Mining......Page 537 16.1 Clustering and Classification......Page 541 16.2 Ordering and Ranking Multivariate Data......Page 560 16.3 Linear Principal Components......Page 570 16.4 Variants of Principal Components......Page 582 16.5 Projection Pursuit......Page 586 16.6 Other Methods for Identifying Structure......Page 594 16.7 Higher Dimensions......Page 595 Notes and Further Reading......Page 600 Exercises......Page 602 Statistical Models of Dependencies......Page 607 17.1 Regression and Classification Models......Page 610 17.2 Probability Distributions in Models......Page 619 17.3 Fitting Models to Data......Page 622 17.4 Classification......Page 642 17.5 Transformations......Page 650 Notes and Further Reading......Page 656 Exercises......Page 658 Appendices......Page 663 Monte Carlo Studies in Statistics......Page 665 A.1 Simulation as an Experiment......Page 666 A.2 Reporting Simulation Experiments......Page 667 A.3 An Example......Page 668 A.4 Computer Experiments......Page 675 Exercises......Page 677 Some Important Probability Distributions......Page 679 C.1 General Notation......Page 685 C.2 Computer Number Systems......Page 687 C.3 Notation Relating to Random Variables......Page 688 C.4 General Mathematical Functions and Operators......Page 690 C.5 Models and Data......Page 697 Solutions and Hints for Selected Exercises......Page 699 Bibliography......Page 711 E.1 Literature in Computational Statistics......Page 712 E.3 References to the Literature......Page 715 B......Page 737 C......Page 738 D......Page 739 F......Page 740 H......Page 741 J......Page 742 L......Page 743 N......Page 744 P......Page 745 R......Page 746 S......Page 747 T......Page 748 Z......Page 749 Computational Inference Has Taken Its Place Alongside Asymptotic Inference And Exact Techniques In The Standard Collection Of Statistical Methods. Computational Inference Is Based On An Approach To Statistical Methods That Uses Modern Computational Power To Simulate Distributional Properties Of Estimators And Test Statistics. This Book Describes Computationally-intensive Statistical Methods In A Unified Presentation, Emphasizing Techniques, Such As The Pdf Decomposition, That Arise In A Wide Range Of Methods. The Book Assumes An Intermediate Background In Mathematics, Computing, And Applied And Theoretical Statistics. The First Part Of The Book, Consisting Of A Single Long Chapter, Reviews This Background Material While Introducing Computationally-intensive Exploratory Data Analysis And Computational Inference. The Six Chapters In The Second Part Of The Book Are On Statistical Computing.^ This Part Describes Arithmetic In Digital Computers And How The Nature Of Digital Computations Affects Algorithms Used In Statistical Methods. Building On The First Chapters On Numerical Computations And Algorithm Design, The Following Chapters Cover The Main Areas Of Statistical Numerical Analysis, That Is, Approximation Of Functions, Numerical Quadrature, Numerical Linear Algebra, Solution Of Nonlinear Equations, Optimization, And Random Number Generation. The Third And Fourth Parts Of The Book Cover Methods Of Computational Statistics, Including Monte Carlo Methods, Randomization And Cross Validation, The Bootstrap, Probability Density Estimation, And Statistical Learning. The Book Includes A Large Number Of Exercises With Some Solutions Provided In An Appendix. James E. Gentle Is University Professor Of Computational Statistics At George Mason University. He Is A Fellow Of The American Statistical Association (asa) And Of The American Association For The Advancement Of Science.^ He Has Held Several National Offices In The Asa And Has Served As Associate Editor Of Journals Of The Asa As Well As For Other Journals In Statistics And Computing. He Is Author Of Random Number Generation And Monte Carlo Methods And Matrix Algebra. Pt. I, Preliminaries. Mathematical And Statistical Preliminaries -- Pt. Ii, Statistical Computing. Computer Storage And Arithmetic ; Algorithm And Programming ; Approximation Of Functions And Numerical Quadrature ; Numerical Linear Algebra ; Solution Of Nonlinear Equations And Optimization ; Generation Of Random Numbers -- Pt. Iii, Methods Of Computational Statistics. Graphical Methods In Computational Statistics ; Tools For Identification Of Structure In Data ; Estimation Of Functions ; Monte Carlo Methods For Statistical Inference ; Data Randomization, Partitioning, And Augmentation ; Bootstrap Methods -- Pt. Iv, Exploring Data Density And Relationships. Estimation Of Probability Density Functions Using Parametric Models ; Nonparametric Estimation Of Probability Density Functions ; Statistical Learning And Data Mining ; Statistical Models Of Dependencies. James E. Gentle. Includes Bibliographical References (p. 690-714) And Index. Computational inference is based on an approach to statistical methods that uses modern computational power to simulate distributional properties of estimators and test statistics. This book describes computationally intensive statistical methods in a unified presentation, emphasizing techniques, such as the PDF decomposition, that arise in a wide range of methods. "This book began as a revision of 'Elements of Computational Statistics, published ... in 2002... The present book includes most of the topics from 'Elements' ... The emphasis is still on computationally-intensive statistical methods, but there is a substantial portion on the numerical methods supporting the statistical applications"--PREFACE