Assessing and Improving Prediction and Classification : Theory and Algorithms in C++

Assess the quality of your prediction and classification models in ways that accurately reflect their real-world performance, and then improve this performance using state-of-the-art algorithms such as committee-based decision making, resampling the dataset, and boosting. This book presents many important techniques for building powerful, robust models and quantifying their expected behavior when put to work in your application.Considerable attention is given to information theory, especially as it relates to discovering and exploiting relationships between variables employed by your models. This presentation of an often confusing subject avoids advanced mathematics, focusing instead on concepts easily understood by those with modest background in mathematics.All algorithms include an intuitive explanation of operation, essential equations, references to more rigorous theory, and commented C++ source code. Many of these techniques are recent developments, still not in widespread use. Others are standard algorithms given a fresh look. In every case, the emphasis is on practical applicability, with all code written in such a way that it can easily be included in any program.**What You'll Learn** * Compute entropy to detect problematic predictors. * Compute confidence and tolerance intervals for predictions, as well as confidence levels for classification decisions. * Improve numeric predictions using constrained and unconstrained combinations, variance-weighted interpolation, and kernel-regression smoothing. * Improve classification decisions using Borda counts, MinMax and MaxMin rules, union and intersection rules, logistic regression, selection by local accuracy, maximization of the fuzzy integral, and pairwise coupling. * Use information-theoretic techniques to rapidly screen large numbers of candidate predictors, identifying those that are especially promising. * Use Monte-Carlo permutation methods to assess the role of good luck in performance results. **Who This Book is For**Anyone who creates prediction or classification models will find a wealth of useful algorithms in this book. Although all code examples are written in C++, the algorithms are described in sufficient detail that they can easily be programmed in any language. Table of Contents About the Author About the Technical Reviewers Preface 1 Chapter 1: Assessment of Numeric Predictions Notation Overview of Performance Measures Consistency and Evolutionary Stability Selection Bias and the Need for Three Datasets Cross Validation and Walk-Forward Testing Bias in Cross Validation Overlap Considerations Assessing Nonstationarity Using Walk-Forward Testing Nested Cross Validation Revisited Common Performance Measures Mean Squared Error Mean Absolute Error R-Squared RMS Error Nonparametric Correlation Success Ratios Alternatives to Common Performance Measures Stratification for Consistency Confidence Intervals The Confidence Set Serial Correlation Multiplicative Data Normally Distributed Errors Empirical Quantiles as Confidence Intervals Confidence Bounds for Quantiles Tolerance Intervals 2 Chapter 2: Assessment of Class Predictions The Confusion Matrix Expected Gain/Loss ROC (Receiver Operating Characteristic) Curves Hits, False Alarms, and Related Measures Computing the ROC Curve Area Under the ROC Curve Cost and the ROC Curve Optimizing ROC-Based Statistics Optimizing the Threshold: Now or Later? Maximizing Precision Generalized Targets Maximizing Total Gain Maximizing Mean Gain Maximizing the Standardized Mean Gain Confidence in Classification Decisions Hypothesis Testing Confidence in the Confidence Bayesian Methods Multiple Classes Hypothesis Testing vs. Bayes’ Method Final Thoughts on Hypothesis Testing Confidence Intervals for Future Performance 3 Chapter 3: Resampling for Assessing Parameter Estimates Bias and Variance of Statistical Estimators Plug-in Estimators and Empirical Distributions Bias of an Estimator Variance of an Estimator Bootstrap Estimation of Bias and Variance Code for Bias and Variance Estimation Plug-in Estimators Can Provide Better Bootstraps A Model Parameter Example Confidence Intervals Is the Interval Backward? Improving the Percentile Method Hypothesis Tests for Parameter Values Bootstrapping Ratio Statistics Jackknife Estimates of Bias and Variance Bootstrapping Dependent Data Estimating the Extent of Autocorrelation The Stationary Bootstrap Choosing a Block Size for the Stationary Bootstrap The Tapered Block Bootstrap Choosing a Block Size for the Tapered Block Bootstrap What If the Block Size Is Wrong? 4 Chapter 4: Resampling for Assessing Prediction and Classification Partitioning the Error Cross Validation Bootstrap Estimation of Population Error Efron’s E0 Estimate of Population Error Efron’s E632 Estimate of Population Error Comparing the Error Estimators for Prediction Comparing the Error Estimators for Classification Summary 5 Chapter 5: Miscellaneous Resampling Techniques Bagging A Quasi-theoretical Justification The Component Models Code for Bagging AdaBoost Binary AdaBoost for Pure Classification Models Probabilistic Sampling for Inflexible Models Binary AdaBoost When the Model Provides Confidence AdaBoost.MH for More Than Two Classes AdaBoost.OC for More Than Two Classes Comparing the Boosting Algorithms A Binary Classification Problem A Multiple-Class Problem Final Thoughts on Boosting Permutation Training and Testing The Permutation Training Algorithm Partitioning the Training Performance A Demonstration of Permutation Training 6 Chapter 6: Combining Numeric Predictions Simple Average Code for Averaging Predictions Unconstrained Linear Combinations Constrained Linear Combinations Constrained Combination of Unbiased Models Variance-Weighted Interpolation Combination by Kernel Regression Smoothing Code for the GRNN Comparing the Combination Methods 7 Chapter 7: Combining Classification Models Introduction and Notation Reduction vs. Ordering The Majority Rule Code for the Majority Rule The Borda Count The Average Rule Code for the Average Rule The Median Alternative The Product Rule The MaxMax and MaxMin Rules The Intersection Method The Union Rule Logistic Regression Code for the Combined Weight Method The Logit Transform and Maximum Likelihood Estimation Code for Logistic Regression Separate Weight Sets Model Selection by Local Accuracy Code for Local Accuracy Selection Maximizing the Fuzzy Integral What Does This Have to Do with Classifier Combination? Code for the Fuzzy Integral Pairwise Coupling Pairwise Threshold Optimization A Cautionary Note Comparing the Combination Methods Small Training Set, Three Models Large Training Set, Three Models Small Training Set, Three Good Models, One Worthless Large Training Set, Three Good Models, One Worthless Small Training Set, Worthless and Noisy Models Included Large Training Set, Worthless and Noisy Models Included Five Classes 8 Chapter 8: Gating Methods Preordained Specialization Learned Specialization After-the-Fact Specialization Code for After-the-Fact Specialization Some Experimental Results General Regression Gating Code for GRNN Gating Experiments with GRNN Gating 9 Chapter 9: Information and Entropy Entropy Entropy of a Continuous Random Variable Partitioning a Continuous Variable for Entropy An Example of Improving Entropy Joint and Conditional Entropy Code for Conditional Entropy Mutual Information Fano’s Bound and Selection of Predictor Variables Confusion Matrices and Mutual Information Extending Fano’s Bound for Upper Limits Simple Algorithms for Mutual Information The TEST_DIS Program Continuous Mutual Information The Parzen Window Method Adaptive Partitioning The TEST_CON Program Predictor Selection Using Mutual Information Maximizing Relevance While Minimizing Redundancy The MI_DISC and MI_CONT Programs A Contrived Example of Information Minus Redundancy A Superior Selection Algorithm for Binary Variables Screening Without Redundancy Asymmetric Information Measures Uncertainty Reduction Transfer Entropy: Schreiber’s Information Transfer References References Index Assess the quality of your prediction and classification models in ways that accurately reflect their real-world performance, and then improve this performance using state-of-the-art algorithms such as committee-based decision making, resampling the dataset, and boosting. This book presents many important techniques for building powerful, robust models and quantifying their expected behavior when put to work in your application. Considerable attention is given to information theory, especially as it relates to discovering and exploiting relationships between variables employed by your models. This presentation of an often confusing subject avoids advanced mathematics, focusing instead on concepts easily understood by those with modest background in mathematics. All algorithms include an intuitive explanation of operation, essential equations, references to more rigorous theory, and commented C++ source code. Many of these techniques are recent developments, still not in widespread use. Others are standard algorithms given a fresh look. In every case, the emphasis is on practical applicability, with all code written in such a way that it can easily be included in any program. What You'll Learn Compute entropy to detect problematic predictors Improve numeric predictions using constrained and unconstrained combinations, variance-weighted interpolation, and kernel-regression smoothing Carry out classification decisions using Borda counts, MinMax and MaxMin rules, union and intersection rules, logistic regression, selection by local accuracy, maximization of the fuzzy integral, and pairwise coupling Harness information-theoretic techniques to rapidly screen large numbers of candidate predictors, identifying those that are especially promising Use Monte-Carlo permutation methods to assess the role of good luck in performance results Compute confidence and tolerance intervals for predictions, as well as confidence levels for classification decisions Who This Book is For Anyone who creates prediction or classification models will find a wealth of useful algorithms in this book. Although all code examples are written in C++, the algorithms are described in sufficient detail that they can easily be programmed in any language. Assess the quality of your prediction and classification models in ways that accurately reflect their real-world performance, and then improve this performance using state-of-the-art algorithms such as committee-based decision making, resampling the dataset, and boosting. This book presents many important techniques for building powerful, robust models and quantifying their expected behavior when put to work in your application. Considerable attention is given to information theory, especially as it relates to discovering and exploiting relationships between variables employed by your models. This presentation of an often confusing subject avoids advanced mathematics, focusing instead on concepts easily understood by those with modest background in mathematics. All algorithms include an intuitive explanation of operation, essential equations, references to more rigorous theory, and commented C++ source code. Many of these techniques are recent developments, still not in widespread use. Others are standard algorithms given a fresh look. In every case, the emphasis is on practical applicability, with all code written in such a way that it can easily be included in any program. What You'll Learn Compute entropy to detect problematic predictors. Compute confidence and tolerance intervals for predictions, as well as confidence levels for classification decisions. Improve numeric predictions using constrained and unconstrained combinations, variance-weighted interpolation, and kernel-regression smoothing. Improve classification decisions using Borda counts, MinMax and MaxMin rules, union and intersection rules, logistic regression, selection by local accuracy, maximization of the fuzzy integral, and pairwise coupling. Use information-theoretic techniques to rapidly screen large numbers of candidate predictors, identifying those that are especially promising. Use Monte-Carlo permutation methods to assess the role of good luck in performance results. Who This Book is For Anyone who creates prediction or classification models will find a wealth of useful algorithms in this book. Although all code examples are written in C++, the algorithms are described in sufficient detail that they can easily be programmed in any language Carry out practical, real-life assessments of the performance of prediction and classification models written in C++. This book discusses techniques for improving the performance of such models by intelligent resampling of training/testing data, combining multiple models into sophisticated committees, and making use of exogenous information to dynamically choose modeling methodologies. Rigorous statistical techniques for computing confidence in predictions and decisions receive extensive treatment. Finally, the last part of the book is devoted to the use of information theory in evaluating and selecting useful predictors. Special attention is paid to Schreiber's Information Transfer, a recent generalization of Grainger Causality. Well commented C++ code is given for every algorithm and technique. You will: Discover the hidden pitfalls that lurk in the model development process Work with some of the most powerful model enhancement algorithms that have emerged recently Effectively use and incorporate the C++ code in your own data analysis projects Combine classification models to enhance your projects Carry out practical, real-life assessments of the performance of prediction and classification models written in C++. This book discusses techniques for improving the performance of such models by intelligent resampling of training/testing data, combining multiple models into sophisticated committees, and making use of exogenous information to dynamically choose modeling methodologies. Rigorous statistical techniques for computing confidence in predictions and decisions receive extensive treatment.Ʃnally, the last part of the book is devoted to the use of information theory in evaluating and selecting useful predictors. Special attention is paid to Schreiber's Information Transfer, a recent generalization of Grainger Causality. Well commented C++ code is given for every algorithm and technique.ٯu will: Discover the hidden pitfalls that lurk in the model development process Work with some of the most powerful model enhancement algorithms that have emerged recently Effectively use and incorporate the C++ code in your own data analysis projects Combine classification models to enhance your projects