Modern Data Mining Algorithms in C++ and CUDA C : Recent Developments in Feature Extraction and Selection Algorithms for Data Science

Discover a variety of data-mining algorithms that are useful for selecting small sets of important features from among unwieldy masses of candidates, or extracting useful features from measured variables. As a serious data miner you will often be faced with thousands of candidate features for your prediction or classification application, with most of the features being of little or no value. You’ll know that many of these features may be useful only in combination with certain other features while being practically worthless alone or in combination with most others. Some features may have enormous predictive power, but only within a small, specialized area of the feature space. The problems that plague modern data miners are endless. This book helps you solve this problem by presenting modern feature selection techniques and the code to implement them. Some of these techniques are: Forward selection component analysis Local feature selection Linking features and a target with a hidden Markov model Improvements on traditional stepwise selection Nominal-to-ordinal conversion All algorithms are intuitively justified and supported by the relevant equations and explanatory material. The author also presents and explains complete, highly commented source code. The example code is in C++ and CUDA C but Python or other code can be substituted; the algorithm is important, not the code that's used to write it. What You Will Learn Combine principal component analysis with forward and backward stepwise selection to identify a compact subset of a large collection of variables that captures the maximum possible variation within the entire set. Identify features that may have predictive power over only a small subset of the feature domain. Such features can be profitably used by modern predictive models but may be missed by other feature selection methods. Find an underlying hidden Markov model that controls the distributions of feature variables and the target simultaneously. The memory inherent in this method is especially valuable in high-noise applications such as prediction of financial markets. Improve traditional stepwise selection in three ways: examine a collection of 'best-so-far' feature sets; test candidate features for inclusion with cross validation to automatically and effectively limit model complexity; and at each step estimate the probability that our results so far could be just the product of random good luck. We also estimate the probability that the improvement obtained by adding a new variable could have been just good luck. Take a potentially valuable nominal variable (a category or class membership) that is unsuitable for input to a prediction model, and assign to each category a sensible numeric value that can be used as a model input. Who This Book Is For Intermediate to advanced data science programmers and analysts. C++ and CUDA C experience is highly recommended. However, this book can be used as a framework using other languages such as Python. Table of Contents About the Author About the Technical Reviewer Chapter 1: Introduction Chapter 2: Forward Selection Component Analysis Introduction to Forward Selection Component Analysis The Mathematics and Code Examples Maximizing the Explained Variance Code for the Variance Maximization Criterion Backward Refinement Multithreading Backward Refinement Orthogonalizing Ordered Components Putting It All Together Components from a Forward-Only Subset Components from a Backward Refined Subset An Example with Contrived Variables Chapter 3: Local Feature Selection Intuitive Overview of the Algorithm What This Algorithm Reports A Brief Detour: The Simplex Algorithm The Linear Programming Problem Interfacing to the Simplex Class A Little More Detail A More Rigorous Approach to LFS Intra-Class and Inter-Class Separation Computing the Weights Maximizing Inter-Class Separation Minimizing Intra-Class Separation Testing a Trial Beta A Quick Note on Threads CUDA Computation of Weights Integrating the CUDA Code into the Algorithm Initializing the CUDA Hardware Computing Differences from the Current Case Computing the Distance Matrix Computing the Minimum Distances Computing the Terms for the Weight Equation Transposing the Term Matrix Summing the Terms for the Weights Moving the Weights to the Host An Example of Local Feature Selection A Note on Runtime Chapter 4: Memory in Time Series Features A Gentle Mathematical Overview The Forward Algorithm The Backward Algorithm Correct Alpha and Beta, for Those Who Care Some Mundane Computations Means and Covariances Densities The Multivariate Normal Density Function Starting Parameters Outline of the Initialization Algorithm Perturbing Means Perturbing Covariances Perturbing Transition Probabilities A Note on Random Number Generators The Complete Optimization Algorithm Computing State Probabilities Updating the Means and Covariances Updating Initial and Transition Probabilities Assessing HMM Memory in a Time Series Linking Features to a Target Linking HMM States to the Target A Contrived and Inappropriate Example A Sensible and Practical Example Chapter 5: Stepwise Selection on Steroids The Feature Evaluation Model Code for the Foundation Model The Cross-Validated Performance Measure The Stepwise Algorithm Finding the First Variable Adding a Variable to an Existing Model Demonstrating the Algorithm Three Ways Chapter 6: Nominal-to-Ordinal Conversion Implementation Overview Testing for a Legitimate Relationship An Example from Equity Price Changes Code for Nominal-to-Ordinal Conversion The Constructor Printing the Table of Counts Computing the Mapping Function Monte-Carlo Permutation Tests Index As a serious data miner you will often be faced with thousands of candidate features for your prediction or classification application, with most of the features being of little or no value. You'll know that many of these features may be useful only in combination with certain other features while being practically worthless alone or in combination with most others. Some features may have enormous predictive power, but only within a small, specialized area of the feature space. The problems that plague modern data miners are endless. This book helps you solve this problem by presenting modern feature selction techniques and the code to implement them. Some of these techniques are: Forward selection component analysis ; Local feature selection ; Linking features and a target with a hidden markov model ; Improvements on traditional stepwise selection ; Niminal-to-ordinal conversion. All algorithms are intuitively justified and supported by the relevant equations and explanatory material. The author also presents and explains complete, highly commented source code. The example code is in C++ and CUDA C but Python or other code can be substituted; the algorithm is important, not the code that's used to write it